clusterjob.backends package

Package for default backends

class clusterjob.backends.ClusterjobBackend[source]

Bases: object

Abstract base class for all clusterjob backends. All backends must inherit from this and implement the interface specified below.

Attributes:
  • name (str) – (default) name of the backend
  • extension (str) – extension to be used for job scripts
common_keys = ['name', 'queue', 'time', 'nodes', 'ppn', 'threads', 'mem', 'stdout', 'stderr']
cmd_submit(jobscript)[source]

Given a JobScript instance, return a command that submits the job to the scheduler. The returned command must be be a sequence of program arguments or a string, see args argument of subprocess.Popen.

get_job_id(response)[source]

Given the stdout from the command returned by cmd_submit(), return a job ID as a str, or None if the job ID cannot be determined

cmd_status(run, finished=False)[source]

Given a AsyncResult instance, return a command (cf. cmd_submit()) that queries the scheduler for the job status. If finished=True, the command should be appropriate for a run that has already finished.

get_status(respone, finished=False)[source]

Given the stdout from the command returned by cmd_status(), return one of the status code defined in clusterjob.status, or None if the status cannot be determined.

cmd_cancel(run)[source]

Given a AsyncResult instance, return a command (cf. cmd_submit()) that cancels the run.

resource_headers(jobscript)[source]

Given a JobScript instance, return a list of lines (no trailing newlines) that encode the resource requirements, to be added at the top of the rendered job script, between the shbang and the script body. At the very least, keys in the jobscript resources dict that are in the list of common_keys must be handled, or a ResourcesNotSupportedError must be raised.

replace_body_vars(body)[source]

Given a multiline string that is the body of the job script, replace the placeholders for environment variables with backend-specific realizations, and return the modified body

At a minimum the following environment variables should be handled:

Core Environment Variables

$CLUSTERJOB_ID
The job ID assigned by the scheduler after submission
$CLUSTERJOB_WORKDIR
The directory on the cluster from which the job script was submitted.
$CLUSTERJOB_SUBMIT_HOST
The hostname on which the job script was submitted.
$CLUSTERJOB_NAME
The name of the job.
$CLUSTERJOB_NODELIST
The hostname(s) on which the job script is running.
exception clusterjob.backends.ResourcesNotSupportedError[source]

Bases: exceptions.Exception

Exception to indicate that a backend is unable to encode a resource requirement