clusterjob.backends.sge module

SGE (Sun Grid Engine) backend

class clusterjob.backends.sge.SgeBackend

Bases: clusterjob.backends.ClusterjobBackend

SGE Backend

Attributes:

• name (str) – Name of the backend
• extension (str) – Extension for job scripts
• prefix (str) – The prefix for every line in the resource header
• resource_replacements (dict) – mapping of the common clusterjob resource keys to command line options of the qsub command.
• job_vars (dict) – mapping of core environment variables to PBS-specific environment variables.

Note

Nodes and threads are not directly supported on SGE, but must be set up using “parallel environments”. The configuration is set by the administrator so you have to check what they’ve called the parallel environments:

%> qconf -spl
pe1
omp

look for one with $pe_slots in the config:

%> qconf -sp pe1
%> qconf -sp omp

Call qsub with that environment and number of cores you want to use:

qsub -pe omp 8 -cwd ./myscript

Depending on how the cluster is set up, it may be necessary to pass the shell as e.g. -S /bin/bash. If this definition is missing, the run can crash with some very unclear error messages

name = 'sge'

extension = 'sge'

prefix = '#$'

cmd_submit(jobscript)

Given a JobScript instance, return a qsub command that submits the job to the scheduler, as a list of program arguments.

get_job_id(response)

Given the stdout from the command returned by cmd_submit(), return a job ID

cmd_status(run, finished=False)

Given a AsyncResult instance, return a qstat command that queries the scheduler for the job status. The same command is used for running and finished jobs.

get_status(response, finished=False)

Given the stdout from the command returned by cmd_status(), return one of the status code defined in clusterjob.status, or None if the status cannot be determined

cmd_cancel(run)

Given a AsyncResult instance, return a qdel command that cancels the run, as a list of command arguments.

resource_headers(jobscript)

Given a JobScript instance, return a list of lines that encode the resource requirements, to be added at the top of the rendered job script

replace_body_vars(body)

Given a multiline string that is the body of the job script, replace the placeholders for environment variables with backend-specific realizations, and return the modified body. See the job_vars attribute for the mappings that are performed.