Package schrodinger :: Package job :: Module jobcontrol

Module jobcontrol

Core job control for python.

There are currently four major sections of this module - "Job database," "Job launching," "Job backend," and "Job hosts." The job database section deals with getting info about existing Jobs, the job launching section deals with starting up a subjob, and the job backend section provides utilities for a python script running as a job.

The philosophy of this module is to reinvent as little as possible (i.e. use the mmjob C libs where we can) and keep things fairly simple. If we need more complicated features of job control, we can hopefully add them as we go along.

Classes

[hide private]

_JobResourceHolder
A helper class to deal with mmlib resource de-allocation.

_JobInitializer
A class to handle class initialization and termination.

_JobManager

Job
A class to access a specific record in the job database.

_Backend
An interface to mmjobbe.

Host
A class to encapsulate host info from the schrodinger.hosts file.

Functions

[hide private]

get_maestro()
Can't do this at import, as jobcontrol.py is imported very early in maestro startup, and this will fail.

timestamp(msg)

read(file_)
Create and return a Job object from a jobcontrol file.

get_jobs(username='', query='all')
Return a list of Jobs in the database.

get_job_selection(username='')
Return a list of jobs from the current mmjob query selection.

launch_job(cmd, print_output=False, expandvars=True)
Run a process under job control and return a Job object.

_fix_cmd(cmd, expandvars=True)
A function to clean up the command passed to launch_job.

_fix_launch_cmd(cmd)
Split the string command into program and arguments, then fix the program path if necessary.

_fix_program_path(prog)
If the program executable doesn't exist as a file and isn't an absolute path, prepend the SCHRODINGER directory if it exists there.

get_backend()
A convenience function to see if we're running under job control.

get_runtime_path(pathname)
Return the runtime path for the input file 'pathname'.

under_job_control()
Returns True if this process is running under job control; False otherwise.

get_hostfile()
Return the name of the host file.

get_hosts()
Return a list of all Hosts in the schrodinger.hosts file.

get_host(name)
Return a Host object for the named host.

_get_host(name)
Return a Host object for 'name', with all attributes read in from the corresponding entry in schrodinger.hosts file.

host_str_to_list(hosts_str)
Convert a hosts string (Ex: "galina:1 monica:4") to a list of tuples.

host_list_to_str(host_list)
Converts a hosts list (Ev: [ ('host1',1), ('host2', 10) ] ) to a string.

get_command_line_host_list()
Return a list of (host, ncpu) tuples corresponding to the host list that is specified on the command line.

get_backend_host_list()
Return a list of (host, ncpu) tuples corresponding to the host list as determined from the SCHRODINGER_NODEFILE.

calculate_njobs(host_list=None)
Derive the number of jobs from the specified host list.

Variables

[hide private]

__doc__ = "...

_version = '$Revision: 1.105 $'

logger = logging.getLogger("schrodinger.jobcontrol")

profiling = 0
hash(x)

jobid_re = re.compile(r'(?im)^JobId:\s+(\S+)')

timestamp_format = '%Y-%m-%d-%H:%M:%S'

entry_fields = ['name', 'base', 'host', 'schrodinger', 'tmpdir...

entry_number = {'accountcodes': 21, 'base': 1, 'env': 5, 'gpgp...

_job_manager = _JobManager()

_backend_singleton = None
hash(x)

_backend_hosts = []

__package__ = 'schrodinger.job'

k = 'accountcodes'

v = 21

Function Details

[hide private]

get_jobs(username=`''`, query=`'all'`)

Return a list of Jobs in the database.

The query can be "all" or "active"; otherwise, raises an Exception.

launch_job(cmd, print_output=False, expandvars=True)

Run a process under job control and return a Job object. For a process to be under job control, it must print a valid JobId: line to stdout. If such a line isn't printed, a RuntimeError will be raised.

The cmd argument should be a list of command arguments (including the executable) as expected by the subprocess module.

If the executable is present in $SCHRODINGER or $SCHRODINGER/utilities, an absolute path does not need to be specified.

Parameters:

print_output (bool) - Determines if the output from jlaunch is printed to the terminal or not. Output will be logged (to stderr by default) if Python or JobControl debugging is turned on or if there is a launch failure, even if 'print_output' is False.
expandvars (bool) - If True, any environment variables of the form $var or ${var} will be expanded with their values by the os.path.expandvars function.

Raises:

RuntimeError - If there is a problem launching the job (e.g., no JobId gets printed). If running within Maestro, an error dialog will first be shown to the user.

_fix_cmd(cmd, expandvars=True)

A function to clean up the command passed to launch_job.

Parameters:

cmd (list of strings) - A command in a form that can be passed to subprocess.Popen.
expandvars (bool) - If True, any environment variables of the form $var or ${var} will be expanded with their values by the os.path.expandvars function.

Returns:

A tuple of the fixed up command and a boolean for the shell setting to pass to Popen.

_fix_program_path(prog)

If the program executable doesn't exist as a file and isn't an absolute path, prepend the SCHRODINGER directory if it exists there.

Return a list of command line arguments so this function can be used from launch_job when 'cmd' is a list.

get_backend()

A convenience function to see if we're running under job control. If so, return a _Backend object. Otherwise, return None.

get_runtime_path(pathname)

Return the runtime path for the input file 'pathname'.

If the pathname is of a type that job control will not copy to the job directory or no runtime file can be found, returns the original path name.

get_hostfile()

Return the name of the host file. The search path follows the standard hierarchy (local dir, $HOME/.schrodinger, $SCHRODINGER).

get_hosts()

Return a list of all Hosts in the schrodinger.hosts file. Use 'username' to use the host info for someone other than the current user.

get_host(name)

Return a Host object for the named host. If the host is not found, we return a Host object with the provided name and details that match localhost. This matches behavior that jobcontrol uses.

_get_host(name)

Return a Host object for 'name', with all attributes read in from the corresponding entry in schrodinger.hosts file. Requires that mmjob be initialized.

host_str_to_list(hosts_str)

Convert a hosts string (Ex: "galina:1 monica:4") to a list of tuples. First value of each tuple is the host, second value is # of cpus.

host_list_to_str(host_list)

Converts a hosts list (Ev: [ ('host1',1), ('host2', 10) ] ) to a string. Output example: "host1:1,host2:10"

get_command_line_host_list()

Return a list of (host, ncpu) tuples corresponding to the host list that is specified on the command line.

This function is meant to be called by scripts that are running under a toplevel job control script but are not running under jlaunch.

The host list is determined from the following sources:

SCHRODINGER_NODELIST
JOBHOST (if only a single host is specified)
"localhost" (if no host is specified)

If no SCHRODINGER_NODELIST is present in the environment, None is returned.

get_backend_host_list()

Return a list of (host, ncpu) tuples corresponding to the host list as determined from the SCHRODINGER_NODEFILE.

This function is meant to be called from scripts that are running under jlaunch (i.e. backend scripts).

Returns None if SCHRODINGER_NODEFILE is not present in the environment.

calculate_njobs(host_list=None)


Derive the number of jobs from the specified host list.
This function is useful to determine number of subjobs if user didn't
specified the '-NJOBS' option.

@type host_list: String
                 or
                 List of tuples
@param host_list: String of hosts along with optional number of cpus
                  or
                  List of tuples consist of hosts along with optional
                  number of cpus

If host list is not specified then it uses get_command_line_host_list() to
determine njobs, else uses the user provided host list.

Please note: the host_list can be passes as string or as a list of tuples.
    For example: for -GLIDE_HOST "host1 host2:4" host_list would become
        [('host1', None), ('host2', 4)]

Variables Details

[hide private]

doc

Value:

""" 
Core job control for python.

There are currently four major sections of this module - "Job database
,"
"Job launching," "Job backend," and "Job hosts." The job database sect
ion
deals with getting info about existing Jobs, the job launching section
...

entry_fields

Value:

['name',
 'base',
 'host',
 'schrodinger',
 'tmpdir',
 'env',
 'gpgpu',
 'proxyhost',
...

entry_number

Value:

{'accountcodes': 21,
 'base': 1,
 'env': 5,
 'gpgpu': 6,
 'host': 2,
 'maestrocontrols': 18,
 'memory': 20,
 'name': 0,
...

Module jobcontrol

get_jobs(username='', query='all')

launch_job(cmd, print_output=False, expandvars=True)

_fix_cmd(cmd, expandvars=True)

_fix_program_path(prog)

get_backend()

get_runtime_path(pathname)

get_hostfile()

get_hosts()

get_host(name)

_get_host(name)

host_str_to_list(hosts_str)

host_list_to_str(host_list)

get_command_line_host_list()

get_backend_host_list()

calculate_njobs(host_list=None)

__doc__

entry_fields

entry_number

get_jobs(username=`''`, query=`'all'`)

doc