Package schrodinger :: Package job :: Module jobcontrol

Module jobcontrol

Core job control for python.

There are currently four major sections of this module - "Job database," "Job launching," "Job backend," and "Job hosts." The job database section deals with getting info about existing Jobs, the job launching section deals with starting up a subjob, and the job backend section provides utilities for a python script running as a job.

The philosophy of this module is to reinvent as little as possible (i.e. use the mmjob C libs where we can) and keep things fairly simple. If we need more complicated features of job control, we can hopefully add them as we go along.

Classes

[hide private]

JobcontrolException

JobLaunchFailure

MissingFrontendException

MissingHostsFileException

UnreadableHostsFileException

_JobResourceHolder
A helper class to deal with mmlib resource de-allocation.

_JobInitializer
A class to handle class initialization and termination.

_JobManager

Job
A class to access a specific record in the job database.

_Backend
An interface to mmjobbe.

Host
A class to encapsulate host info from the schrodinger.hosts file.

Functions

[hide private]

timestamp(msg)

read(file_)
Create and return a Job object from a jobcontrol file.

get_active_jobs()
Returns list of jobs that are not completed.

get_jobs_by_programs(program_names)
Find jobs with specific Program attributes.

_get_jobs(username='', query='all')
Return a list of Jobs in the database.

_get_job_selection(username='')
Return a list of jobs from the current mmjob query selection.

_launch_qapp()
Launch QCoreApplication so mmjob can run async calls under the hood.

launch_job(cmd, print_output=False, expandvars=True, launch_dir=None)
Run a process under job control and return a Job object.

fix_cmd(cmd, expandvars=True)
A function to clean up the command passed to launch_job.

_fix_launch_cmd(cmd)
Split the string command into program and arguments, then fix the program path if necessary.

_fix_program_path(prog)
If the program executable doesn't exist as a file and isn't an absolute path, prepend the SCHRODINGER directory if it exists there.

list2jmonitorcmdline(cmdlist)
Turn a command in list form to a single string that can be executed by jmonitor.

_get_job_spec_launch_command(job_spec, launch_parameters)
Get the launch command for a job based on its specification.

launch_from_job_spec(job_spec, launch_parameters)
Launch a job based on its specification.

get_backend()
A convenience function to see if we're running under job control.

get_runtime_path(pathname)
Return the runtime path for the input file 'pathname'.

under_job_control()
Returns True if this process is running under job control; False otherwise.

get_hostfile()
Return the name of the schrodinger.hosts file last used by get_hosts().

hostfile_is_empty(host_filepath)
Return if the given host_filepath host is empty, meaning it contains only the localhost entry.

get_installed_hostfiles(root_dir='')
Return the pathname for the schrodinger.hosts file installed in the most recent previous installation directory we can find.

get_hosts()
Return a list of all Hosts in the schrodinger.hosts file.

_temp_hostfile(*args, **kwds)

tuple

hostfile_is_valid(fname)
Returns: a (bool, str) tuple indicating whether the host file is valid

to_hostfile(hosts)
Return a string representation for the given list of Host objects that would be suitable for use as a hosts file.

sorted_hosts(hosts)
Return a list of the given Host objects re-ordered to satisfy the conditions that, 1.

get_host(name)
Return a Host object for the named host.

_get_num_hosts()
Returns the number of hosts in the schrodinger.hosts file.

_get_host(name)
Return a Host object for 'name', with all attributes read in from the corresponding entry in schrodinger.hosts file.

host_str_to_list(hosts_str)
Convert a hosts string (Ex: "galina:1 monica:4") to a list of tuples.

host_list_to_str(host_list)
Converts a hosts list (Ev: [ ('host1',1), ('host2', 10) ] ) to a string.

get_command_line_host_list()
Return a list of (host, ncpu) tuples corresponding to the host list that is specified on the command line.

get_backend_host_list()
Return a list of (host, ncpu) tuples corresponding to the host list as determined from the SCHRODINGER_NODEFILE.

calculate_njobs(host_list=None)
Derive the number of jobs from the specified host list.

is_valid_hostip(hostip)
Checks if it is a valid ip address.

is_valid_hostname(hostname)
Checks if the hostname is valid.

Variables

[hide private]

__doc__ = ...

_version = '$Revision: 1.105 $'

logger = logging.getLogger("schrodinger.jobcontrol")

profiling = 0
hash(x)

jobid_re = re.compile(r'(?im)^JobId:\s+(\S+)')

timestamp_format = '%Y-%m-%d-%H:%M:%S'

entry_fields = ['name', 'base', 'host', 'nodelist', 'user', 'q...

field_sortkey = {'accountcodes': 22, 'base': 1, 'env': 13, 'gp...

INSTALL_ROOT = '/opt/schrodinger'

HOSTS_FILE = 'schrodinger.hosts'

qapp = None
hash(x)

_job_manager = _JobManager()

_backend_singleton = None
hash(x)

_backend_hosts = []

__package__ = 'schrodinger.job'

k = 'accountcodes'

v = 22

Function Details

[hide private]

get_active_jobs()

Returns list of jobs that are not completed.

Returns:: list of Job objects

get_jobs_by_programs(program_names)

Find jobs with specific Program attributes.

Parameters:

program_names (iterable of str) - list of matching Program (s)

Returns:

list of Job objects

_get_jobs(username=`''`, query=`'all'`)

Return a list of Jobs in the database.

The query can be "all" or "active"; otherwise, raises an Exception.

_launch_qapp()

Launch QCoreApplication so mmjob can run async calls under the hood. If we are calling from maestro or other PyQt4 GUI, this should already be instantiated, so don't put this somewhere that will get called on import.

launch_job(cmd, print_output=False, expandvars=True, launch_dir=None)

Run a process under job control and return a Job object. For a process to be under job control, it must print a valid JobId: line to stdout. If such a line isn't printed, a RuntimeError will be raised.

The cmd argument should be a list of command arguments (including the executable) as expected by the subprocess module.

If the executable is present in $SCHRODINGER or $SCHRODINGER/utilities, an absolute path does not need to be specified.

NOTE: UI events will be processed while jlaunch is executing.

Parameters:

print_output (bool) - Determines if the output from jlaunch is printed to the terminal or not. Output will be logged (to stderr by default) if Python or JobControl debugging is turned on or if there is a launch failure, even if 'print_output' is False.
expandvars (bool) - If True, any environment variables of the form $var or ${var} will be expanded with their values by the os.path.expandvars function.
lauch_dir - Launch the job from the specified directory. Will CD into this directory before launching the job, and CD out of it before running the event loop while jlaunch is running.
launch_dir (str)

Raises:

RuntimeError - If there is a problem launching the job (e.g., no JobId gets printed). If running within Maestro, an error dialog will first be shown to the user.

fix_cmd(cmd, expandvars=True)

A function to clean up the command passed to launch_job.

Parameters:

cmd (list of strings) - A command in a form that can be passed to subprocess.Popen.
expandvars (bool) - If True, any environment variables of the form $var or ${var} will be expanded with their values by the os.path.expandvars function.

Returns:

The command to be launched

_fix_program_path(prog)

If the program executable doesn't exist as a file and isn't an absolute path, prepend the SCHRODINGER directory if it exists there.

Return a list of command line arguments so this function can be used from launch_job when 'cmd' is a list.

_get_job_spec_launch_command(job_spec, launch_parameters)

Get the launch command for a job based on its specification.

Parameters:

job_spec (schrodinger.job.launchapi.JobSpecification) - Data defining the job.

Returns:

Command to execute as a list of strings.

launch_from_job_spec(job_spec, launch_parameters)

Launch a job based on its specification.

Parameters:

job_spec (schrodinger.job.launchapi.JobSpecification) - Data defining the job.

Returns:

A schrodinger.job.jobcontrol.Job object.

get_backend()

A convenience function to see if we're running under job control. If so, return a _Backend object. Otherwise, return None.

get_runtime_path(pathname)

Return the runtime path for the input file 'pathname'.

If the pathname is of a type that job control will not copy to the job directory or no runtime file can be found, returns the original path name.

get_hostfile()

Return the name of the schrodinger.hosts file last used by get_hosts(). The file is found using the standard search path ($SCHRODINGER_HOSTS, local dir, $HOME/.schrodinger, $SCHRODINGER).

hostfile_is_empty(host_filepath)

Return if the given host_filepath host is empty, meaning it contains only the localhost entry. If the host_filepath str is empty or invalid, then this function will raise an invalid path exception - IOError.

Parameters:

host_filepath (str) - schrodinger.hosts file to use.

Returns:

bool

get_installed_hostfiles(root_dir=`''`)

Return the pathname for the schrodinger.hosts file installed in the most recent previous installation directory we can find.

If a root pathname is passed in, previous installations are searched for there. Otherwise, we look in the standard install locations.

get_hosts()

Return a list of all Hosts in the schrodinger.hosts file. After this is called, get_hostfile() will return the pathname for the schrodinger.hosts file that was used. Raises UnreadableHostsFileException or MissingHostsFileException on error.

_temp_hostfile(*args, **kwds)

Decorators:

@contextlib.contextmanager

hostfile_is_valid(fname)

Parameters:

fname (str) - The full path of the host file to validate

Returns: tuple

a (bool, str) tuple indicating whether the host file is valid

sorted_hosts(hosts)


Return a list of the given Host objects re-ordered to satisfy the
conditions that,
1. the localhost entry (if any) is first, and
2. any entry used as a base entry is defined before it's used.
Otherwise, the input order is preserved.

get_host(name)

Return a Host object for the named host. If the host is not found, we return a Host object with the provided name and details that match localhost. This matches behavior that jobcontrol uses. Raises UnreadableHostsFileException or MissingHostsFileException on error.

_get_num_hosts()

Returns the number of hosts in the schrodinger.hosts file. Will raise UnreadableHostsFileException or MissingHostsFileException on error.

_get_host(name)

Return a Host object for 'name', with all attributes read in from the corresponding entry in schrodinger.hosts file. Requires that mmjob be initialized. Raises UnreadableHostsFileException or MissingHostsFileException on error.

host_str_to_list(hosts_str)

Convert a hosts string (Ex: "galina:1 monica:4") to a list of tuples. First value of each tuple is the host, second value is # of cpus.

host_list_to_str(host_list)

Converts a hosts list (Ev: [ ('host1',1), ('host2', 10) ] ) to a string. Output example: "host1:1,host2:10"

get_command_line_host_list()

Return a list of (host, ncpu) tuples corresponding to the host list that is specified on the command line.

This function is meant to be called by scripts that are running under a toplevel job control script but are not running under jlaunch.

The host list is determined from the following sources:

SCHRODINGER_NODELIST
JOBHOST (if only a single host is specified)
"localhost" (if no host is specified)

If no SCHRODINGER_NODELIST is present in the environment, None is returned.

get_backend_host_list()

Return a list of (host, ncpu) tuples corresponding to the host list as determined from the SCHRODINGER_NODEFILE.

This function is meant to be called from scripts that are running under jlaunch (i.e. backend scripts).

Returns None if SCHRODINGER_NODEFILE is not present in the environment.

calculate_njobs(host_list=None)


Derive the number of jobs from the specified host list.
This function is useful to determine number of subjobs if user didn't
specified the '-NJOBS' option.

@type host_list: String
                 or
                 List of tuples
@param host_list: String of hosts along with optional number of cpus
                  or
                  List of tuples consist of hosts along with optional
                  number of cpus

If host list is not specified then it uses get_command_line_host_list() to
determine njobs, else uses the user provided host list.

Please note: the host_list can be passes as string or as a list of tuples.
    For example: for -GLIDE_HOST "host1 host2:4" host_list would become
        [('host1', None), ('host2', 4)]

is_valid_hostip(hostip)

Checks if it is a valid ip address. Logic comes from mmjob.cpp

Parameters:

hostip - host ip address
type - string

is_valid_hostname(hostname)

Checks if the hostname is valid.

Parameters:

hostname - host name
type - string

Variables Details

[hide private]

doc

Value:

"""
Core job control for python.

There are currently four major sections of this module - "Job database
,"
"Job launching," "Job backend," and "Job hosts." The job database sect
ion
deals with getting info about existing Jobs, the job launching section
...

entry_fields

Value:

['name',
 'base',
 'host',
 'nodelist',
 'user',
 'queue',
 'qargs',
 'schrodinger',
...

field_sortkey

Value:

{'accountcodes': 22,
 'base': 1,
 'env': 13,
 'gpgpu': 18,
 'host': 2,
 'maestrocontrols': 19,
 'memory': 21,
 'name': 0,
...

Module jobcontrol

get_active_jobs()

get_jobs_by_programs(program_names)

_get_jobs(username='', query='all')

_launch_qapp()

launch_job(cmd, print_output=False, expandvars=True, launch_dir=None)

fix_cmd(cmd, expandvars=True)

_fix_program_path(prog)

_get_job_spec_launch_command(job_spec, launch_parameters)

launch_from_job_spec(job_spec, launch_parameters)

get_backend()

get_runtime_path(pathname)

get_hostfile()

hostfile_is_empty(host_filepath)

get_installed_hostfiles(root_dir='')

get_hosts()

_temp_hostfile(*args, **kwds)

hostfile_is_valid(fname)

sorted_hosts(hosts)

get_host(name)

_get_num_hosts()

_get_host(name)

host_str_to_list(hosts_str)

host_list_to_str(host_list)

get_command_line_host_list()

get_backend_host_list()

calculate_njobs(host_list=None)

is_valid_hostip(hostip)

is_valid_hostname(hostname)

__doc__

entry_fields

field_sortkey

_get_jobs(username=`''`, query=`'all'`)

get_installed_hostfiles(root_dir=`''`)

doc