schrodinger.tasks.queue module¶

class schrodinger.tasks.queue.TaskQueue(*args, **kwargs)¶

Bases: schrodinger.tasks.tasks.SignalTask

A task that runs a queue of tasks. The TaskQueue is done when all its added tasks have completed, regardless of whether they completed successfully or failed. To use, add tasks with addTask and then start the task queue.

queuedTaskFinished¶

max_running_tasks¶

Base class for all Param classes. A Param is a descriptor for storing data, which means that a single Param instance will manage the data values for multiple instances of the class that owns it. Example:

class Coord(CompoundParam):
    x: int
    y: int

An instance of the Coord class can be created normally, and Params can be accessed as normal attributes:

coord = Coord()
coord.x = 4

When a Param value is set, the valueChanged signal is emitted. Params can be serialized and deserialized to and from JSON. Params can also be nested:

class Atom(CompoundParam):
    coord: Coord
    element: str

__init__(*args, **kwargs)¶: Initialize self. See help(type(self)) for accurate signature.

addTask(task)¶

setUpMain()¶

AUTO_TASKDIR = <object object>¶

CMDLINE = 1¶

DEFAULT_TASKDIR_SETTING = None¶

DONE = 3¶

DataClass¶: alias of builtins.object

FAILED = 2¶

GUI = 2¶

INTERRUPT_ENABLED = False¶

RUNNING = 1¶

TEMP_TASKDIR = <object object>¶

WAITING = 0¶

addFuncToGroup(func, group=None, order=None)¶

Adds a function to the specified chain. Typically used for adding functions that are not methods of this object.

The function may optionally be decorated with a FuncGroupMarker. If so, the default group and order will be determined by the decorator. Any group or order explicitly passed in to addFuncToGroup will take precedence over the decorator settings.

Parameters:	func – the function to add group (FuncGroupMarker or None) – the group marker. If the function is decorated with a FuncGoupMarker, that group marker will be the default. order (float or None) – the sorting order. If the function is decorated with a FuncGoupMarker, the order specified in the decorator will be the default.

addPostprocessor(func, order=0)¶

Adds a postproceessor function to this task instance. If the function has been decorated with @postprocessor, the order specified by the decorator will be used.

Parameters:	func (typing.Callable) – the function to add order (float) – the sorting order for the function relative to all other preprocessors. Takes precedence over order specified by the preprocessor decorator.

addPreprocessor(func, order=-2000)¶

Adds a preproceessor function to this task instance. If the function has been decorated with @preprocessor, the order specified by the decorator will be used as the default.

Parameters:	func – the function to add order (float) – the sorting order for the function relative to all other preprocessors. Takes precedence over order specified by the preprocessor decorator.

classmethod addSubParam(name, param, update_owner=True)¶

blockSignals(self, bool) → bool¶

block_signal_propagation()¶

childEvent(self, QChildEvent)¶

children(self) → List[QObject]¶

classmethod configureParam()¶: Override this class method to set up the abstract param class (e.g. setParamReference on child params.)

connectNotify(self, QMetaMethod)¶

customEvent(self, QEvent)¶

classmethod defaultValue(*args, **kwargs)¶

deleteLater(self)¶

destroyed¶: destroyed(self, object: QObject = None) [signal]

disconnect(self)¶

disconnectNotify(self, QMetaMethod)¶

dumpObjectInfo(self)¶

dumpObjectTree(self)¶

dynamicPropertyNames(self) → List[QByteArray]¶

event(self, QEvent) → bool¶

eventFilter(self, QObject, QEvent) → bool¶

findChild(self, type, name: str = '', options: Union[Qt.FindChildOptions, Qt.FindChildOption] = Qt.FindChildrenRecursively) → QObject¶: findChild(self, Tuple, name: str = ‘’, options: Union[Qt.FindChildOptions, Qt.FindChildOption] = Qt.FindChildrenRecursively) -> QObject

findChildren(self, type, name: str = '', options: Union[Qt.FindChildOptions, Qt.FindChildOption] = Qt.FindChildrenRecursively) → List[QObject]¶: findChildren(self, Tuple, name: str = ‘’, options: Union[Qt.FindChildOptions, Qt.FindChildOption] = Qt.FindChildrenRecursively) -> List[QObject] findChildren(self, type, QRegExp, options: Union[Qt.FindChildOptions, Qt.FindChildOption] = Qt.FindChildrenRecursively) -> List[QObject] findChildren(self, Tuple, QRegExp, options: Union[Qt.FindChildOptions, Qt.FindChildOption] = Qt.FindChildrenRecursively) -> List[QObject] findChildren(self, type, QRegularExpression, options: Union[Qt.FindChildOptions, Qt.FindChildOption] = Qt.FindChildrenRecursively) -> List[QObject] findChildren(self, Tuple, QRegularExpression, options: Union[Qt.FindChildOptions, Qt.FindChildOption] = Qt.FindChildrenRecursively) -> List[QObject]

classmethod fromJson(json_obj)¶

A factory method which constructs a new object from a given dict loaded from a json string or file.

Parameters:	json_obj (dict) – A json-loaded dictionary to create an object from.
Returns:	An instance of this class.

:rtype : cls

classmethod fromJsonFilename(filename)¶

classmethod fromJsonImplementation(json_dict)¶: Sets the value of this compound param value object from a JSON dict.

getAbstractParam(*args, **kwargs)¶

getAddedFuncs(group=None)¶

getFuncGroup(group=None)¶

Retrieve the functions belonging to the specified group.

Parameters:	group (FuncGroupMarker) – the group marker
Returns:	the functions in the specified group, in order
Return type:	list

classmethod getJsonBlacklist()¶

Override to customize what params are serialized.

Implementations should return a list of abstract params that should be omitted from serialization.

..NOTE: Returned abstract params must be direct child params of cls, e.g. cls.name, not cls.coord.x.

classmethod getParamSignal(*args, **kwargs)¶

classmethod getParamValue(*args, **kwargs)¶

classmethod getSubParam(name)¶

Get the value of a subparam using the string name:

c = Coord()
assert c.getSubParam('x') == 0

Note

Using the string name to accss params is generally discouraged, but can be useful for serializing/deserializing param data.

Parameters:	name (str) – The name of the subparam to get the value for.

classmethod getSubParams()¶: Return a dictionary mapping subparam names to their values.

getTaskDir()¶: Returns the full path of the task directory. This is only available if the task directory exists (after creation of the taskdir or, if no task dir is specified, any time).

getTaskFilename(fname)¶: Return the appropriate absolute path for an input or output file in the taskdir.

get_version()¶: Method to get the version of a particular object. Defaults to the current version of mmshare. This class can be overridden for custom versioning behavior.

guard()¶: Context manager that saves any Exception raised inside

static guard_method(func)¶

inherits(self, str) → bool¶

initAbstract()¶

initConcrete()¶: Override to customize initialization of concrete params.

initializeValue()¶: @overrides: parameters.CompoundParam

input¶

inputChanged¶

inputReplaced¶

installEventFilter(self, QObject)¶

classmethod isAbstract()¶: Whether the param is an “abstract” param.

isDefault(*args, **kwargs)¶

isInterruptionRequested()¶

isRunning()¶

isSignalConnected(self, QMetaMethod) → bool¶

isStartable()¶

isWidgetType(self) → bool¶

isWindowType(self) → bool¶

kill()¶

Implementations are responsible for immediately stopping the task. No threads or processes should be running after this method is complete.

This method should be called sparingly since in many contexts the task will be forced to terminate without a chance to clean up or free resources.

killTimer(self, int)¶

mainDone¶

max_progress¶

max_progressChanged¶

max_progressReplaced¶

max_running_tasksChanged¶

max_running_tasksReplaced¶

metaObject(self) → QMetaObject¶

moveToThread(self, QThread)¶

name¶

nameChanged¶

nameReplaced¶

objectName(self) → str¶

objectNameChanged¶: objectNameChanged(self, str) [signal]

output¶

outputChanged¶

outputReplaced¶

classmethod owner()¶

Get the owner of the param:

# Can be called on an abstract param:
assert Coord.x.owner() == Coord

# ...or on an instance of a CompoundParam
a = Atom()
assert a.coord.owner() == a

classmethod ownerChain()¶

Returns a list of param owners starting from the toplevel param and ending with self. Examples:

foo.bar.atom.coord.ownerChain() will return [foo, bar, atom, coord] where every item is a concrete param.

Foo.bar.atom.coord.x.ownerChain() will return [Foo, Foo.bar, Foo.atom.coord, Foo.atom.coord.x] where every item is an abstract params.

classmethod paramName()¶

Get the name of the param:

# Can be called on an abstract param:
print(Coord.x.paramName()) # 'x'

# ...or on an instance of a CompoundParam
a = Atom()
a.coord.paramName() # 'coord'

parent(self) → QObject¶

postprocessors()¶

Returns:	A list of postprocessors, both decorated methods on the task and external functions that have been added via `addPostprocessor()`
Return type:	list[typing.Callable]

preprocessors()¶

Returns:	A list of preprocessors (both decorated methods on the task and external functions that have been added via addPreprocessor)

processFuncChain(chain=None, result_callback=None)¶

Execute each function in the specified chain sequentially in order.

The result_callback is called after each function with the return value of that function. This can be used to respond to the return value (e.g. present information to the user, get user feedback, log the result, etc.)

The return value of the result_callback determines whether processing will proceeed to the next function.

Parameters:	chain (FuncChainDecorator) – which chain to process result_callback – the callback that will get called with the result of each function in the chain
Returns:	a list of the results from the functions

progress¶

progressChanged¶

progressReplaced¶

property(self, str) → Any¶

pyqtConfigure(...)¶: Each keyword argument is either the name of a Qt property or a Qt signal. For properties the property is set to the given value which should be of an appropriate type. For signals the signal is connected to the given value which should be a callable.

receivers(self, PYQT_SIGNAL) → int¶

removeEventFilter(self, QObject)¶

replicate()¶: Create a new task with the same input and settings (but no output)

requestInterruption()¶

Request the task to stop.

To enable this feature, subclasses should periodically check whether an interruption has been requested and terminate if it has been. If such logic has been included, INTERRUPT_ENABLED should be set to True.

reset(*args, **kwargs)¶

run()¶

classmethod runFromCmdLine()¶

runPostprocessing(callback=None)¶

runPreprocessing(callback=None, calling_context=None)¶

Run the preprocessors one-by-one. By default, any failing preprocessor will raise a TaskFailure exception and terminate processing. This behavior may be customized by supplying a callback function which will be called after each preprocessor with the result of that preprocessor.

Parameters:

callback – a function that takes result and returns a bool that indicates whether to continue on to the next preprocessor
calling_context – specify a value here to indicate the context in which this preprocessing is being called. This value will be stored in an instance variable, self.calling_context, which can be accessed from any preprocessor method on this task. Typically this value will be either self.GUI, self.CMDLINE, or None, but any value may be supplied here and checked for in the preprocessor methods. self.calling_context always reverts back to None at the end of runPreprocessing.

sender(self) → QObject¶

senderSignalIndex(self) → int¶

setObjectName(self, str)¶

classmethod setParamValue(*args, **kwargs)¶

setParent(self, QObject)¶

setProperty(self, str, Any) → bool¶

classmethod setReference(param1, param2)¶

Call this class method from configureParam to indicate that two params should be kept in sync. The initial values will start with the default value of param1. Example:

class Square(CompoundParam):
    width: float = 5
    height: float = 10

    @classmethod
    def configureParam(cls):
        super().configureParam()
        cls.setReference(cls.width, cls.height)

square = Square()
assert square.width == square.height == 5 # Default value of width
                                          # takes priority
square.height = 7
assert square.width == square.height == 7
square.width = 6
assert square.width == square.height == 6

Parameters:	param1 – The first abstract param to keep synced param2 – The second abstract param. After instantiation, this param will take on the value of param1.

setValue(*args, **kwargs)¶

signalsBlocked(self) → bool¶

skip_eq_check()¶

specifyTaskDir(taskdir_spec)¶

Specify the taskdir creation behavior. Use one of the following options:

A directory name (string). This may be a relative or absolute path

None - no taskdir is requested. The task will use the CWD as its taskdir

AUTO_TASKDIR - a new subdirectory will be created in the CWD using the task name as the directory name.

TEMP_TASKDIR - a temporary directory will be created in the schrodinger temp dir. This directory is cleaned up when the task is deleted.

Parameters:	taskdir_spec – one of the four options listed above

start(skip_preprocessing=False)¶

This is the main method for starting a task. Start will check if a task is not already running, run preprocessing, and then run the task.

Failures in preprocessing will interrupt the task start, and the task will never enter the RUNNING state.

Parameters:	skip_preprocessing (bool) – whether to skip preprocessing. This can be useful if preprocessing was already performed prior to calling start.

startTimer(self, int, timerType: Qt.TimerType = Qt.CoarseTimer) → int¶

staticMetaObject = <PyQt5.QtCore.QMetaObject object>¶

status¶

statusChanged¶

statusReplaced¶

taskDirSetting()¶: Returns the taskdir spec. See specifyTaskDir() for details.

taskDone¶

taskFailed¶

taskStarted¶

thread(self) → QThread¶

timerEvent(self, QTimerEvent)¶

toDict(*args, **kwargs)¶

toJson(_mark_version=True)¶

Create and returns a data structure made up of jsonable items.

Return type:	An instance of one the classes from NATIVE_JSON_DATATYPES

toJsonImplementation(*args, **kwargs)¶

tr(self, str, disambiguation: str = None, n: int = -1) → str¶

valueChanged¶

wait(timeout=None)¶

Block until the task is finished executing or timeout seconds have passed.

Parameters:	timeout (NoneType or int) – Amount of time in seconds to wait before timing out. If None or a negative number, this method will wait until the task is finished.

class schrodinger.tasks.queue.TaskDJ(hosts=None, local=False, max_retries=None, default_max_retries=0, max_failures=None, verbosity='quiet', job_class=<class 'schrodinger.job.queue.JobControlJob'>, update_delay=None)¶

Bases: schrodinger.job.queue.JobDJ

A subclass of JobDJ that supports running tasks.

addTask(task)¶

updatedTasks()¶: Return a generator which yields a task every time its status changes. This could be a change from WAITING to RUNNING, so it’s up to the caller to check what the new status is rather than assume the task is finished.

__init__(hosts=None, local=False, max_retries=None, default_max_retries=0, max_failures=None, verbosity='quiet', job_class=<class 'schrodinger.job.queue.JobControlJob'>, update_delay=None)¶

Constructor.

Parameters:

hosts (A list of (<hostname>, <available cpus>) tuples, where <hostname> is a string and <available cpus> is an integer.) – A host list specification. To convert a string representation from -HOST to a list, use the jobcontrol.host_str_to_list() function. If no hosts argument is specified, the hosts will be read from the job control SCHRODINGER_NODEFILE environment variable.
local (bool) – If True, then all jobs will run with -LOCAL. Default is False.
max_retries (int) – Number of allowed retries per subjob. If this is set, it is never overridden by the SCHRODINGER_MAX_RETRIES environment variable. If it is not set, the value in default_max_retries is used, and SCHRODINGER_MAX_RETRIES is allowed to override. If you wish to disable restarting altogether, set this value to zero.
default_max_retries (int) – Number of allowed retries per subjob. This value can always be overridden by the SCHRODINGER_MAX_RETRIES environment variable. Default is zero.
max_failures (int) – Total number of allowed subjob failures before JobDJ exits. If it is not defined, a default of zero will be used (exit on any failure after attempting to restart), but this can be overridden with the SCHRODINGER_MAX_FAILURES environment variable. To allow an unlimited number of subjob failures, set max_failures to the module level NOLIMIT constant.
verbosity (str) – There are three allowed verbosity levels: “quiet” - only warnings and errors are printed; “normal” - JobDJ progress is printed; and “verbose” - additional debugging info is printed. Default is “quiet”.
job_class (BaseJob subclass) – The class to use as the default job constructor when the addJob argument is not a BaseJob instance.
update_delay (int) – The number of seconds to wait between job control database reads for JobControlJob jobs. (This delay is for an individual job, not for any job database read.) Default is None, which causes the module level constant UPDATE_DELAY to be used.

active_jobs¶

addJob(job, add_connected=True, **kwargs)¶

Add a job to run. If job is not a BaseJob instance, a BaseJob instance is constructed with job as the first argument. The default BaseJob class for the JobDJ instance can be specified in the constructor.

Additional keyword arguments are passed on to the Job constructor.

All job prerequisites and dependencies need to be specified before adding a job to JobDJ.

Parameters:	add_connected (bool) – If True, for jobs with dependencies only one job per connected group should be added and all connected jobs will be discovered and added automatically. If False, it is the user’s responsibility to make sure that any prerequisites of a job are also added.

all_jobs¶

disableSmartDistribution()¶

Disable smart distribution of jobs.

Smart distribution allows subjobs to run on the machine that JobDJ is running on when JobDJ itself is running under a queuing system. This is usually desirable since the JobDJ process doesn’t generally consume significant computational resources and you don’t want to leave a queue slot mostly idle.

done_jobs¶: Successfully completed jobs, sorted into the order they were marked as completed by JobDJ.

dump(filename)¶: Pickle the JobDJ instance to the specified file name.

failed_jobs¶

getActiveProcCounts()¶: Return a dictionary containing the number of active jobs on each host.

hasStarted()¶: Returns True if JobDJ has started already

isComplete()¶: Returns True if JobDJ has completed, False otherwise.

killJobs()¶: Kill all active jobs

markForRestart(job, action)¶

Mark a job as dead, but make sure that it gets restarted.

Parameters:	action – Describes the reason the job is being restarted.

printStatus(job=None, action=None)¶

Prints the status of JobDJ and the action/status for the job.

If no job is specified, prints the status header.

If no action is specified, the status_string attribute of the job is used.

run(status_change_callback=None, periodic_callback=None, callback_interval=300, restart_failed=True)¶

Call this method to run all jobs that have been added. The method will return control when all jobs have completed.

Parameters:

status_change_callback – A command to call every time the status changes. An alternative approach is to use the updatedJobs generator, which makes it easy to determine which job changed its state.
periodic_callback (callable) – A command to call periodically, regardless of whether job status has changed or not. The function will be called without any arguments.
callback_interval (int) – The interval at which the periodic interval will be called. This time is only approximately enforced and will depend on the timing delay settings (e.g. MONITOR_DELAY).
restart_failed (bool) – True (default) if previously failed jobs should be restarted, False if not.

setHostList(host_list)¶

Tell JobDJ to use the specified host list.

Repeated hostnames will combine the available cpus and keep the list position of the first occurrence.

Active jobs are not affected by a change in the host list.

Parameters:	host_list (A list of (<hostname>, <available cpus>) tuples, where <hostname> is a string and <available cpus> is an integer.) – A host list specification.

total_active¶: The number of jobs currently running.

total_added¶: The number of individual jobs that have been added to the JobDJ instance.

total_failed¶: The number of jobs that have failed.

total_finished¶: The number of jobs that have finished successfully.

updatedJobs(periodic_callback=None, callback_interval=300, restart_failed=True)¶

A generator that starts job distribution and yields each job as its status changes. A status change occurs when a job is started, when it finishes, or fails. The state property of the yielded job can be examined to determine its current state.

Use as:

for job in jobdj.updatedJobs():
    if job.state == DONE and isinstance(job, JobControlJob):
        print "%s is done." % job.getJob().JobId

Parameters:	restart_failed (bool) – True (default) if previously failed jobs should be restarted, False if not.

waiting_jobs¶: Jobs waiting to be started.

Previous topic

Next topic

schrodinger.tasks.queue module¶