Class App
A class to handle all aspects of job setup and running. Application
scripts based on a sub-class of App will be able to run from the command
line in top-level mode (e.g., starting jobs with '-HOST <host>' or
'-i <inputfile>' args) or restart mode (using the pickled instance
file instead of the input file), run under Job Control by using Launcher
for startup functionality, and as a backend (using '-NOJOBID'). A user
only needs to write one script to be able to run a job with full Job
Control features. Optionally, the script can be used to launch a GUI in
a standard fashion.
Non-toplevel, but JC-type options like -SAVE and -LOCAL are supported
by App, so a user doesn't need to make special arrangements.
The general approach for using App is for a script author to create a
derived class that overrides several key methods that define the
particular behaviors of the script -- commandLine() for turning
command-line arguments and input parameters into instance attributes,
backend() for doing the actual work of the job, and optionally gui() for
enabling a graphical interface.
The __init__ method determines the mode in which the App will run from
the arguments passed to it. Therefore, all that is necessary in the
"if __name__ == '__main__'" block of the script is to create
the App instance, passing it __file__ (i.e., the script name) and the
command-line arguments. If the script is runnable via a GUI from
Maestro, the will be an additional function needed that creates the App
instance with "-gui" as the only argument.
Note that there is no 'run' method; the App machinery starts working
as soon as the instance is created.
The App instance is saved as a pickled file prior to submission to Job
Control. This pickled file (i.e., a 'restart' file) is transferred to
the JobDir/Compute Host and then read prior to running backend(). It
also is possible for a user to start a job from a restart file via the
'-RESTART' option. If the backend() method updates the instance with the
current state of the job, bases its actions on the recorded state, and
calls the dumpBE() method at critical checkpoints (so the .restart file
is written and registered as an output file with Job Control), the
restart mechanism can be used to resume interrupted jobs.
For restartability of some types of jobs, it is necessary to preserve
the contents of the scratch directory (e.g., when backend() is a driver
for other subjobs, and not all of the subjob input/output files are
registered as input/output files of the driver). Preserving the JobDir
can be accomplished by running with -LOCAL (which avoids the use of a
scratch directory).
If a job is ran without -LOCAL (using scratch directory), and it
fails; then job control will compress the scratch dir and copy it to the
launch dir.
FIXME: the mechanism is not yet in place to restart a job from a
compressed scratch directory (.zip) file.
|
|
__init__(self,
script,
args=[])
When an App instance is constructed and __init__ is run, App
determines from the 'args' list whether... |
|
|
|
|
run(self)
If '-gui' passed to the constructor, runs the GUI OR... |
|
|
|
|
commandLine(self,
args)
Overwrite this method with code that parses the arguments and
initializes the instance variables to appropriate values. |
|
|
|
|
gui(self)
Overwrite this method with your GUI code. |
|
|
|
|
backend(self)
Overwrite this function with your backend (main) code. |
|
|
|
|
dumpFE(self,
relpath='.')
Dumps the App instance to file named '<jobname>.restart'. |
|
|
|
|
dumpBE(self,
relpath='.')
Dumps the App instance to file named '<jobname>.restart'. |
|
|
|
|
setJobName(self,
jobname)
Call this method from commandLine() to set the jobname. |
|
|
|
|
getJobName(self)
Returns the jobname of the App. |
|
|
|
|
setProgramName(self,
progname)
Call this method from commandLine() to set the program name. |
|
|
|
|
getProgramName(self)
Returns the program name of the App. |
|
|
|
|
getHostStr(self)
What user specified as -HOST when running the script. |
|
|
|
|
alwaysLocal(self)
Call this method from the commandLine() method to make the backend
always run in the local (i.e., launch) directory instead of in a
scratch directory. |
|
|
|
|
useLocalDriver(self)
The backend will be run on the local host, regardless of the -HOST
setting. |
|
|
|
|
addInputFile(self,
file)
Call from commandLine() to specify job input files. |
|
|
|
|
addOutputFileFE(self,
file)
Call from commandLine() to specify output files. |
|
|
|
|
addOutputFileBE(self,
file)
Call from backend() to specify output files. |
|
|
|
|
setStructureOutputFileFE(self,
file)
Call from commandLine() to specify the structure output file (i.e.,
the file that gets incorporated into Maestro upon job completion). |
|
|
|
|
setStructureOutputFileBE(self,
file)
Call from backend() to specify the structure output file (i.e., the
file that gets incorporated into Maestro upon job completion). |
|
|
|
|
addLogFile(self,
file)
Call from commandLine() to specify additional log files. |
|
|
|
|
log(self,
*args)
Use this method to print from within backend(). |
|
|
|
|
lognotret(self,
*args)
Use this method to print from within backend(). |
|
|
|
jobcontrol.Job object
|
launchBackend(self,
args=[])
Call this method from the gui() method to run the job. |
|
|
|
|
addCommandLineOptions(self,
parser,
distributed=False,
use_group=False)
Add top-level and App options to the OptionParser 'parser'. |
|
|
|
|
|
|
|
_printJobInfo(self)
This method is called right before calling backend(). |
|
|
|
|
_readDumpFile(self,
file)
Returns an instance of App loaded from the supplied dump file. |
|
|
|
|
_startup(self,
args,
scratch_file,
local,
wait,
run_under_jobcontrol)
Run the job. |
|
|
__init__(self,
script,
args=[])
(Constructor)
|
|
When an App instance is constructed and __init__ is run, App
determines from the 'args' list whether...
-
The GUI should be launched ('-gui'), in which case the gui() method
is called.
-
A restart file should be read ('-RESTART') or job parameters should
be determined from command-line arguments.
-
The backend should be launched directly (-NOJOBID) or under Job
Control. This is handled by the _startup() method.
GUI/startup/backend execution doesn't occur until the 'run' method is
called.
Another option ('--scratch') is for a previously-generated scratch
directory file to be used during a restart job. This file is generated
by jobcontrol when the job fails.
The _startup() method also handles the -LOCAL argument.
'script' is the name of the user application script (often passed in
via the builtin '__file__').
|
|
If '-gui' passed to the constructor, runs the GUI OR...
Reads the restart file (-RESTART, existing job) or command-line
arguments (default, new job) and then runs in startup mode (default) or
backend mode (-NOJOBID).
Raises a SyntaxError if the commandLine() method returns no
arguments.
|
|
Overwrite this method with code that parses the arguments and
initializes the instance variables to appropriate values. This method
should also set the job name and input/output files for the
scriptlauncher via dedicated App methods. The method should return a
revised list of arguments (e.g., removing arguments that won't be needed
when running the backend on the restart file). Output files also can be
registered in the backend() method.
|
Overwrite this method with your GUI code. When a job is ready to be
submited, call 'self.launchBackend()' with appropriate arguments.
'launchBackend' then will run the rerun the script in non-GUI mode, so
the argument list should be like a command-line invocation.
Ex: self.launchBackend(['-i', input_file, '-j', 'testjob',
'-HOST', hostname, '-USER', myname])
|
|
Overwrite this function with your backend (main) code. This method
also may set output files via dedicated App methods for jobs that are
running under Job Control. To enable restartability at intermediate
stages of the job, record the state of the job in an instance variable,
have the method base its actions on the current state, and periodically
save the instance via the dumpBE() method.
|
dumpFE(self,
relpath='.')
|
|
Dumps the App instance to file named '<jobname>.restart'. This
version is called on the front end to create the initial restart file
during job submission. This is the file from which a user would resume an
interrupted job via a command like...
<myapp>.py -RESTART
When the user runs the application with '-RESTART' option, the App
instance is recovered from the dump file instead of initializing the
instance via the commandLine() method.
If the job was submitted with -LOCAL, then the restart file will be in
the launch directory, otherwise restart file will be in the
<jobname>.zip archive that is created by jobcontrol.
A 'relpath' optional argument is available, in case the restart file
needs to be written somewhere other than the CWD (e.g., in case the
backend is running in a workdir).
|
dumpBE(self,
relpath='.')
|
|
Dumps the App instance to file named '<jobname>.restart'. This
function is to be called by backend() at restartable places in the code.
This is the file from which a user would resume an interrupted job via a
command like...
<myapp>.py -RESTART
When the user runs the application with '-RESTART' option, the App
instance is recovered from the dump file instead of initializing the
instance via the commandLine() method.
If the job was submitted with -LOCAL, then the restart file will be in
the launch directory, otherwise restart file will be in the
<jobname>.zip archive that is created by jobcontrol.
A 'relpath' optional argument is available, in case the restart file
needs to be written somewhere other than the CWD (e.g., in case the
backend is running in a workdir).
|
What user specified as -HOST when running the script. Use in
backend() when running jobs with JobDJ. Should be in format...
"host1:ncpu1 host2:ncpu2"
NOTE: NOT COMPLETE YET. ALSO, THIS INFORMATION IS AVAILABLE TO JOBDJ
VIA THE SCHRODINGER_NODEFILE.
|
|
Call this method from the commandLine() method to make the backend
always run in the local (i.e., launch) directory instead of in a scratch
directory. This enforces -LOCAL without the user having to use that
command-line option. Running in the local directory removes the need to
use addInputFile(), addOutputFile*(), etc., but it imposes a requirement
that the local directory be directly accessible to any machine where the
backend will run, which may or may not be true for remote jobs. Running
locally also can facilitate restarting if the backend is a driver for
other subjobs.
|
|
The backend will be run on the local host, regardless of the -HOST
setting. This is so the -HOST option can be used to set the subjob host
info, while letting the driver/backend run locally.
|
|
Call from commandLine() to specify additional log files. The main log
file (i.e., the stdout/stderr of the backend() is registered
automatically.
|
|
Use this method to print from within backend(). Unlike log(), it does
not print a carriage return at end of logged text.
|
launchBackend(self,
args=[])
|
|
Call this method from the gui() method to run the job. Returns the
Job object of the launched job. In order to set up the appropriate
environment variables, a top-level script must be called. Therefore,
launchBackend() will invoke the script in command-line mode, with all
arguments passed to it. The 'args' should be a list of command-line
arguments (e.g., ['-i', input_file, '-HOST', hostname]).
- Returns: jobcontrol.Job object
- The job object for the launched backend job.
- Raises:
RuntimeError - If there is a problem launching the job (e.g., no JobId gets
printed). If running within Maestro, an error dialog will first be
shown to the user.
|
addCommandLineOptions(self,
parser,
distributed=False,
use_group=False)
|
|
Add top-level and App options to the OptionParser 'parser'. With this
method, the application-specific parser can include top-level and App
options in its usage/help statement, though the actual parsing of those
arguments is handled elsewhere. The 'distributed' boolean option
(default is False) alters the option descriptions slightly, making
references to the job as the "driver job". If 'use_group' is
True (default is False), the options will be added as an OptionGroup.
|
_extractArguments(self,
args)
|
|
Extract the App-specific command-line arguments. These control
whether the job will bypass Job Control ('-NOJOBID), whether a restart
('-RESTART) or scratch ('--scratch <file>') will be used, whether
the job will run in the local directory ('-LOCAL'), and whether the
scratch directory, if used, should be cleaned up ('-SAVE'). -NOJOBID,
-LOCAL, and -SAVE are JC-like options that must be handled by startup
scripts (i.e., here) instead of the standard top-level script. The
-LOCAL and -SAVE arguments are kept in the argument list in case the
application wants to use that information (e.g., to propagate -LOCAL to
subjobs). Returns 1) list of unused and retained args, 2) name of the
restart file, 3) name of the scratch file, 4) whether -LOCAL is used, 5)
whether -SAVE is used, and 6) whether -NOJOBID is used.
|
|
This method is called right before calling backend(). It prints job
information to the terminal/log file.
|
_readDumpFile(self,
file)
|
|
Returns an instance of App loaded from the supplied dump file. Raises
a RuntimeError if the specified file doesn't exist or if there is a
problem reading it.
|
_startup(self,
args,
scratch_file,
local,
wait,
run_under_jobcontrol)
|
|
Run the job. Submits the script under Job Control to be run from the
dump/restart file. If -NOJOBID is used, backend() is called directly
(i.e., no Job Control). 'schrodinger.job.launcher' handles setting all
the jlaunch arguments and issuing the jlaunch command. Returns the Job
object if running under Job Control (or None, otherwise).
|