Trees | Indices | Help |
|
---|
|
object --+ | JobDJ
Class for running commands/jobs in parallel under jobcontrol.
Create an instance of this class, add commands to run with .addJob(), and then call run(). Alternatively, use the the updatedJobs() iterator to get access to each job as it changes state.
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
Inherited from |
|
|||
waiting_jobs Jobs waiting to be started. |
|||
done_jobs Successfully completed jobs, sorted into the order they were marked as completed by JobDJ. |
|||
active_jobs | |||
failed_jobs | |||
all_jobs | |||
total_added The number of individual jobs that have been added to the JobDJ instance. |
|||
total_active The number of jobs currently running. |
|||
total_finished The number of jobs that have finished successfully. |
|||
total_failed The number of jobs that have failed. |
|||
_host_list | |||
|
Constructor.
|
Mark a job as dead, but make sure that it gets restarted.
|
Check to see if use of job control messaging is possible. It is necessary to get a backend object to communicate via jmonitor messages.
|
Add a job to run. If Additional keyword arguments are passed on to the Job constructor. All job prerequisites and dependencies need to be specified before
adding a job to
|
Disable smart distribution of jobs. Smart distribution allows subjobs to run on the machine that JobDJ is running on when JobDJ itself is running under a queuing system. This is usually desirable since the JobDJ process doesn't generally consume significant computational resources and you don't want to leave a queue slot mostly idle. |
Check to see if we should use smart distribution. Smart distribution runs jobs on the localhost when a host is consumed by the JobDJ driver. Smart distribution is used if:
In this situation, the preference is to always keep a single job running on localhost. If the JobDJ job has a HostEntry that is present in the host list, the maximum number of jobs for that host is decreased by one. This ensures a "smart" job distributor that runs jobs on the same host that the JobDJ driver is on, rather than leaving it mostly idle. |
Return the next available host name from the host list. Return None if all hosts are saturated.
|
Tell JobDJ to use the specified host list. Repeated hostnames will combine the available cpus and keep the list position of the first occurrence. Active jobs are not affected by a change in the host list.
|
Prints the status of If no job is specified, prints the status header. If no action is specified, the |
Perform startup activities for
|
Call this method to run all jobs that have been added. The method will return control when all jobs have completed.
|
A generator that starts job distribution and yields each job as its
status changes. A status change occurs when a job is started, when it
finishes, or fails. The Use as: for job in jobdj.updatedJobs(): if job.state == DONE and isinstance(job, JobControlJob): print "%s is done." % job.getJob().JobId |
Cancel a job in the 'submitted' state if possible and move it back into the job queue to be restarted. This method is only useful to call when a known non-queue host is available. |
A generator to monitor the running jobs and update their state. Yields a job every time it finishes or fails. |
A private generator that implements the guts of the updatedJobs generator. It yields a job every time its status changes, and also yields None at the end of each cycle so that the caller can gain control if desired. (Despite being private, the pipeline.stage module calls this generator directly. So, we can't assume that this function has been called from updateJobs.) The JobDJ._start method must be called before invoking this method.
|
Take some standard steps on successful completion of a job. Add the completed job to the finished list, call its finalize method, and look for new jobs from completed dependencies. |
A generator to start as many jobs as possible, given the current open hosts. Each job that is successfully started (or restarted) is yielded. The optional max_starts argument can be used to limit the number of jobs that will be started before this generator terminates.
|
|
waiting_jobsJobs waiting to be started.
|
done_jobsSuccessfully completed jobs, sorted into the order they were marked as completed by JobDJ.
|
active_jobs
|
failed_jobs
|
all_jobs
|
total_addedThe number of individual jobs that have been added to the JobDJ instance.
|
total_activeThe number of jobs currently running.
|
total_finishedThe number of jobs that have finished successfully.
|
total_failedThe number of jobs that have failed.
|
_host_list
|
Trees | Indices | Help |
|
---|
Generated by Epydoc 3.0.1 on Wed Feb 3 06:05:21 2016 | http://epydoc.sourceforge.net |