Manage jobs

Since Lab Management 2.4, background processes (crons) are run by a job manager. The manager ensures that a job isn't run if another instance is running, kills off long-running or stuck jobs, records statistics about the jobs, and more.

The bin/job_manager.py script runs jobs by their labels (section headers) as defined in the site.conf file.

Fundamental job definitions are picked up from installation-support/default-conf/cubit/cubit-runtime.conf whenever recreate-runtime.py is run. Unless there's a specific reason - say you need to add a new cron job for root or Apache, or override an existing job definition - you're encouraged to NOT edit the definition in site.conf.

To define an additional cron job or adjust a current cron, define (or re-define) it in your site.conf file. See the inline documentation in installation-support/default-conf/cubit/cubit-runtime.conf for the structure of the job definition.
To prevent the job manager from managing a cron, you can add nomanage = true to that job definition. The job will just be added as a regular cron to the specified user's crontab.

To monitor the health of the Lab Management crons, you can "watch" the processes start and finish in real-time, list running jobs, review the current batch of statistics on jobs, or view the cumulative historical statistics. Use these commands:

# run 'watch' on the pid dirs:
./bin/job_manager.py --monitor
          
# list the currently running jobs:
./bin/job_manager.py --list
          
# display statistics for the current period (week to date):
./bin/job_manager.py --showstats
          
# display historical statistical data:
./bin/job_manager.py --showsummary