diff options
| author | Matthijs van der Wild <matthijs.van-der-wild@durham.ac.uk> | 2025-07-08 16:12:51 -0400 |
|---|---|---|
| committer | Matthijs van der Wild <matthijs.van-der-wild@durham.ac.uk> | 2025-07-08 16:15:34 -0400 |
| commit | 05725c46cd33d72db60d5ea5954541f073163dec (patch) | |
| tree | 74330d42b88abf0a374755ee5689d4a89dace6ee | |
| parent | f247bf85b36ca38e820bfc066f36ac64c3ec9d5c (diff) | |
Expose the batching system
The queuing system was hard-coded to slurm. It may be, at times,
convenient to be able to run workflows on the local node (or use
a different queuing system altogether).
The old behaviour is maintained by setting the batching system
to slurm by default; the user can change the system by using
--batch_system.
| -rw-r--r-- | README.md | 3 | ||||
| -rw-r--r-- | pilot.sh | 8 |
2 files changed, 8 insertions, 3 deletions
@@ -35,6 +35,9 @@ Options can be the following: * `--outdir` is a path relative to which intermediate files and final data products will be written. Will be created if it does not exist. If not specified, `$BINDDIR` will be used instead. +* `--batch_system` specifies the queuing system to be used. + Defaults to `slurm`. + Use `single_machine` to run on the local node. * `<workflow name>` is the workflow file name without extension, e.g. `delay-calibration` or `concatenate-flag` for the VLBI pipeline or `HBA_calibrator` or `HBA_target` for LINC. ## Notes @@ -1,7 +1,7 @@ #!/bin/sh -eu usage() { - echo "Usage: ${PROGRAM} [-r|--restart] [-h|--help] [-c|--container <singularity container>] [--scratch <scratch dir>] [--outdir <output dir>] (-f <input file>) (-p <pipeline>) <Workflow> <input dir>" + echo "Usage: ${PROGRAM} [-r|--restart] [-h|--help] [-c|--container <singularity container>] [--scratch <scratch dir>] [--outdir <output dir>] [--batch_system <batch system>] (-f <input file>) (-p <pipeline>) <Workflow> <input dir>" exit 0 } @@ -19,13 +19,14 @@ set_container() { fi } -opts=$(getopt -o rhf:c:p: --long restart,help,container:,scratch:,outdir: \ +opts=$(getopt -o rhf:c:p: --long restart,help,container:,scratch:,outdir:,batch_system: \ -n 'pilot' -- "$@") eval set -- "$opts" RESTART="" SCRATCH="" +BATCH_SYSTEM="slurm" while true; do case "$1" in -c | --container) set_container "${2}"; shift 2 ;; @@ -35,6 +36,7 @@ while true; do -r | --restart) RESTART="--restart"; shift ;; --scratch) SCRATCH="${2}"; shift 2 ;; --outdir) OUTDIR="${2}"; shift 2;; + --batch_system) BATCH_SYSTEM="${2}"; shift 2;; * ) shift; break ;; esac done @@ -99,7 +101,7 @@ TOIL_COMMAND="toil-cwl-runner ${RESTART} ${TMPDIR_PREFIX} \ --writeLogs ${JOB_LOG_DIR} \ --stats \ --clusterStats ${STATS_DIR} \ - --batchSystem slurm \ + --batchSystem ${BATCH_SYSTEM} \ --batchLogsDir ${BATCH_LOG_DIR} \ --tmp-outdir-prefix ${TMP_OUTDIR} \ --workDir ${WORK_DIR} \ |