Thanks for this fantastic tool! I tried to use it for Snakemake, but it did not recognize Snakemake's option sections. I wonder about your opinion on this. Should we choose a different naming for those option sections so that it is easier for flycomp to parse (I am the Snakemake author):
usage: snakemake [-h] [--dry-run] [--profile PROFILE]
[--workflow-profile WORKFLOW_PROFILE] [--cache [RULE ...]]
[--snakefile FILE] [--cores N] [--jobs N] [--local-cores N]
[--resources NAME=INT [NAME=INT ...]]
[--set-threads RULE=THREADS [RULE=THREADS ...]]
[--max-threads MAX_THREADS]
[--set-resources RULE:RESOURCE=VALUE [RULE:RESOURCE=VALUE ...]]
[--set-scatter NAME=SCATTERITEMS [NAME=SCATTERITEMS ...]]
[--set-resource-scopes RESOURCE=[global|local] [RESOURCE=[global|local] ...]]
[--default-resources [NAME=INT ...]]
[--preemptible-rules [PREEMPTIBLE_RULES ...]]
[--preemptible-retries PREEMPTIBLE_RETRIES]
[--configfile FILE [FILE ...]] [--config [KEY=VALUE ...]]
[--replace-workflow-config] [--envvars VARNAME [VARNAME ...]]
[--directory DIR] [--touch] [--keep-going]
[--rerun-triggers {code,input,mtime,params,software-env} [{code,input,mtime,params,software-env} ...]]
[--force] [--executor {local,dryrun,touch}] [--forceall]
[--forcerun [TARGET ...]]
[--consider-ancient RULE=INPUTITEMS [RULE=INPUTITEMS ...]]
[--prioritize TARGET [TARGET ...]]
[--batch RULE=BATCH/BATCHES] [--until TARGET [TARGET ...]]
[--omit-from TARGET [TARGET ...]] [--rerun-incomplete]
[--shadow-prefix DIR]
[--strict-dag-evaluation {cyclic-graph,functions,periodic-wildcards} [{cyclic-graph,functions,periodic-wildcards} ...]]
[--scheduler [{greedy,ilp}]]
[--conda-base-path CONDA_BASE_PATH] [--no-subworkflows]
[--precommand PRECOMMAND] [--groups GROUPS [GROUPS ...]]
[--group-components GROUP_COMPONENTS [GROUP_COMPONENTS ...]]
[--report [FILE]] [--report-after-run]
[--report-stylesheet CSSFILE] [--report-metadata FILE]
[--reporter PLUGIN] [--draft-notebook TARGET]
[--edit-notebook TARGET] [--notebook-listen IP:PORT]
[--lint [{text,json}]] [--generate-unit-tests [TESTPATH]]
[--containerize [{dockerfile,apptainer}]] [--export-cwl FILE]
[--list-rules] [--list-target-rules]
[--dag [{dot,mermaid-js}]] [--rulegraph [{dot,mermaid-js}]]
[--filegraph] [--d3dag] [--summary] [--detailed-summary]
[--archive FILE] [--cleanup-metadata FILE [FILE ...]]
[--cleanup-shadow] [--skip-script-cleanup] [--unlock]
[--list-changes {code,input,params}] [--list-input-changes]
[--list-params-changes] [--list-untracked]
[--delete-all-output | --delete-temp-output]
[--keep-incomplete] [--drop-metadata] [--version]
[--printshellcmds] [--debug-dag] [--nocolor]
[--quiet [{all,host,progress,reason,rules} ...]]
[--print-compilation] [--verbose] [--force-use-threads]
[--allow-ambiguity] [--nolock] [--ignore-incomplete]
[--max-inventory-time SECONDS] [--trust-io-cache]
[--max-checksum-file-size SIZE] [--latency-wait SECONDS]
[--wait-for-free-local-storage WAIT_FOR_FREE_LOCAL_STORAGE]
[--wait-for-files [FILE ...]] [--wait-for-files-file FILE]
[--runtime-source-cache-path PATH]
[--queue-input-wait-time SECONDS]
[--omit-flags OMIT_FLAGS [OMIT_FLAGS ...]] [--notemp]
[--all-temp] [--unneeded-temp-files FILE [FILE ...]]
[--keep-storage-local-copies] [--not-retrieve-storage]
[--target-files-omit-workdir-adjustment]
[--allowed-rules ALLOWED_RULES [ALLOWED_RULES ...]]
[--max-jobs-per-timespan MAX_JOBS_PER_TIMESPAN]
[--max-status-checks-per-second MAX_STATUS_CHECKS_PER_SECOND]
[--seconds-between-status-checks SECONDS_BETWEEN_STATUS_CHECKS]
[--retries RETRIES] [--wrapper-prefix WRAPPER_PREFIX]
[--default-storage-provider DEFAULT_STORAGE_PROVIDER]
[--default-storage-prefix DEFAULT_STORAGE_PREFIX]
[--local-storage-prefix LOCAL_STORAGE_PREFIX]
[--remote-job-local-storage-prefix REMOTE_JOB_LOCAL_STORAGE_PREFIX]
[--shared-fs-usage {input-output,persistence,software-deployment,software-deployment-cache,source-cache,sources,storage-local-copies,none} [{input-output,persistence,software-deployment,software-deployment-cache,source-cache,sources,storage-local-copies,none} ...]]
[--scheduler-greediness SCHEDULER_GREEDINESS]
[--scheduler-subsample SCHEDULER_SUBSAMPLE] [--no-hooks]
[--debug] [--runtime-profile FILE]
[--local-groupid LOCAL_GROUPID] [--attempt ATTEMPT]
[--show-failed-logs] [--logger {} [{} ...]]
[--job-deploy-sources] [--benchmark-extended]
[--persistence-backend {db,file}]
[--persistence-backend-db-url PERSISTENCE_BACKEND_DB_URL]
[--container-image IMAGE] [--immediate-submit]
[--jobscript SCRIPT] [--jobname NAME]
[--software-deployment-method {apptainer,conda,env-modules} [{apptainer,conda,env-modules} ...]]
[--container-cleanup-images] [--use-conda]
[--conda-not-block-search-path-envvars] [--list-conda-envs]
[--conda-prefix DIR] [--conda-cleanup-envs]
[--conda-cleanup-pkgs [{tarballs,cache}]]
[--conda-create-envs-only] [--conda-frontend {conda,mamba}]
[--use-apptainer] [--apptainer-prefix DIR]
[--apptainer-args ARGS] [--use-envmodules]
[--deploy-sources QUERY CHECKSUM]
[--target-jobs TARGET_JOBS [TARGET_JOBS ...]]
[--mode {subprocess,remote,default}]
[--scheduler-solver-path SCHEDULER_SOLVER_PATH]
[--max-jobs-per-second MAX_JOBS_PER_SECOND]
[--report-html-path VALUE]
[--report-html-stylesheet-path VALUE]
[--scheduler-greedy-greediness VALUE]
[--scheduler-greedy-omit-prioritize-by-temp-and-input]
[--scheduler-ilp-solver VALUE]
[--scheduler-ilp-solver-path VALUE]
[targets ...]
Snakemake is a Python based language and execution environment for GNU Make-
like workflows.
options:
-h, --help show this help message and exit
EXECUTION:
targets Targets to build. May be rules or files.
--dry-run, --dryrun, -n
Do not execute anything, and display what would be
done. If you have a very large workflow, use `--dry-
run --quiet` to just print a summary of the DAG of
jobs. (default: False)
--profile PROFILE Profile to use for configuring the Snakemake run with
settings regarding the compute environment. Every key
in this YAML file gets parsed into the respective
command line argument: `executor: slurm` gets parsed
to `--executor slurm`, `default-resources: mem_mb:
16000` is interpreted as `--default-resources
mem_mb=16000`, etc. You can specify a Snakemake
profile as (i) a profile name, (ii) a relative path to
a folder or (iii) the relative path to the profile
YAML file itself. Snakemake will look for a folder
with the profile name or the existence of the relative
path in `/etc/xdg/snakemake`,
`/home/johannes/.config/snakemake` and the current
working directory. Alternatively, you can also specify
absolute paths. If a profile name or folder is given,
it has to contain a file `profile.yaml` (or a
`config.yaml` file, for backwards compatibility). This
file can have an optional infix specifying a minimal
snakemake version (for example `profile.v9+.yaml`).
The profile can also be set via the environment
variable `$SNAKEMAKE_PROFILE`. However, once you
provide a profile via the command line argument
`--profile`, this environment variable is ignored. And
to override this variable without setting another one,
provide the value `none` to this argument. Finally,
you can specify this argument multiple times. In this
case, the profiles get merged with the later
`--profile` instances overriding top-level entries in
profiles specified earlier. For example, if the last
`--profile` specifies the top level `default-
resources:` keyword, all entries under that keyword
from previous `--profile`s will be ignored. Similarly,
also specifying any of the top-level keys from your
profile as a command line argument will overwrite this
whole top-level key. Example profiles for certain
compute infrastructure can be obtained at
https://github.com/snakemake/snakemake-cluster-
profiles. [env var: SNAKEMAKE_PROFILE]
--workflow-profile WORKFLOW_PROFILE
Profile to use for configuring this Snakemake run with
parameters specific for this workflow (like
resources). For settings specific to the compute
environment (for example a specific compute cluster),
use global `--profile`s. Generally, an entry like
`set-resources: a: mem_mb=8` in the YAML file, will
become `--set-resources a:mem_mb=8` for the
`snakemake` run. The profile can be specified as a
file name with a full relative path from the current
working directory. In this case, the YAML profile file
can be named arbitrarily. In all other cases the
respective folder(s) will be searched for a
`profile.yaml` file (or a `config.yaml` file, for
backwards compatibility). This file can have an
optional infix specifying a minimal snakemake version
(for example `profile.v9+.yaml`). And any of the
following options will always search relative to both
the current working directory and the location of the
Snakefile: (i) If this option is not provided, the
directory `profiles/default/` will be searched (and
used, if a profile is present; override this implicit
usage with `--workflow-profile none`). (ii) If a
profile name is given, the subdirectory of that name
under `profiles/` will be searched. (iii) If a full
relative path is given, this directory will be
searched. Settings made in the workflow profile will
override settings made in the general profile (see
`--profile`) on a per-key basis. For example, if you
specify `default-resources:` in the workflow profile,
all `default-resources:` entries from other profiles
will be ignored; but if you don't specify `default-
resources` in your workflow profile, `default-
resources` from other profiles will get passed
through. Similarly, also specifying any of the top-
level keys from your workflow specific profile via
command line arguments will completely overwrite their
entries.
--cache [RULE ...] Store output files of given rules in a central cache
given by the environment variable
`$SNAKEMAKE_OUTPUT_CACHE`. Likewise, retrieve output
files of the given rules from this cache if they have
been created before (by anybody writing to the same
cache), instead of actually executing the rules.
Output files are identified by hashing all steps,
parameters and software stack (conda envs or
containers) needed to create them. If no rules are
given, all rules that are eligible for caching (have a
cache directive, see docs) are cached.
--snakefile, -s FILE The workflow definition in form of a snakefile.
Usually, you should not need to specify this. By
default, Snakemake will search for `Snakefile`,
`snakefile`, `workflow/Snakefile`,
`workflow/snakefile` beneath the current working
directory, in this order. Only if you definitely want
a different layout, you need to use this parameter.
--cores, -c N Use at most N CPU cores/jobs in parallel. If N is
omitted or `all`, the limit is set to the number of
available CPU cores. In case of cluster/cloud
execution, this argument sets the maximum number of
cores requested from the cluster or cloud scheduler.
(See https://snakemake.readthedocs.io/en/stable/snakef
iles/rules.html#resources-remote-execution for more
info.) This number is available to rules via
workflow.cores.
--jobs, -j N Use at most N CPU cluster/cloud jobs in parallel. For
local execution this is an alias for `--cores` (it is
though recommended to use `--cores` in that case).
Note: Set to `unlimited` to allow any number of
parallel jobs.
--local-cores N In cluster/cloud mode, use at most N cores of the host
machine in parallel (default: number of CPU cores of
the host). The cores are used to execute local rules.
This option is ignored when not in cluster/cloud mode.
--resources, --res NAME=INT [NAME=INT ...]
Define additional resources that shall constrain the
scheduling analogously to `--cores` (see above). A
resource is defined as a name and an integer value.
E.g. `--resources mem_mb=1000`. Rules can use
resources by defining the resource keyword, e.g.
`resources: mem_mb=600`. If now two rules require 600
of the resource `mem_mb` they won't be run in parallel
by the scheduler. In cluster/cloud mode, this argument
will also constrain the amount of resources requested
from the server. (See https://snakemake.readthedocs.io
/en/stable/snakefiles/rules.html#resources-remote-
execution for more info.) (default: )
SOFTWARE DEPLOYMENT:
--software-deployment-method, --deployment-method, --deployment, --sdm {apptainer,conda,env-modules}
Specify software environment deployment method.
--container-cleanup-images
Remove unused containers (default: False)
Thanks for this fantastic tool! I tried to use it for Snakemake, but it did not recognize Snakemake's option sections. I wonder about your opinion on this. Should we choose a different naming for those option sections so that it is easier for flycomp to parse (I am the Snakemake author):