template

qmap template is a tool aimed to ease the creation of a jobs file to be used with qmap submit.

Features:

• find your current loaded EasyBuild modules and adds them to the output as pre-command
• find your current conda environment and add them to the output as pre-command
• the job parameters passed through command line are added to the generated jobs_file

Generating a file

By default, the output is printed to the standard output. You can provide a file with the -o flag or redirect the output to a file (> file.txt).

Using wildcards

qmap template accepts two types of wildcards:

• user wildcards: indicated with {{...}} They can contain:
  • list of , separated items: the wildcard is replaced by each item
  • file name: he wildcard is replaced by each line in the file
• glob wildcards: if any of * and ** is found, it is assumed to be a glob wildcard and therefore expanded using the python glob module.

How it works

The expansion of wildcards is a two step process. First user wildcards are expanded and glob wildcards are expanded in a second phase. For the latter, any set of characters surrounded by blanks is analysed. If it contains one or more of the mentioned wildcards, a glob search is performed.

Note

Use \ before glob wildcards to avoid their expansion

Named groups

qmap template contains a special feature that allows the user to replace the values of any of the wildcards in different parts of the command.

To use this feature, the wildcard needs to be named using {{?<name>:<value>}} and it can be replaced anywhere using {{?=<name>}}.

The name can be anything, but a glob wildcard character. It cannot start with a number.

Tip

We recommend to limit the names to characters in a-z, A-Z and 0-9.

The value can be anything in a user wildcard or a glob wildcard.

Note

Even if it is possible to use a glob wildcard in a user wildcard (e.g. {{a,*.txt}}) we do not recommend this use for named groups as the result might differ from the expected.

Warning

As mentioned, in a user group you can place anything that is is a user or glob wildcard. Thus, {{?group:*}}.txt recognize the glob wildcard and and will do a glob search for all .txt files. On the other hand, {{?group:*.txt}} assumes it is a user wildcard (as it is not only a glob wildcard) and will try to open a file named *.txt which most likely will not exits and will fail.

Usage

Basic usage:

qmap template "<command>" -m <memory> -c <cores> -o <output file>

Check all options using qmap template --help.

Examples

Easybuild modules and conda environments are recognized:

$ qmap template "sleep 5"
[pre]
module load anaconda3/4.4.0
source activate test_qmap

[jobs]
sleep 5

Job parameters can also be added:

$ qmap template "sleep 5" -c 1 -m 1G
[params]
memory=1G
cores=1

[jobs]
sleep 5

Using user wildcards with lists:

$ qmap template "sleep {{5,10}}"
[jobs]
sleep 5
sleep 10

Using user wildcards with files:

$ qmap template "sleep {{sleep_times.txt}}"
[jobs]
sleep 5
sleep 10

Using glob wildcards:

$ qmap template "mypgrog --input *.txt"
[jobs]
mypgrog --input file1.txt
mypgrog --input file2.txt

Using named wildcards:

$ qmap template "myprog --input {{?f_name:*}}.txt --variable {{?v_name:a,b}} --output {{?=f_name}}_{{?=v_name}}"
[jobs]
myprog --input file1.txt --variable a --output file1_a
myprog --input file2.txt --variable a --output file2_a
myprog --input file1.txt --variable b --output file1_b
myprog --input file2.txt --variable b --output file2_b