Advanced Code Execution

Note

Please consult the detailed usage in the help of each command (use -h or --help argument to display the manual).

Running concurrent experiment sessions

In addition to single-shot code execution as described in Running simple sessions, the run command offers concurrent execution of multiple sessions with different parameters interpolated in the execution command specified in --exec option and environment variables specified as -e / --env options.

To define variables interpolated in the --exec option, use --exec-range. To define variables interpolated in the --env options, use --env-range.

Here is an example with environment variable ranges that expands into 4 concurrent sessions.

backend.ai run -c 'import os; print("Hello world, {}".format(os.environ["CASENO"]))' \
    -r cpu=1 -r mem=256m \
    -e 'CASENO=$X' \
    --env-range=X=case:1,2,3,4 \
    lablup/python:3.6-ubuntu18.04

Both range options accept a special form of argument: “range expressions”. The front part of range option value consists of the variable name used for interpolation and an equivalence sign (=). The rest of range expressions have the following three types:

Expression	Interpretation
`case:CASE1,CASE2,...,CASEN`	A list of discrete values. The values may be either string or numbers.
`linspace:START,STOP,POINTS`	An inclusive numerical range with discrete points, in the same way of `numpy.linspace()`. For example, `linspace:1,2,3` generates a list of three values: 1, 1.5, and 2.
`range:START,STOP,STEP`	A numerical range with the same semantics of Python’s `range()`. For example, `range:1,6,2` generates a list of values: 1, 3, and 5.

If you specify multiple occurrences of range options in the run command, the client spawns sessions for all possible combinations of all values specified by each range.

Note

When your resource limit and cluster’s resource capacity cannot run all spawned sessions at the same time, some of sessions may be queued and the command may take a long time to finish.

Warning

Until all cases finish, the client must keep its network connections to the server alive because this feature is implemented in the client-side. Server-side batch job scheduling is under development!