Storage Management


Please consult the detailed usage in the help of each command (use -h or --help argument to display the manual).

Backend.AI abstracts shared network storages into per-user slices called “virtual folders” (aka “vfolders”), which can be shared between users and user group members.

Creating vfolders and managing them

The command-line interface provides a set of subcommands under vfolder to manage vfolders and files inside them.

To list accessible vfolders including your own ones and those shared by other users:

$ vfolder list

To create a virtual folder named “mydata1”:

$ vfolder create mydata1 mynas

The second argument mynas corresponds to the name of a storage host. To list up storage hosts that you are allowed to use:

$ vfolder list-hosts

To delete the vfolder completely:

$ vfolder delete mydata1

File transfers and management

To upload a file from the current working directory into the vfolder:

$ vfolder upload mydata1 ./bigdata.csv

To download a file from the vfolder into the current working directory:

$ vfolder download mydata1 ./bigresult.txt

To list files in the vfolder’s specific path:

$ vfolder ls mydata1 .

To delete files in the vfolder:

$ vfolder rm mydata1 ./bigdata.csv


All file uploads and downloads overwrite existing files and all file operations are irreversible.

Running sessions with storages

The following command spawns a Python session where the virtual folder “mydata1” is mounted. The execution options are omitted in this example. Then, it downloads ./bigresult.txt file (generated by your code) from the “mydata1” virtual folder.

$ vfolder upload mydata1 ./bigdata.csv
$ run --rm -m mydata1 python:3.6-ubuntu18.04 ...
$ vfolder download mydata1 ./bigresult.txt

In your code, you may access the virtual folder via /home/work/mydata1 (where the default current working directory is /home/work) just like a normal directory. If you want to mount vfolders in other path, add ‘/’ as prefix at the forefont of the vfolder path.

By reusing the same vfolder in subsequent sessions, you do not have to download the result and upload it as the input for next sessions, just keeping them in the storage.

Creating default files for kernels

Backend.AI has a feature called ‘dotfile’, created to all the kernels user spawns. As you can guess, dotfile’s path should start with .. The following command creates dotfile named .aws/config with permission 755. This file will be created under /home/work every time user spawns Backend.AI kernel.

$ dotfile create .aws/config < ~/.aws/config