Using Mocked Accelerators

For developers who do not have access to physical accelerator devices such as CUDA GPUs, we provide a mock-up plugin to simulate the system configuration with such devices, allowing development and testing of accelerator-related features in various components including the web UI.

Configuring the mock-accelerator plugin

Check out the examples in the configs/accelerator directory.

Here are the description of each field:

  • slot_name: The resource slot’s main key name. The plugin’s resource slot name has the form of "<slot_name>.<subtype>", where the subtype may be something such as device (default), shares (for the fractional allocation mode). For CUDA MIG devices, it becomes a string including the slice size from the device memory size such as 10g-mig.

    To configure the fractional allocation mode, you should also specify the etcd accelerator plugin configuration like the following JSON, where unit_mem and unit_proc is used as the divisor to calculate 1.0 fraction:

    {
   "config": {
     "plugins": {
       "accelerator": {
         "<slot_name>": {
           "allocation_mode": "fractional",
           "unit_mem": 1073741824,
           "unit_proc": 10
         }
       }
     }
   }
}

    In the above example, the 10 subprocessors and 1 GiB of device memory is regarded as 1.0 fractional device. You may store it as a JSON file and put in the etcd configuration tree like:

    $ ./backend.ai mgr etcd put-json '' mydevice-fractional-mode.json

  • device_plugin_name: The class name to use as the actual implementation. Currently there are two: CUDADevice and MockDevice.

  • formats.<subtype>: The tables for per-subtype formatting details

    • display_icon: The device icon type displayed in the UI.

    • display_unit: The resource slot unit displayed in the UI, alongside the amount numbers.

    • human_readable_name: The device name displayed in the UI.

    • description: The device description displayed in the UI.

    • number_format: The number formatting string used for the UI.

      • binary: A boolean flag to indicate whether to use the binary suffixes (divided by 2^(10n) instead of 10^(3n))

      • round_length: The length of fixed points to wrap the numeric value of this resource slot. If zero, the number is treated as an integer.

  • devices: The list of mocked device declarations

    • mother_uuid: The unique ID of the device, which may be random-generated

    • model_name: The model name to report to the manager as metadata

    • numa_node: The NUMA node index to place this device.

    • subproc_count: The number of sub-processing cores (e.g., the number of streaming multi-processors of CUDA GPUs)

    • memory_size: The size of on-device memory represented as human-readable binary sizes

    • is_mig_devices: (CUDA-specific) whether this device is a MIG slice or a full device

Activating the mock-accelerator plugin

Add "ai.backend.accelerator.mock" to the agent.toml’s [agent].allowed-compute-plugins field. Then restart the agent.