Daily Development Workflows
Since 22.09, we have migrated to Pants as our primary build system and dependency manager for the mono-repository of Python components.
Pants is a graph-based async-parallel task executor written in Rust and Python. It is tailored to building programs with explicit and auto-inferred dependency checks and aggressive caching.
The command pattern:
$ ./pants [GLOBAL_OPTS] GOAL [GOAL_OPTS] [TARGET ...]
scripts/install-dev.shsays that you need to use
./pants, replace all
./pantsin the following command examples with
Goal: an action to execute
You may think this as the root node of the task graph executed by Pants.
Target: objectives for the action, usually expressed as
The targets are declared/defined by
The global configuration is at
Recommended reading: https://www.pantsbuild.org/docs/concepts
Inspecting build configurations
Display all targets
$ ./pants list ::
This list includes the full enumeration of individual targets auto-generated by collective targets (e.g.,
python_source()targets by globbing the
Display all dependencies of a specific target (i.e., all targets required to build this target)
$ ./pants dependencies --transitive src/ai/backend/common:lib
Display all dependees of a specific target (i.e., all targets affected when this target is changed)
$ ./pants dependees --transitive src/ai/backend/common:lib
Pants statically analyzes the source files to enumerate all its imports
and determine the dependencies automatically. In most cases this works well,
but sometimes you may need to manually declare explicit dependencies in
Running lint and check
Run lint/check for all targets:
$ ./pants lint :: $ ./pants check ::
To run lint/check for a specific target or a set of targets:
$ ./pants lint src/ai/backend/common:: tests/common:: $ ./pants check src/ai/backend/manager::
Currently running mypy with pants is slow because mypy cannot utilize its own cache as pants invokes mypy per file due to its own dependency management scheme.
(e.g., Checking all sources takes more than 1 minutes!)
This performance issue is being tracked by pantsbuild/pants#10864. For now, try using a
smaller target of files that you work on and use an option to select the
targets only changed (
If you encounter failure from
isort, you may run the formatter to automatically fix the import ordering issues.
$ ./pants fmt :: $ ./pants fmt src/ai/backend/common::
Running unit tests
Here are various methods to run tests:
$ ./pants test :: $ ./pants test tests/manager/test_scheduler.py:: $ ./pants test tests/manager/test_scheduler.py:: -- -k test_scheduler_configs
You may also try
--changed-since option like
To specify extra environment variables for tests, use the
$ ./pants test \ > --test-extra-env-vars=MYVARIABLE=MYVALUE \ > tests/common:tests
Running integration tests
$ ./backend.ai test run-cli user,admin
Building wheel packages
To build a specific package:
$ ./pants \ > --tag="wheel" \ > package \ > src/ai/backend/common:dist $ ls -l dist/*.whl
If the package content varies by the target platform, use:
$ ./pants \ > --tag="wheel" \ > --tag="+platform-specific" \ > --platform-specific-resources-target=linux_arm64 \ > package \ > src/ai/backend/runner:dist $ ls -l dist/*.whl
Using IDEs and editors
Pants has an
export goal to auto-generate a virtualenv that contains all
external dependencies installed in a single place.
This is very useful when you work with IDEs and editors.
To (re-)generate the virtualenv, run:
$ ./pants export ::
Then configure your IDEs/editors to use
dist/export/python/virtualenvs/python-default/VERSION/bin/python as the
interpreter for your code, where
VERSION is the interpreter version
To make LSP (language server protocol) services like PyLance to detect our source packages correctly,
you should also configure
PYTHONPATH to include the repository root’s
src directory and
plugins/*/ directories if you have added Backend.AI plugin checkouts.
To activate flake8/mypy checks (in Vim) and get proper intelli-sense support for pytest (in VSCode), just install them in the exported venv as follows. (You need to repeat this when you re-export!)
$ ./py -m pip install flake8 mypy pytest
For Vim, you also need to explicitly activate the exported venv.
Switching between branches
When each branch has different external package requirements, you should run
./pants export ::
before running codes after
git switch-ing between such branches.
To run a Python program within the unified virtualenv, use the
script. It automatically passes additional arguments transparently to the
Python executable of the unified virtualenv.
./backend.ai is an alias of
./py -m ai.backend.cli.
$ ./py -m ai.backend.storage.server $ ./backend.ai mgr start-server $ ./backend.ai ps
Working with plugins
To develop Backend.AI plugins together, the repository offers a special location
./plugins where you can clone plugin repositories and a shortcut script
scripts/install-plugin.sh that does this for you.
$ scripts/install-plugin.sh lablup/backend.ai-accelerator-cuda-mock
This is equivalent to:
$ git clone \ > https://github.com/lablup/backend.ai-accelerator-cuda-mock \ > plugins/backend.ai-accelerator-cuda-mock
These plugins are auto-detected by scanning
setup.cfg of plugin subdirectories
ai.backend.plugin.entrypoint module, even without explicit editable installations.
Writing test cases
Mostly it is just same as before: use the standard pytest practices. Though, there are a few key differences:
Tests are executed in parallel in the unit of test modules.
Therefore, session-level fixtures may be executed multiple times during a single run of
If you interrupt (Ctrl+C, SIGINT) a run of
./pants test, it will
immediately kill all pytest processes without fixture cleanup. This may
accumulate unused Docker containers in your system, so it is a good practice
docker ps -a periodically and clean up dangling containers.
To interactively run tests, see Debugging test cases (or interactively running test cases).
Here are considerations for writing Pants-friendly tests:
Ensure that it runs in an isolated/mocked environment and minimize external dependency.
If required, use the environment variable
BACKEND_TEST_EXEC_SLOT(an integer value) to uniquely define TCP port numbers and other resource identifiers to allow parallel execution. Refer the Pants docs.
ai.backend.testutils.bootstrapto populate a single-node Redis/etcd/Postgres container as fixtures of your test cases. Import the fixture and use it like a plain pytest fixture.
These fixtures create those containers with OS-assigned public port numbers and give you a tuple of container ID and a
ai.backend.common.types.HostPortPairfor use in test codes. In manager and agent tests, you could just refer
local_configto get a pre-populated local configurations with those port numbers.
In this case, you may encounter
flake8complaining about unused imports and redefinition. Use
# noqa: F401and
# noqa: F811respectively for now.
About using /tmp in tests
If your Docker service is installed using Snap (e.g., Ubuntu 20.04 or
later), it cannot access the system
/tmp directory because Snap applies a
private “virtualized” tmp directory to the Docker service.
You should use other locations under the user’s home directory (or
.tmp in the working copy directory) to avoid mount failures
for the developers/users in such platforms.
It is okay to use the system
/tmp directory if they are not mounted inside
Create a new pyenv virtualenv based on Python 3.10.
$ pyenv virtualenv 3.10.4 venv-bai-docs
Activate the virtualenv and run:
$ pyenv activate venv-bai-docs $ pip install -U pip setuptools wheel $ pip install -U -r docs/requirements.txt
You can build the docs as follows:
$ cd docs $ pyenv activate venv-bai-docs $ make html
To locally serve the docs:
$ cd docs $ python -m http.server --directory=_build/html
(TODO: Use Pants’ own Sphinx support when pantsbuild/pants#15512 is released.)
Adding new external dependencies
Add the package version requirements to the unified requirements file (
module_mappingfield in the root build configuration (
./BUILD) if the package name and its import name differs.
type_stubs_module_mappingfield in the root build configuration if the package provides a type stubs package separately.
$ ./pants generate-lockfiles $ ./pants export ::
Merging lockfile conflicts
When you work on a branch that adds a new external dependency and the main branch has also
another external dependency addition, merging the main branch into your branch is likely to
make a merge conflict on
In this case, you can just do the followings since we can just regenerate the lockfile
$ git merge main ... it says a conflict on python.lock ... $ git checkout --theirs python.lock $ ./pants generate-lockfiles --resolve=python-default $ git add python.lock $ git commit
If Pants behaves strangely, you could simply reset all its runtime-generated files by:
$ killall pantsd $ rm -r .tmp .pants.d ~/.cache/pants
After this, re-running any Pants command will automatically reinitialize itself and all cached data as necessary.
Debugging test cases (or interactively running test cases)
When your tests hang, you can try adding the
--debug flag to the
./pants test command:
$ ./pants test --debug ...
so that Pants runs the designated test targets serially and interactively.
This means that you can directly observe the console output and Ctrl+C to
gracefully shutdown the tests with fixture cleanup. You can also apply
additional pytest options such as
-s, etc. by passing them
after target arguments and
-- when executing
./pants test command.
Boosting the performance of Pants commands
Since Pants uses temporary directories for aggressive caching, you could make
.tmp directory under the working copy root a tmpfs partition:
$ sudo mount -t tmpfs -o size=4G tmpfs .tmp
To make this persistent across reboots, add the following line to
tmpfs /path/to/dir/.tmp tmpfs defaults,size=4G 0 0
The size should be more than 3GB. (Running
./pants test ::consumes about 2GB.)
To change the size at runtime, you could simply remount it with a new size option:
$ sudo mount -t tmpfs -o remount,size=8G tmpfs .tmp
Making a new release
./VERSIONfile to set a new version number. (Remove the ending new line, e.g., using
set noeolin Vim. This is also configured in
LOCKSET=tools/towncrier ./py -m towncrierto auto-generate the changelog.
You may append
--draftto see a preview of the changelog update without actually modifying the filesytem.
Make a new git commit with the commit message: “release: <version>”.
Make an annotated tag to the commit with the message: “Release v<version>” or “Pre-release v<version>” depending on the release version.
Push the commit and tag. The GitHub Actions workflow will build the packages and publish them to PyPI.
Backporting to legacy per-pkg repositories
git applyinstead of
To perform a three-way merge for conflicts, add
-3option to the
You may need to rewrite some codes as the package structure differs. (The new mono repository has more fine-grained first party packages divided from the
When referring the PR/issue numbers in the commit for per-pkg repositories, update them like