Environment management

This document explains in detail how environments are managed in an installation of OrangeQS Juice. An environment is the Python installation together with all installed packages. OrangeQS Juice uses different tools to manage environments in different contexts. There are two main types of places where a Python environment is required:

• On the host system: Host environment.

  • Used by the Juice Orchestrator.

• Inside a container: Containerized environments.

  • Used by OrangeQS Juice services.

  • Used by JupyterHub user containers.

A graphical overview of the environment management is shown below. This example uses the system package and a container based on a uv environment. Note that while this example shows a single environment image, there can be multiple images each containing their own environment.

Host environment

The host environment runs directly on the host system of the OrangeQS Juice installation. The main goal of this environment is to provide access to the Juice CLI. The system package provides a managed host environment (see below), but it is also possible to use a custom host environment. For a custom environment, one is free to manage it manually, as long as the juice command is present in the $PATH. This environment should contain the orangeqs-juice-core package, along with its dependencies. Both an editable install or a regular install of the orangeqs-juice-core package are supported in a custom environment.

System package

The OrangeQS Juice system package provides a managed environment for the Juice CLI. The environment is an isolated virtual environment built using virtualenv-tools3 This environment uses the Python binary from the system package manager. For more information see Package contents.

The system package is used in the Installation guide as the official installation method.

Containerized environments

OrangeQS Juice services and JupyterHub user containers run in isolated containers with a Python environment built into the image. These images are built directly on the host machine. In principle the image can use any environment manager, e.g. micromamba, virtualenv, uv. An alternative implementation could even not build the image locally and just pull the image from a registry.

For now Juice implements only environments based on uv.

Environment configuration

Each container that is started within OrangeQS Juice uses its own environment image. If the container configuration does not explicitly specify an environment configuration in orchestration.toml::services.{key}.environment, the environment configuration will be copied from orchestration.toml::environments.default. Note that in this case Juice still builds a separate image for the service, but just based on the same environment configuration. This is useful if one only wants to rebuild the environment of a specific service.

Below is a figure showing how the environment configuration is used to create the containerized environment. It shows how each component depends on the environment configuration and how the environment configuration is used to build the image, which is then used to run the container.

While a lot of files are built into the image (such as the Python binary and the installed packages), some files are mounted into the container at runtime (and during buildtime of the image):

• Any editable packages in the container under ~/shared/lib/{folder_name}.

• The shared folder: ~/shared.

• The system config path: /etc/juice/config.

• The shared runtime data path: /var/run/juice.

• The system package wheel if it exists: /opt/orangeqs/juice/dist/{wheel_name}.whl.

• Any other volumes specified in the environment configuration.

Building the environment

Below is a detailed step-by-step explanation of how the environment image is built for a containerized environment. These steps below are specific for a uv environment. They can be adapted for any other environment manager.

1. Read the environment configuration from the Juice configuration.

2. Generate a podman systemd build unit for the build process named juice-<name>-build.service, where <name> is the name of the environment. The build process consists of multiple steps.

   • ExecStartPre.

     • Determine an installation source for the orangeqs-juice-core python package. There are three possibilities:

       • It’s bundled in the system package as orangeqs-juice-core-*-py3-none-any.whl.

       • Juice is installed from a Python package registry, so determine the version and install from the registry.

       • Juice is installed as editable install, so we determine the source directory.

     • Create a folder on the host system specific for this environment: /var/juice/envs/<name>. In this folder:

       • Generate a /var/juice/envs/<name>/pyproject.toml with the required dependencies and pointing to the installation source of the orangeqs-juice-core package.

       • Generate a /var/juice/envs/<name>/Containerfile that defines the image.

   • ExecStart. Build the image using podman build with the generated Containerfile. The Containerfile defines the build steps, which include:

     • Install system packages if specified in the environment configuration.

     • Install the required python version under /env/.python.

     • Install a uv environment under /env/.venv.

   • ExecStopPost. Run podman image prune -f to remove dangling images after the build.

3. Reload the system daemon such that the service is loaded.

4. Start the systemd service juice-<name>-build.service, which builds the image.

Limitations

• We do not yet use uv.lock to lock the environment. This means environments are not 100% reproducible.

• The uv cache is configured at ~/shared/.uv-cache, which is shared between all environments and services. As the cache crosses a filesystem boundary, uv has to copy files from the cache to the environment instead of being able to hard-link them, which would be much faster. See link-mode in the uv documentation for more information.

• After updating Juice on the host system, all environment images need to be rebuilt to update the orangeqs-juice-core package. If some images fail to rebuild, reverting to the previous version of Juice on the host system will not work directly. Namely, images that were rebuild successfully using the new version are not compatible with the previous version of Juice. In that case a rebuild using the old version is required again, which is time-consuming (and not guaranteed to succeed!).