Manage dependencies using environments | Databricks on AWS

On classic compute, the Environments dependency mode lets you manage dependencies at the workload level using base environments instead of installing compute-scoped libraries. This is the same model used by serverless compute.

Environment settings stay isolated from compute-level changes, so workloads don’t break when the compute changes underneath them. And because Databricks caches base environments, dependencies don’t resolve and install at startup, so compute starts faster and avoids failures from package resolution, repository availability, or network reliability.

After you connect to a compute that uses the Environments dependency mode, you can select a base environment and add dependencies from the notebook’s Environment side pane. See Use environments in a notebook.

Requirements​

The dependency mode setting is only available on compute that meets the following requirements:

• Databricks Runtime 19 (Beta) or above
• Standard access mode
• All-purpose compute run in interactive notebooks

For support details, see Limitations.

Set the dependency mode​

To create a compute that uses the Environments dependency mode:

1. In the workspace sidebar, click Compute then open the All-purpose compute tab.

2. Click Create compute.

3. Under Performance, select 19 Beta or above from the Databricks Runtime version drop-down menu.

4. Expand the Advanced section, then click Dependencies.

5. For Dependency mode, keep Auto selected to let Databricks resolve the mode, or click Manual and select Environments to set it yourself. See Dependency mode options.

   

6. Configure the rest of the compute, then click Create.

After the compute is running, attach a notebook and configure its environment as described in Use environments in a notebook.

Dependency mode options​

The Dependency mode control has two settings that determine how the mode is chosen:

• Auto: Databricks resolves the mode based on the compute configuration. Auto resolves to Environments unless the compute specifies Docker, Spark environment variables, or init scripts, in which case it resolves to Cluster libraries.
• Manual: You select Environments or Cluster libraries yourself.

When you select Manual, choose the mode that fits your needs:

• Environments: Manage dependencies at the workload level using base environments. Cluster libraries, Docker, Spark environment variables, and init scripts are disabled. Use this mode for a consistent, portable dependency experience that matches serverless compute.
• Cluster libraries: The classic behavior. Install dependencies on the compute using cluster libraries, init scripts, or Docker. Use this mode if your workload requires any of those settings.

Update an existing compute​

If you are editing the settings for existing compute, switching to Environments mode is blocked if the compute has incompatible settings, such as init scripts, cluster libraries, or Docker. If you remove the incompatible settings first, you can then edit the dependency mode.

If you switch to Cluster libraries mode from Environments mode, attached notebooks retain their environment selections but those selections become inactive. If you switch the compute back to Environments mode, those selections become active again.

Use environments in a notebook​

To use environments in a notebook, attach the notebook to a classic compute resource that uses the Environments dependency mode. After you attach the notebook, you can configure the notebook’s dependencies from the Environment side pane , the same way you do on serverless compute.

Select a base environment​

A base environment determines the pre-installed libraries and environment version available for your notebook. The Base environment selector in the Environment side pane is where you choose your environment. Databricks recommends using the latest version to get the most up-to-date notebook features. To see details on each environment version, see Environment versions.

The Base environment selector includes the following options:

• Standard: The default base environment with Databricks-provided libraries.
• ML: A base environment with the Python and system packages from Databricks Runtime for Machine Learning pre-installed.
• More: Expands to show additional options:
  • Previous versions of Standard and ML environments.
  • Custom: Specify a custom environment using a YAML file.
• Workspace environments: Lists all compatible base environments configured for your workspace by workspace admins.

Add dependencies to the notebook​

Databricks caches your notebook’s virtual environment, so dependencies don’t reinstall every time you reopen a notebook or resume after inactivity.

To individually install a dependency:

1. In the Dependencies section, enter the path of the dependency in the field then click +Add dependency. You can specify a dependency in any format that is valid in a requirements.txt file. Python wheel files or Python projects (for example, the directory containing a pyproject.toml or a setup.py) can be located in workspace files or Unity Catalog volumes.

   • If using a workspace file, the path should be absolute and start with /Workspace/.
   • If using a file in a Unity Catalog volume, the path should be in the following format: /Volumes////.whl.

2. Click Apply to install the dependencies and restart the Python process.

To view installed dependencies, click the Installed tab in the Environment side pane. Open the pip installation logs for the notebook environment by clicking pip logs at the bottom of the pane.

Limitations​

The Environments dependency mode supports Python workloads and dependencies in interactive notebooks. The following limitations apply:

• Classic jobs don’t use the Environments dependency mode. When a job runs on classic all-purpose compute that specifies Environments mode, the setting is ignored and the job runs in Cluster libraries mode instead.
• %scala code is not supported and will fail.
• All standard access mode limitations apply, including no support for R. See Standard compute requirements and limitations.
• You cannot currently enforce dependency mode settings using compute policies.

Incompatible settings​

When Environments dependency mode is enabled, the following compute settings are disabled:

• Docker (Databricks Container Services)
• Spark environment variables
• Init scripts
• Cluster libraries