Python Package Management

Package Installation

Posit Connect installs the Python package dependencies of Python-based content when that content is deployed. This includes FastAPI and Flask APIs, Jupyter notebooks, apps built using Dash, Bokeh, or Streamlit, and R projects that include Python.

Package dependencies are captured in one of two ways:

1. If a requirements.txt file exists in the directory containing the project being deployed, then the contents of that file specify the dependencies. See the pip documentation for details. If you provide a requirements.txt file, you must ensure that the listed dependencies are correct for the content you are deploying.

2. Otherwise, the pip freeze command is used to produce a full specification of the current Python environment including all installed packages and their version numbers. In a Jupyter notebook, pip freeze will be run with the version of Python being used by the active Jupyter notebook kernel. In the RStudio IDE, the environment variable RETICULATE_PYTHON will be used to determine which Python environment to inspect.

The resulting package list is included in the bundle archive file, which is uploaded to Posit Connect. Posit Connect receives the bundle archive file, unpacks it, and uses venv and pip to install the identified package dependencies. In addition to the specified dependencies, Posit Connect will also install packages that it uses to deploy and render your content.

The execution environment created by Posit Connect contains the same package versions you are using in your development environment. The use of venv isolates environments from one another to avoid package version conflicts.

Requirements Files

A requirements.txt file lists the packages that are required by a piece of content and (optionally) their versions. The pip package manager allows additional options in requirements files, giving authors more flexibility and control over package installation. See the pip documentation for details.

Environment Caching

Posit Connect maintains a cache of Python environments to enable faster deployments. New environments are created as needed, based on the list of package dependencies received in the bundle and the python version in use.

Subsequent deployments that have the same list of dependencies will reuse the previously-built environment. If any dependencies are different, a new environment will be created. This enables published content to make use of different versions of dependent packages without conflict.

Providing a requirements.txt file which is the same across multiple projects is one way to facilitate environment reuse and enable faster deployments. A similar benefit is achieved in the automatic (pip freeze) case if the Python environment on the publishing computer remains the same between deployments.

Additionally, environments in Posit Connect always inherit packages from the system-wide environment configured in Posit Connect. Providing an empty requirements.txt file (i.e., a placeholder requirements.txt file without content) allows you to use system-wide packages without adding any to the base set. This is useful in situations where packages are centrally managed (e.g., internal company packages). To set up system-wide packages, see the External Package Installation section below.

Posit Connect will periodically delete Python virtual environments that are no longer in use by any deployed content. The setting Application.PythonEnvironmentReapFrequency can be used to control how often this occurs.

External Package Installation

Warning

Adding external packages decreases the reproducibility and isolation of content on Posit Connect, and should only be done as a last resort.

You can indicate that a system-wide installation of a package should be used instead of one fetched by pip. Use the Python.External setting to enumerate system-provided Python packages that should not be managed by Connect.

For example, to make numpy and scipy external packages:

Install these packages in every Python installation that Posit Connect will be using, e.g.:

/opt/Python/3.8.1/bin/python -m pip install numpy scipy

Then configure Posit Connect to treat those packages as external:

; /etc/rstudio-connect/rstudio-connect.gcfg 
[Python] 
External = numpy
External = scipy

Posit Connect will report an error during startup if some of the external Python packages are missing from any configured Python installation. If you are not able to install an external Python package into all Python installations (perhaps because that package is not compatible with some versions of Python), use the Python.ExternalsCheckIsFatal setting to prevent this check.

; /etc/rstudio-connect/rstudio-connect.gcfg
[Python]
ExternalsCheckIsFatal = false

External Package Version Matching

By default, Posit Connect attempts to match external packages by name. In the example above, if an uploaded bundle requests numpy==1.15 and the Python installation in Posit Connect has numpy 1.18.1 installed, the external version will be used even though the version number does not match. This is similar to how external R packages are handled. Effectively, the version number specified in the incoming requirements.txt file is ignored for external packages.

To require strict version matching, honoring exactly what is specified in the bundle's requirements.txt file, you can set Python.ExternalVersionMatching to true. In this case, the version requested in the bundle's requirements.txt file will be installed if needed, and the external version will only be used if it's the correct version.

Excluding Python Packages

To exclude a certain Python package from being utilized, set Python.ProhibitedPackage to the name of the package. Specify this property once for each Python package that is to be excluded.

This may be used, for example, when deploying Python-enabled content that utilizes OS-specific packages which are unavailable on the OS that Posit Connect runs on.

Posit Connect excludes certain Python packages by default. For a list of these packages please see the Python configuration appendix.

Configuring pip

Since Posit Connect uses pip to install Python packages, you can set package installation options by creating or modifying the pip.conf file.

The global pip configuration file is /etc/pip.conf. Alternatively, since all Python environment restore processes are run under the user account specified in the Applications.RunAs configuration, you can configure options in the default RunAs user's pip.conf file. For example, if the default RunAs user is rstudio-connect, the configuration file might be at /home/rstudio-connect/.config/pip/pip.conf.

For more information about configuring pip, refer to the pip user guide.

Specifying a Package Repository

If you have a Python package repository for your own Python packages, or have a PyPI mirror inside your firewall, you can configure Posit Connect to use that package repository when installing packages.

For example, to configure a private package repository with a timeout of 60 seconds, add the following to pip.conf:

[global]
timeout = 60
index-url = https://my-python-package-repo.internal.com

Note that setting index-url replaces pip's default repository (PyPI). To add a new repository, use the extra-index-url setting.