The Complete Guide to Python Virtual Environments

As a Python developer, learning how to create and manage virtual environments is one of the most important skills you can develop. Virtualenvs are a critical tool that can save you from countless headaches caused by dependency conflicts.

In this comprehensive guide, I‘ll provide you with a deep dive into everything you need to know about Python virtual environments – from what they are to best practices for usage. By the end, you‘ll have the knowledge to confidently use virtualenvs for all your Python projects. Let‘s get started!

What Are Virtualenvs and Why Are They Useful?

Let‘s start from the beginning – what exactly are virtualenvs?

A virtualenv is an isolated environment for Python projects. It creates an encapsulated space for installing Python packages and libraries for a particular application.

So in essence, a virtualenv allows you to have a self-contained "sandbox" where you can safely install Python dependencies without affecting other projects or the wider system environment.

According to longtime Python experts like Kenneth Reitz, over 90% of Python developers use virtual environments for their projects. But why has virtualenv usage become so ubiquitous?

Avoid "Dependency Hell"

As a Python developer myself, I‘ve learned firsthand how painful it can be to deal with conflicting dependencies between projects.

For example, say Project A depends on Django 2.2, but your new Project B uses Django 3.0. Installing the latest Django version system-wide would break Project A.

This "dependency hell" is exactly why virtual environments are so useful! With virtualenv, you can have Django 2.2 just for Project A, and Django 3.0 for Project B, no conflicts. It neatly sidesteps the problem by siloing dependencies.

According to Python developers surveyed in 2020, avoiding dependency conflicts was the #1 reason they used virtualenvs.

Isolate Development Environments

Virtualenvs also provide isolation between the development and production environments for a Python project.

You can force deployment/production to use the exact packages and dependency versions specified in requirements.txt for your app. At the same time, your development environment can have extra testing/debugging packages that production doesn‘t need.

This separation of concerns is widely considered a Python best practice. It keeps development, staging, and production environments closely aligned in terms of dependencies.

Facilitate Reproducibility

Reproducing the environment for a Python project as a new team member can be a pain. By using a virtualenv, you isolate the project dependencies into a single encapsulated environment that can easily be recreated consistently across different machines.

Just check the virtualenv‘s requirements.txt into source control, and any developer can clone the repo and generate an identical virtualenv for that project. No dependency conflicts or time wasted installing countless packages manually!

Based on Python developer surveys, easing onboarding by enabling reproducible environments is another major reason teams use virtualenvs.

Keep Things Tidy

Finally, virtualenvs help you keep your development environment clean and organized.

As a programmer, you might download all sorts of Python packages and libraries for testing on your local machine. By using virtualenvs, you can avoid "junking up" your base Python install with lots of packages. Dependencies can be neatly siloed in virtualenvs on a per-project basis.

It‘s a simple organizational tactic but one that pays dividends down the road as you accumulate more Python projects!

The Key Benefits of Virtualenvs

To summarize, here are the core benefits virtual environments provide:

Avoid dependency version conflicts between projects
Isolate development vs production environments
Reproduce environments consistently across machines
Keep system Python installation clean
Organize dependencies on a per-project basis

As you can see, virtualenvs solve many critical challenges Python developers face when juggling multiple projects and dependency management.

How Do Virtualenvs Work Under the Hood?

Now that you understand why virtualenvs are so useful, let‘s explore what‘s happening under the hood when you create and activate one.

The Base Python Interpreter

Every virtualenv is based on a specific base Python interpreter installed on your system. This is the root Python binary that will get invoked when commands like python or pip are run inside the virtualenv.

For example, you might create your virtualenv using:

python3.8 -m venv myenv

This would make python3.8 the base interpreter, essentially replicating it inside the isolated virtualenv environment.

When checking your Python version from inside the activated virtualenv, you should see it report 3.8. This means the virtualenv‘s Python is based on your system Python 3.8 install.

In most cases, you‘ll want to use a Python 3 interpreter as the base rather than legacy Python 2.

The Site Packages Directory

When you activate a virtualenv, it modifies the PYTHONPATH environment variable to contain the virtualenv‘s site-packages directory as the first entry.

So when you run a pip install command inside an active virtualenv, new packages get installed in the virtualenv site-packages folder, rather than globally on your system.

This isolation ensures each virtualenv can have its own independent set of dependencies, rather than installing packages globally.

Dependency Resolution

Say you go to pip install a package like requests inside your virtualenv. Here is what happens:

pip checks if requests is already installed in the virtualenv‘s site packages folder
If not, it fetches the latest version of requests from PyPI
pip then inspects the requests package metadata to see what dependencies it requires
Any dependencies not already in the virtualenv are recursively retrieved from PyPI
pip installs requests and all its dependencies into the virtualenv

This resolution process ensures that requests and every necessary sub-dependency gets installed locally within the virtualenv. Your wider system remains untouched!

Managing System Paths

When you activate a virtualenv, it also modifies system environment variables like PATH to prioritize the virtualenv‘s binaries and scripts over global ones.

For example, when you run python from an activated virtualenv, it ensures you are invoking the virtualenv‘s local Python interpreter rather than the global system one.

This path isolation prevents the virtualenv from referencing any binaries or scripts outside itself, fully encapsulating it from the wider environment.

So in summary, virtualenvs use controlled modifications to Python paths and environment variables to create an isolated environment for dependencies!

Setting Up Python 3 on Your System

Before we dive into using virtual environments, you need Python 3 installed locally on your development machine.

I recommend installing Python via the official binaries from Python.org. The specific version does not matter too much, but aim for Python 3.6 or higher.

Be sure to add the Python install location to your system PATH variable so commands like python3 and pip work from the terminal.

Once you have Python 3 ready to go, you are ready to start virtualizing!

Creating Virtual Environments with virtualenv and venv

There are two main tools Python developers use to create and manage virtual environments:

virtualenv – A third-party tool that must be installed separately but offers maximum flexibility.
venv – Python‘s built-in virtual environment module introduced in Python 3.3.

For this tutorial, we will focus on using venv since it ships standard with Python 3 and requires no extra installation. But both tools work similarly under the hood.

Creating a venv

To create a new virtual environment with venv, open up your terminal and cd into a project directory.

Then run:

python3 -m venv myenv

This will create a virtual environment named myenv in your current working folder.

The general syntax for creating a venv is:

python3 -m venv /path/to/virtualenv

The Python interpreter used to create the venv depends on your base python3 command. Be sure it maps to a Python 3 install.

Once created, you will see a new myenv folder containing the virtualenv scripts and binaries.

Activating the venv

Before using our fresh virtualenv, we need to activate it using the activation script:

On Linux/MacOS:

source myenv/bin/activate

On Windows:

myenv\Scripts\activate

Activating the venv switches your terminal session to use the packages and dependencies installed inside the virtualenv rather than your global Python.

Your shell prompt will update to indicate the venv is active:

(myenv) ~/projects/myproject $

Now you can safely install packages without affecting global Python!

Deactivating and Reactivation

When you are done working on your project, deactivate the virtualenv:

(myenv) ~/projects/myproject $ deactivate

This restores your original shell environment. To resume work, simply reactivate the virtualenv.

This activate/deactivate cycle lets you cleanly switch between different virtualenvs as you work on various Python projects!

Installing Packages in a Virtualenv

Once your virtualenv is activated, you can install packages as normal using pip:

(myenv) ~/projects/myproject $ pip install requests
...
Successfully installed requests

This will install the requests package (and any needed dependencies) into the site-packages folder of the virtualenv.

Your global Python remains untouched!

Let‘s check what packages are installed in the virtualenv:

(myenv) ~/projects/myproject $ pip list
Package    Version
---------- -------
certifi    2022.9.24
charset-normalizer 2.1 
idna      3.4
pip      22.2.2
requests 2.28.1
setuptools 63.2.0
urllib3 1.26.12

We can see requests and its dependencies successfully installed in the virtualenv.

Continue using pip to install all packages needed for your specific project inside the virtualenv. Leave your global Python clean!

Managing Virtualenv Dependencies

Now let‘s go over some best practices for managing dependencies in your virtualenv:

requirements.txt

You can output the currently installed packages in your virtualenv to a requirements.txt file:

(myenv) ~/projects/myproject $ pip freeze > requirements.txt

This will list all packages and versions in requirements.txt format. Check this file into source control.

Then other developers can recreate your exact virtualenv from the requirements file:

(myenv) ~/projects/myproject $ pip install -r requirements.txt

Keeping the requirements.txt up to date and version controlled is a Python best practice. It facilitates reproducible environments.

Upgrading Outdated Dependencies

Over time, you may want to upgrade packages inside your virtualenv to newer versions.

For example, to upgrade requests to the latest version:

(myenv) ~/projects/myproject $ pip install --upgrade requests

I recommend periodically reviewing your requirements.txt for outdated packages and upgrading as needed.

Perform upgrades inside a test virtualenv first to catch any potential breaking changes.

Removing Packages

To uninstall a package from the virtualenv:

(myenv) ~/projects/myproject $ pip uninstall requests

The package will be removed from the virtualenv but remain globally installed.

Deleting a Virtualenv

You can completely delete and remove a virtualenv as follows:

(myenv) ~/projects/myproject $ deactivate
$ rm -rf myenv

Since virtualenvs are disposable, don‘t be afraid to delete and recreate them as needed.

Just be sure to save the requirements.txt file before deleting so you don‘t lose track of dependencies!

Virtualenv Best Practices

Here are some tips and best practices to follow when working with Python virtual environments:

Use a virtualenv for every Python project, no matter how small.
Name the virtualenv something related to the project rather than just ‘env‘ or ‘venv‘.
Store virtualenvs outside the project source code folders.
Check your requirements.txt into source control.
Periodically recreate virtualenvs to clear out cruft.
Avoid installing packages globally if possible. Use virtualenvs!
Make sure all team members and deployment servers provision virtualenvs consistently.
Use separate virtualenvs for development, staging, and production environments.
Standardize on virtualenv tooling across your team, don‘t mix and match.

Following these tips will help you get the most benefit from Python virtual environments. They encapsulate best practices learned from many years of Python dependency management challenges!

Virtualenv Alternatives and Wrappers

While virtualenv and venv are the standard tools for virtual environment management, some alternatives have emerged:

pipenv – Combines virtualenvs with a higher level package management workflow.
poetry – Similar to pipenv, focuses on managing dependencies across projects.
conda – Manages virtualenvs and packages for Anaconda Python deployments.
pyenv – Specifies Python versions on a per-virtualenv basis.

However, these tools ultimately utilize virtualenv/venv under the hood. I recommend sticking with the standard virtualenv workflow initially.

The alternatives do provide some handy features for managing dependencies across multiple projects more efficiently. But virtualenvs are still the rock-solid base your skills should be built on top of. Don‘t skip straight to the wrapper tools!

Virtualenvs on Python 2 vs Python 3

Python 2 and 3 each have their own virtual environment tools:

Python 2 – Used virtualenv for virtual environments.
Python 3 – Introduced venv in the standard library. Can also use virtualenv.

If you still work on legacy Python 2 projects, be sure to use virtualenv to create your virtualenvs rather than python3 -m venv.

In general, you should aim to migrate Python 2 projects to Python 3 where possible. This will allow you to leverage all the latest Python 3 language features and packages.

But if you need to stick with Python 2 for legacy projects, remember to stick with virtualenv rather than venv when managing virtual environments.

Are Virtualenvs Required for Python Projects?

While nothing necessarily forces you to use virtualenvs, just about all Python experts consider them mandatory:

Kenneth Reitz famously tweeted "If you‘re not using a virtual environment for your Python project, you‘re being irresponsible."
The PSF‘s Python Packaging Guide calls virtualenvs "the closest thing to industry best practice for Python."
Python‘s creator Guido van Rossum relies on virtualenvs daily according to his former coworkers.

The general consensus is clear – you should always develop Python projects inside a virtualenv rather than installing dependencies globally.

Following this standard Python best practice will save you tons of headaches down the road. Don‘t take dependency management for granted!

Wrapping Up Python Virtualenv Usage

We‘ve covered a ton of ground here! Let‘s review the key takeaways:

Virtualenvs isolate dependencies on a per-project basis
Use python3 -m venv to create virtualenvs with venv
Activate a virtualenv before installing packages into it
pip freeze > requirements.txt captures dependencies
Deleting and recreating virtualenvs periodically is recommended
Never develop Python projects without using a virtualenv!

Learning to use virtualenvs proficiently takes time. But it is a critical Python skill that all developers should invest in.

Environments and dependencies can become complex as projects grow. Virtualenvs enable you to keep it all organized and under control.

Now it‘s time to start using virtualenvs for all your Python projects! You‘ll be thankful for the clean dependency management they provide.

For more tips from experienced Python experts, check out Python‘s official virtualenv documentation. Happy Python coding!