Python Environment Management
Published On 2022/06/13 Monday, Singapore
This post covers Python environment and package management using different tools for macOS. Specifically, it discusses the following tools: venv, Anaconda, and Miniforge.
VENV
The venv module has been included with Python since version 3.3. The venv module provides support for creating lightweight “virtual environments” with their own site directories, optionally isolated from system site directories. Since it comes with Python installation, you don’t need to install it separately.
Create Virtual Environment
python3 -m venv /Users/joye/venv/joye_env
The above script creates an environment under /Users/joye/venv/ called joye_env. Under the folder /Users/joye/venv/joye_env, it creates the following files:
As you can see, it will only have 2 site-packages pip and setuptools by default. Both are used for package management and installation.
Activate Environment
Once a virtual environment has been created, it can be activated using a script in the virtual environment’s binary directory.
source /Users/joye/venv/joye_env/bin/activate
Install Packages
Once the environment has been activated, you can install packages needed for your project. pip can be used to install packages as follows:
pip install pandas
If you don’t specify the version of the package, it will install the most recent version. If you want to install a specific version of a package, you can specify it as follows:
pip install pandas==1.3.5
In some cases, you may want to install packages from a requirements.txt file. You can do this as follows:
pip install -r requirements.txt
Export Packages
At the end of the project, you may want to export all the packages in the environment to a requirements.txt file. You can do this as follows:
pip freeze > requirements.txt
It’s worth mentioning that depending on your virtual environment configuration of include-system-site-packages, the packages in requirements.txt will be different. By default, it is set to false, which means only packages under /Users/joye/venv/joye_env/lib/python3.8/site-packages will be exported to requirements.txt.
Deactivate Environment
deactivate
Anaconda
Anaconda is a distribution of the Python and R programming languages for scientific computing, aiming to simplify package management and deployment. The major difference between conda and the pip package manager is in how package dependencies are managed, which is a significant challenge for Python data science and the reason conda exists.
Anaconda was free and open-source but is no longer free for commercial use since April 2020. So if you are using it for commercial purposes, make sure you have purchased the license. They still provide a free edition for students, academics, and hobbyists.
Installation
To install Anaconda, you can download it from https://www.anaconda.com/products/distribution and then follow the installation instructions step by step. By default, it will be installed at /Users/joye/opt/anaconda3.
Once finished, run the following:
source /Users/joye/opt/anaconda3/bin
You can check the path of your conda as follows:
If you are using bash:
which conda
If you are using zsh:
which -p conda
Create Environment
conda create --name joye_env python=3.8
Activate Environment
Once a virtual environment has been created, it can be activated using a script in the virtual environment’s binary directory.
conda activate joye_env
Install Packages
You can still install packages using pip as shown in the previous section. Alternatively, you can use conda install:
conda install pandas
conda install --file requirements.txt
In some cases, you may want to install from a yaml file environment.yml.
conda env update --file environment.yml
Export Environment
conda env export > environment.yml --no-builds
pip freeze --path /Users/joye/opt/anaconda3/envs/joye_env/lib/python3.8/site-packages > requirements.txt
Deactivate Environment
conda deactivate
You may find the conda cheat sheet useful.
Miniforge
Install Packages
For macOS, Miniforge can be directly installed with brew:
brew install miniforge
Once finished, run the following:
source /usr/local/Caskroom/miniforge/base/bin/activate
The usage of Miniforge conda is the same as Anaconda conda.
References & Resources
- Anaconda (Python distribution), Wikipedia
- Venv, Python Documentation 3.10.5