Python Environment Management
Published On 2022/06/13 Monday, Singapore
This post covers the python environment and packages management with different tools for Mac OS. Specifically, it covers the following tools: venv, Anaconda, and Miniforge.
VENV
The venv module comes with python since version 3.3. The venv module provides support for creating lightweight “virtual environments” with their own site directories, optionally isolated from system site directories. Because it comes with python installation, you don’t need to install it separately.
Create Virtual Environment
python3 -m venv /Users/joye/venv/joye_env
The above script creates an environment under /Users/joye/venv/ call joye_env. Under the folder /Users/joye/venv/joye_env, it creates the following files:
As you can see, it will only have 2 site-packages pip and setuptools by default. Both are used for package management and installation.
Activate Environment
Once a virtual environment has been created, it can be “activated” using a script in the virtual environment’s binary directory.
source /Users/joye/venv/joye_env/bin/activate
Install Packages
Once the environment has been activated, you can install packages that are needed for your project. pip can be used to install the package as follows.
pip install pandas
If you didn’t specify the version of the package, it will install the most updated version of the package. If you want to install specific version of package, you can specify the version of the package as follows:
pip install pandas==1.3.5
In some cases, you want to install the package from the requirements.txt, you can do it as follows.
pip install -r requirements.txt
Export Packages
At the end of the project, you may want to export all the packages in the environment to requirements.txt. You can do it as follows:
pip freeze >requirements.txt
It is worth mentioning here that depending on your configuration of the virtual environment on include-system-site-packages, the packages in requirements.txt will be different. By default, it equals false which means only package under /Users/joye/venv/joye_env/lib/python3.8/site-packages will be export to requirements.txt
Deactivate Environment
deactivate
Anaconda
Anaconda is a distribution of the Python and R programming languages for scientific computing, that aims to simplify package management and deployment. The big difference between conda and the pip package manager is in how package dependencies are managed, which is a significant challenge for Python data science and the reason conda exists.
Anaconda was free and open-source, but has been no longer free for commerial use since April, 2020. So if you are using tis for commercial use, make sure you have purchased the licence. They still provide free edition for students, academics and hobbyists.
Installation
To install anaconda, you can download it at https://www.anaconda.com/products/distribution first, and then click to install it by followimg its instructions step by step. By default, it will be install at /Users/joye/opt/anaconda3
Once finished, run the below
source /Users/joye/opt/anaconda3/bin
Once installed, you can check the path of your conda as follows:
if you are using bash
which conda
if you are using zsh
which -p conda
Create Environment
conda create --name joye_env python=3.8
Activate Environment
Once a virtual environment has been created, it can be “activated” using a script in the virtual environment’s binary directory.
conda activate joye_env
Install Packages
You can still install packages use pip as it is showed in previous section. Alternatively, you can use conda install
conda install pandas
conda install --file requirements.txt
In some cases, you may want to install from yaml file environment.yml.
conda env update --file environment.yml
Export Environment
conda env export > environment.yml --no-builds
pip freeze --path /Users/joye/opt/anaconda3/envs/joye_env/lib/python3.8/site-packages>requirements.txt
Deactivate Environment
conda deactivate
You may find the conda cheat sheet useful.
Miniforge
Install Packages
For Mac OS, miniforge can be directly installed with brew
brew install miniforge
Once finished, run the below
source /usr/local/Caskroom/miniforge/base/bin/activate
The usage of Miniforge conda is the same as Anaconda conda.
Reference & Resources
- Anaconda (Python distribution), Wikipedia
- venv — Creation of virtual environments, Python Documentation 3.10.5