Setting Up a Python Dev Environment Using Pyenv

Python isn't MATLAB. It isn't something you update once and then leave alone. It's a dynamic environment that is highly customizable for the project needs at hand. For most engineers shifting from scientific computing with MATLAB to the dynamic world of Python, this is quite the learning curve. As an example, here's a comment on Hacker News from a robotics engineer:

Imagine you had to work on machine learning projects using TensorFlow. There is often a need to work with different versions of TensorFlow. For example, if work involves legacy code or models, the code is often written in TensorFlow 1.XX. Starting entirely new projects, however, will involve the more modern TensorFlow 2.X. On top of this, there may be a need to go back and forth across different Python versions (i.e. The 3.5.X vs. 3.7.X) It is not an efficient method to install all the TensorFlow packages and Python versions you need across your entire system. Not only does it take up a ton of space, it could also lead to conflicts in the way your code is run. It quickly gets messy!

Environment management using version managers and virtual environments is the solution to this problem. Instead of globally installing every software requirement for a specific project, we create smaller, project-specific environments with specific Python versions, package versions, and any other software requirements. This nimbler approach to environment management is crucial to leveraging all the dynamism that Python has to offer, while avoiding issues with unwieldy global installs that start to conflict.

A great tool to do Python version and package management is Pyenv. In a nutshell, Pyenv easily allows you to manage different Python versions on your computer, without installing them globally. Using Pyenv, you can specify exactly what version of Python you want to run in each project directory that you'll be working in. I'll leave the deeper details of how it works to the excellent Pyenv wiki. At a high level, Pyenv creates a set of "shims" or API translators that intercept all the calls to Python that you make and ensure they are passed to the version of Python you care about for your project. The way Pyenv does this is through creating a /.python-version file in each repository that helps the system identify and call the right version of Python.

While Pyenv is a great tool, it can be a little tricky to setup properly for the totally uninitiated beginner. It exists on a GitHub repo primarily, it's not like a thing that you can just click Download and Install for. There are also a number of dependencies it has that are unfamiliar to the total Python beginner. In this tutorial, I will walk you through the exact steps needed to install and make use of Pyenv.

After completing this tutorial, this is what your new and improve Python workflow will look like:

That's it! You've much more cleanly created a custom environment for your project, easily shared its dependencies, and avoided dealing with all the conflicts that having multiple Python versions and package versions on your computer create. Now, let's get to the tutorial. I highly recommend reading through this guide to pyenv on the pyenv Github as you work through this tutorial.

So how do you actually install pyenv? Note: This guide is mainly for MacOS, but if you want to use it for other OS installations, you can simply skip the homebrew step and follow the pyenv guide directly.

The first step to installing pyenv is to first make sure that you have homebrew installed and that it actually works. The reason I bring this up, and that many guides don't, is that a bad brew installation caused me major issues in the past. If you're unfamiliar with homebrew, I recommend the introduction on the Homebrew website; the long and short of it is that is an essential set of tools that don't come preinstalled with macOS. To install homebrew, run the following command in your macOS Terminal. After running this command, you should have homebrew installed perfectly.

The next step is to install pyenv. This step has multiple elements, so please make sure to do all of them. During the installation of pyenv, you will not only be downloading and installing the software itself. You will also be setting up the "shims" or the executables that make pyenv properly. To start this process, use Homebrew, which you installed in the previous steps. Open to terminal, go to your home directory (easily accessible by running the command "cd ~"), and execute the commands:

After you install pyenv via Homebrew, you need to set up the shims to allow pyenv to appropriately direct to the right versions of Python. For convenience, I've copied and pasted the steps needed directly from the pyenv repo. For a more logical process in the context of the entire dev environment, I've slightly modified the order in which the commands suggested by the pyenv maintainers occur.

Important note: please take note of the shell that macOS terminal is running. The latest macOS has changed from bash to zsh as the default shell, and the profile file that is referenced or modified in each command is different for every shell. You can check which shell you are running by opening the terminal and running the command "echo $0".

First, "add pyenv init to your shell to enable shims and autocompletion. Please make sure eval "$(pyenv init -)" is placed toward the end of the shell configuration file since it manipulates PATH during the initialization." Use this command:

echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bash_profile

Then, "restart your shell so the path changes take effect."

$ exec "$SHELL"

Congrats! You've successfully installed pyenv and can now start to use it. To make it useful in the context of your dev environment, we will go through some additional steps.

Many of you may be familiar with Python virtual environments. For those who are not, I suggest going through this tutorial from the folks over at Real Python. In a nutshell, virtual environments help instantiate, manage, and control our package-level dependencies. The same way that pyenv allows us to manage multiple Python versions, virtual environments allow us to manage multiple package versions. As an example, think of a time where you might need to manage two different projects, one based on TensorFlow 1.15 and and the other based on TensorFlow 2.0. Both could be based on the same Python version (i.e. Python 3.6.6). By using a virtual environment, we can "isolate the dependencies" of each project and keep our workflow for each project clean.

Luckily, there is an handy plug-in for pyenv that allows us to easily create and manage virtual environments for the various Python installations we may need. This tool is called pyenv-virtualenv. To install pyenv-virtualenv, run the following commands:

Congrats! Now you have pyenv-virtualenv installed and ready to go. Moreover, you have the fundamental tools, in pyenv and pyenv-virtualenv, needed to manage your project-specific Python development environment. We will now move onto steps that are project specific.

Note: Before doing the following steps, in order to avoid any confusion, I suggest moving into the directory that you want to set up a specific Python environment for, i.e. through using a command like "cd Documents/new_python_project". This is because pyenv can only run specific versions of Python if it can find .python_version files.

Now, we will start to make full use of pyenv and specific Python environments. In this example, we will be using Python 3.7.3.

To get started before installing a specific version, it is recommended that you install build dependencies that Python versions may have. For build dependences, the pyenv wiki suggests Homebrew (which we previously installed) and Xcode Command Line Tools. The Xcode component is recommended, but not necessary. Additionally, you can optionally install Homebrew addendums. For both of these optional installations, use the following commands:

With these additional dependencies addressed we can now install a new version of Python with confidence. To install Python 3.7.3, which will be located in the location $(pyenv root)/versions, execute the command below.

The installation will take a few minutes; after all, this is a new version of Python.

As stated earlier, virtual environments make managing projects and their package level dependencies much easier. We also installed pyenv-virtualenv so that we can easily create and manage virtual environments that are compatible with pyenv. To create the virtual environment, we need to specify the pyenv-installed version of Python and a name for the virtual environment, which will name the directory in which the virtual environment is located. As an example, for Python version 3.7.3 and a virtual environment that we would like to name "music_recommender" after the name of our project, we run the following command:

Our virtual environment is now created. To initialize the virtual environment and associate the directory with the specific virtual environment, run the command (while in the project directory):

This command specifies that the local directory should be associated with the Python version that the virtual environment music_recommender specifies.

Great! We've set up the Python version and also activated a virtual environment for our project (i.e. music_recommender). Let's perform some final tests to make sure that our installation worked correctly. First, let's check that the Python version we specifically installed for the project is being correctly identified. Run the line of code below in the project directory; the output should be Python 3.7.3.

Next, let's make sure that a .python-version file is in our directory, as that is what pyenv depends on to identify the correct Python version. Run the following line in the project directory, and you should see the correct file listed.

This test is a little redundant, but it helps build intuition for what pyenv actually does and needs to work correctly.

If all of these tests passed, you've successfully installed, set up, and used pyenv to manage Python and package level dependencies. Awesome! Traditionally, the last step of setting up your Python environment is to install all the packages you will routinely need (i.e. numpy, sklearn, jupyter). Since we installed a totally new version of Python, and not a distribution like anaconda, you need to install all these packages afresh for the project. An easy way to do this is using a requirements.txt file, which contains a list of all the packages and versions needed for the project. You can read more about how to generate and manage requirements files here. I usually keep a requirements file with common packages laying around and use that to quickly start my projects. Using such a sample file in your project directory, you can quickly install a large number of useful packages using the following command:

That's it! You've installed the entire Python development environment and set it up to be locally isolated, yet easily shareable. For further reading on any of the topics, or on package management or Python management, check out the following links: