Right now I am dealing with forced machine refreshes, and I have to confront once again the dreaded "why is my python not behaving as expected?" issue, as complained in this small blog many times. So before I forget, let me remind myself of the kinds of Pythons I must deal with when developing. It might help you when debugging.
System python
This python is bundled with the distribution. You only deal with this in Linux but not in Windows. When it comes to the more recent system pythons, installing python modules for system management and dependency is not dealt with pip but with apt, in debian and ubuntu-based systems at least. So if say you need ansible in your new system, you don't pip install ansible, you sudo apt install ansible. Debugging system python would require looking into documentation into how it is abstracted.
User-installed/path-detected python
This is the python you have to deal with when using a different python version for your project. If you are dealing with ubuntu apt package management, you'd be faced with adding sudo apt repository ppa:deadsnakes which allows you to install multiple versions of python on your system. If you are using Windows, you are most likely just downloading python from the main website and adding the PATH manually. If you are going to have problems with this, you must wrangle with how you expose your python to be readable without calling a full path to the executable, such as if it is registered in your PATH variable or if it aliased in your bash resource config (.bashrc, .zshrc, etc). If you aliased your custom python path without using the PATH variable, you may want to disable that alias first; it was the cause of many headaches on my part.
Pipx executables
This is a different abstraction than say sudo apt install python-module. PIPX is separate to modules in a way that it allows you to run python binaries from packages. It is installed as a system python module, sudo apt install pipx && pipx ensurepath, and from it you install pipx-ready packages. These packages are then kept in their own virtual environment. You may have problems in this if say you are using a executable binary that is available as pipx and as a module in your package, and at some point in your project, you might be using one when you want the other instead.
Virtual-environment managed python (venv)
This is a package management abstraction for Python projects that allows you to isolate packages and dependencies between projects. Note that in a venv-python, the python it uses is actually symlink in .venv/bin folder. Now, when dealing with venv-pythons, there are a couple of things you must remember.
- Always activate the virtual environment, so that the venv is working as expected. Also exit the venv when no longer working on it, as an active venv can also be used outside the project directory, which can happen when you are traversing folders via command line.
- Always make sure that the .venv folder is being generated inside the project and not outside it. I've read that .venv folder outside the project is a common use case for data science projects, but not for software development projects. The venv outside the project directory is also a default behavior in Poetry, and must be configured to do otherwise. If you find module installs and versions not behaving as expected, you may consider checking if your venv is acting as one for all your projects, as when it is cached outside your project; and if your venv is active in your current terminal, which it needs to be when working on it.
Pyenv pythons
Pyenv is a python-version management system that works like Node Version Manager or Ruby Version Manager. To use pyenv-pythons after installing pyenv, you must install those versions as pyenv install version. Those versions are stored in a specific folder, and is exposed by exposing the shims path to the PATH variable. It seems simple enough, as it works like user-installed pythons as well. But the key thing to remember is that Pyenv can, set the local and global python while it is active. It can also create a shell of an active python. Depending on your scenario, it can make or break your workflow. Here are some stumbling blocks that I've experienced, which may cause some confusion on your part (in my case however, the fault was in my python aliasing).
- How to make sure that your pyenv-python is the one being used by poetry. I personally had to run a pyenv shell before running a poetry shell to deal with this. I have yet to test this out again if this was still necessary, because this was still a pain to debug.
- You can call the system python and a pyenv-python that use the same version, but you may stumble when managing packages for both, because again, system python uses the package manager instead of pip. So if pip is not working or does not exist or if a warning regarding an externally-managed python environment appears, you are most likely using the system python, which in this case, make sure you are using the pyenv-python or use apt install if you must install system python. Again, if pyenv uses system-python, pip install will not work.
Manually aliased/symlinked-python
This is just a variant of type 2, but I emphasize this because there might be an occasion when you yourself manually symlinked a user-installed python to replace the distribution bundled python, or aliased your own version in your shell resource config file. You may also install python-is-python3 package, which symlinks python to python3. It is tempting to go towards this route, since a lot of tutorials have this implicit assumption that the way you installed your python is the same way they installed your python, and they are using python to call the interpreter, when in your system, your python command is elsewhere or installed differently.
Unless you know what you are doing, try to not do this. Undo this if you run to any problems. Comment out your alias and double-check your errors before you fix your symlinks.
A lot of these problems can be solved by just being deliberate and verbose about what python interpreter you are using and where it is installed. Python3 -m pip install instead of pip install. Or /full/path/of/your/python3 -m pip install. Making these commands short and these versions abstracted and aliased is just adding another footgun to the dependency hell of python. And maybe, when developing, to not use the system python at all, and use a user-installed python, even when it is the same version as the system one. That should prevent the missing pip command when using pyenv and poetry. And always try the "whereis" and "which" command in linux; that should help to locate what python is being called.
I don't guarantee the notes above are right, but I hope that these notes can help you out of python misbehaviors as I did.