Distributing Python applications

At Retail Insight, we write several self-contained utilities and applications in Python, typically scheduled in our cloud environment, but they may also be run by users on-demand in their local machines. Distributing these applications is a demanding task due to the relationship between a Python application and its hosting environment.

This is a challenge I often face, so I wanted to chart my journey to explore solutions that will make this distribution simpler. In my case, the application must be deployed to several virtual machines in the cloud and installed on the local machines of multiple users, including Python developers, Analysts, and Product Managers.

As I explore the different options to remediate the challenges I face, I will be aiming to:

• Minimize the setup needed to get the application up and running
• Guarantee a uniform end-user experience
• Enable a smooth deployment process
• Address the mismatch issue between dependencies in the development vs. production environments
• Reduce the barrier to entry, making the application available to as many users as possible

Where to begin?

Due to the complexities of this project, one of the first things you must do is reflect on some critical questions before beginning, the answers to which will heavily influence your design decisions.

Where is your software intended to run?

Once you have confirmed the environment in which the application will run, you should ask some functionality questions including:

• Does it have a Python interpreter installed?

• If the interpreter is installed, is it a version type that the app supports?

• If the interpreter is installed and is the right version, are extra packages already installed that might conflict with your applications?

Who makes up your intended user base and how can they access and run the latest version of the utility?

Making the application available to as many users as possible is a challenge in Python due to the nature of establishing development environments. A multi-step process which involves:

• Downloading and installing the specific Python version required by the pip
• Setting up the pip and virtual environment
• Installing the application along with its dependencies

Now, after answering these questions, it is time to review the distribution landscape for a packaging solution.

Package Distribution Landscape

Selecting a Python distribution package is the first step; the landscape of options you can leverage is vast, yet, my main focus was across the following options:

Wheels: This is the most common approach used by developers whereby the code is packaged into a wheel binary; dependencies are specified as part of it and then pushed to an artifact repository. This approach caters to environments that have installed Python and a user base that knows how to install and manage dependencies.

Python Executable (PEX): A package used by Twitter to deploy Python applications since 2011; it packages your Python code and dependencies into a single executable file. This approach assumes that a Python interpreter is installed in the hosting environment.

Executables (EXE): Also known as ‘freezing your code’, it packages your application code and dependencies into a self-contained executable. The end-user is then able to run the packaged application without installing a Python interpreter or any libraries.

Containers: Container systems facilitate the distribution of a complete filesystem with everything you need to run your application and it requires installing a container system like a docker in the host environment.

Considering these technologies in order of their decreased dependency on the hosting machine would look something like the below. As you move to the right there is a diminishing dependency on an already installed Python interpreter and on installing the application’s dependencies.

In the end, I decided upon an Exe package, this ticks all the boxes and eliminates the setup needed.

Let us dig a bit deeper into the Exe landscape.

Exe

There are plenty of options available here – see table below – so in order to help me choose, I formulated the following questions:

• Does it support Windows, Linux, and macOS?
• Does it support one-file mode (i.e., single file artifact?)
• Does it support Python versions > 3.6, < 3.10?
• Is the project actively maintained? Did it have any releases in the past 6 months?
• Is it easy to set up and use?

After evaluating the type of solution, I landed on PyInstaller as the package of choice. It is actively maintained and with a good level of support. It supports Python 3, one file mode, and all three OS platforms. I have also found it easy to set up and use.

PyInstaller

What does it do?

How does it work?

Usage

I added the PyInstaller library as a dependency to the application’s pyproject.toml and also committed to both a .spec file, which is a Python file that can be customized for your build and a windows specific version_info.txt which will provide the version information needed for Windows applications.

I included the step of building the Exe – as below – as part of a GitHub actions workflow, thereby automating the whole process of testing, building, and publishing.

Command to generate an exe

Code breakdown:

• --onefile: Packages your entire application into a single executable
• --name: Changes the name of your executable and .spec file
• --hiddenimport: Imports the PyInstaller could not detect

Conclusion

As I reflect, my exploration has been a success. The application is running in a UAT environment and will be promoted to production soon! I managed to achieve all of my desired outcomes, but it was not all plain sailing – here are some key learnings I picked up along the way:

• PyInstaller extracts the Exe to a temp folder which can add an overhead to the start-up time of the application.
• Premature termination of the application requires a clean-up to remove the temporary folders created by PyInstaller.
• PyInstaller supports making executables for Windows, Linux, and macOS – but it cannot cross-compile.
• The size of the Exe is larger compared to the Python Wheels.

This article is also featured in Retail Insight's Medium publication.