tryexceptpass

Command Execution Tricks with Subprocess - Designing CI/CD Systems

The most crucial step in any continuous integration process is the one that executes build instructions and tests their output. There’s an infinite number of ways to implement this step ranging from a simple shell script to a complex task system.

Keeping with the principles of simplicity and practicality, today we’ll look at continuing the series on Designing CI/CD Systems with our implementation of the execution script.

Practicality Beats Purity - Modularity

Continuing on the Practicality Beats Purity series, today we’re talking about modularity. While written with python in mind, the discussion here applies to any language that’s highly modular and with a large ecosystem.

As is touted frequently, python is quite famous for being a “batteries included” language with a vast ecosystem of modules and packages that provide almost every possible utility or function you’ll ever need. When building large applications, it’s a great idea to make use of this environment and not reinvent the wheel. This makes rapid development and prototyping real easy.

However, you must keep in mind that every new dependency added is one more variable that you have little to no control over. While you may not write the code yourself, there’s still cost incurred in keeping up with the most recent versions of your dependency and watching for security flaws and their respective fixes. It’s also important to pay attention to the size of the community around those dependencies, their interaction with other modules, responsiveness to reported bugs, and the size of supporting documentation both official (like read-the-docs) and unofficial (like stack overflow).

Following we discuss some of the costs.

The Trusted Packaging Index

A proposal for funding PyPI infrastructure and development

A few days ago, I was listening to the latest episode of Talk Python To Me: Are we failing to fund Python’s core infrastructure?, which had a panel of guests from the Python Software Foundation, PyPI and Read The Docs. As someone that writes open source code, the topic of sustainability is always floating around in my mind. Being able to mostly work on the things that tickle my brain would definitely be awesome, but even if you had a fantastically successful project — which I don’t — it still is extraordinarily difficult to achieve.

I always wondered how organizations like the PSF made it all work, especially with infrastructure and systems that have the level of traffic we see in PyPI. The closest parallel I can draw is to research projects, where a considerable amount of time is dedicated towards finding the right kind of funding.