Join Our Newsletter

Episode 3 - Decoupling Database Migrations at Application Startup

13 mins
  • Data models change and evolve with your application.
  • There’s plenty of tools that keep track of database schemas and automatically generate scripts to upgrade or downgrade them.
  • It’s common for developers to run a migration at the start of their app before running app code.
  • Our author explains two common problems with this approach.
    1. Modern day production deployments and horizontal scaling can get you into a race condition.
    2. You start assuming that new code will only ever run with the new schema.
  • You can decouple migrations from code changes by disabling parallelism during this time.
  • Make it a separate command or lock the database during the upgrade.
  • We can easily implement locking ourselves in any language.
    • Use Redis locks if you’re ok with something external to the DB.
    • Use the DB itself by writing to an extra table to say that you’re upgrading it.
  • Plan your deployment appropriately so you can run old code with new by making migrations additive in the short term.
  • Using a script at startup that optionally performs the migration based on an environment variable integrates wel with Docker and cloud services.
  • Upgrades of both code and data should be part of your testing BEFORE releasing to production.
Continue reading

12 Trending Alternatives for Distributing Python Applications in 2020

One of the more prevalent topics in the Python ecosystem of 2019 was that of packaging and distribution. As the year comes to an end, I wanted to put together a summary of the many paths we currently have available to distribute apps built with Python. Though some of these also apply to any language.

Whether delivering an executable, a virtual environment, your packaged code, or a full application, the following list includes both standard systems and some up-and-comers to keep in mind as we enter 2020.

Continue reading

Practical Log Viewers with Sanic and Elasticsearch - Designing CI/CD Systems

One of the critical pieces in a build system is the ability to view build and test output. Not only does it track progress as the build transitions through the various phases, it’s also an instrument for debugging.

This chapter in the continuous builds series covers how to build a simple log viewer. You’ll find details on retrieving log entries from Docker containers, serving them through Python, linking from a GitHub pull request, and highlighting the data for easy reading.

Continue reading

Command Execution Tricks with Subprocess - Designing CI/CD Systems

The most crucial step in any continuous integration process is the one that executes build instructions and tests their output. There’s an infinite number of ways to implement this step ranging from a simple shell script to a complex task system.

Keeping with the principles of simplicity and practicality, today we’ll look at continuing the series on Designing CI/CD Systems with our implementation of the execution script.

Continue reading

Easy Clustering with Docker Swarm - Designing CI/CD Systems

Building code is sometimes as simple as executing a script. But a full-featured build system requires a lot more supporting infrastructure to handle multiple build requests at the same time, manage compute resources, distribute artifacts, etc. After our last chapter discussing build events, this next iteration in the CI/CD design series covers how to spin-up a container inside Docker Swarm to run a build and test it. What is Docker Swarm When running the Docker engine in Swarm mode your effectively creating a cluster. Continue reading

Effortless Parsing of Build Specifications - Designing CI/CD Systems

Every code repository is different. The execution environment, the framework, the deliverables, or even the linters, all need some sort of customization. Creating a flexible build system requires a mechanism that specifies the steps to follow at different stages of a pipeline.

As the next chapter in the Comprehensive CI/CD Pipeline and System Design series, this article examines which instructions you’ll want to convey to your custom system and how to parse them. The focus is around a common solution, adding a file into the repository’s root directory that’s read by your execution engine when receiving new webhooks.

Continue reading

Comprehensive CI/CD System Design

Continuous integration and delivery is finally becoming a common goal for teams of all sizes. After building a couple of these systems at small and medium scales, I wanted to write down ideas, design choices and lessons learned. This post is the first in a series that explores the design of a custom build system created around common development workflows, using off-the-shelf components where possible. You’ll get an understanding of the basic components, how they interact, and maybe an open source project with example code from which to start your own.

Continue reading

Using Python to Interact with Docker Containers

“You should really be looking at Vagrant” — he said while I struggled with the keyboard, as if pressing the keys harder was going to magically make it work.

I had recently completed a utility — you know, one of those things that slurp in data from some black hole in a distant corner of the universe, marries it with the structure of a different time-space continuum, and magically spawns a pretty visual representation for mere mortals to easily consume — and I was having the hardest time getting all the pieces together for a demo on a laptop running that OS (yes, that one).

My buddy was trying to point me to a system that helps provision virtual machines from a simple configuration file. A solution which was becoming common in application distribution and deployment, as well as a method to standardize development environments.

Continue reading
© Copyright 2020 - tryexceptpass, llc