tryexceptpass

Practical Log Viewers with Sanic and Elasticsearch - Designing CI/CD Systems

One of the critical pieces in a build system is the ability to view build and test output. Not only does it track progress as the build transitions through the various phases, it’s also an instrument for debugging.

This chapter in the continuous builds series covers how to build a simple log viewer. You’ll find details on retrieving log entries from Docker containers, serving them through Python, linking from a GitHub pull request, and highlighting the data for easy reading.

Continue reading

Painless Status Reporting in GitHub Pull Requests - Designing CI/CD Systems

Continuing the build service discussion from the Designing CI/CD Systems series, we’re now at a good point to look at reporting status as code passes through the system.

At the very minimum, you want to communicate build results to our users, but it’s worth examining other steps in the process that also provide useful information.

The code for reporting status isn’t a major feat. However, using it to enforce build workflows can get complicated when implemented from scratch.

Continue reading

Command Execution Tricks with Subprocess - Designing CI/CD Systems

The most crucial step in any continuous integration process is the one that executes build instructions and tests their output. There’s an infinite number of ways to implement this step ranging from a simple shell script to a complex task system.

Keeping with the principles of simplicity and practicality, today we’ll look at continuing the series on Designing CI/CD Systems with our implementation of the execution script.

Continue reading

Conveying Build and Test Information with Repository Badges

When you check out a repository on github, sometimes theres a little bit of flare at the top of the project that catches your eye. This bit of flare is called a badge and can be used to indicate build status, test coverage, documentation generation status, version support, software compatibilty statements or even community links to gitter or discord where you can find more help with the project. I used to think that badges were fancy fluff people added to their projects to make them seem more professional. Continue reading

Easy Clustering with Docker Swarm - Designing CI/CD Systems

Building code is sometimes as simple as executing a script. But a full-featured build system requires a lot more supporting infrastructure to handle multiple build requests at the same time, manage compute resources, distribute artifacts, etc. After our last chapter discussing build events, this next iteration in the CI/CD design series covers how to spin-up a container inside Docker Swarm to run a build and test it. What is Docker Swarm When running the Docker engine in Swarm mode your effectively creating a cluster. Continue reading

Awesome Webhook Handling with Sanic - Designing CI/CD Systems

After covering how to design a build pipeline and define build directives in the continuous builds series, it’s time to look at handling events from a code repository.

As internet standards evolved over the years, the HTTP protocol has become more prevalent. It’s easier to route, simpler to implement and even more reliable. This ubiquity makes it easier for applications that traverse or live on the public internet to communicate with each other. As a result of this, the idea of webhooks came to be as an “event-over-http” mechanism.

With GitHub as the repository management platform, we have the advantage of using their webhook system to communicate user actions over the internet and into our build pipeline.

Continue reading

Effortless Parsing of Build Specifications - Designing CI/CD Systems

Every code repository is different. The execution environment, the framework, the deliverables, or even the linters, all need some sort of customization. Creating a flexible build system requires a mechanism that specifies the steps to follow at different stages of a pipeline.

As the next chapter in the Comprehensive CI/CD Pipeline and System Design series, this article examines which instructions you’ll want to convey to your custom system and how to parse them. The focus is around a common solution, adding a file into the repository’s root directory that’s read by your execution engine when receiving new webhooks.

Continue reading

Best Tips for Running Enterprise CI/CD in the Cloud

There are many solutions to building code, some of them are available as cloud services, others run on your own infrastructure, on private clouds or all of the above. They make it easy to create custom pipelines, as well as simple testing and packaging solutions. Some even offer open source feature-limited “community editions” to download and run on-premises for free.

Following is my experience on the important aspects to consider when deciding on a build system for your organization that depends on cloud services. The original intent was to include a detailed review of various online solutions, but I decided to leave that for another time. Instead, I’m speaking more from an enterprise viewpoint, which is closer to reality in a large organization than the usual how-to’s. We’re here to discuss the implications and the practicality of doing so.

Continue reading

Comprehensive CI/CD System Design

Continuous integration and delivery is finally becoming a common goal for teams of all sizes. After building a couple of these systems at small and medium scales, I wanted to write down ideas, design choices and lessons learned. This post is the first in a series that explores the design of a custom build system created around common development workflows, using off-the-shelf components where possible. You’ll get an understanding of the basic components, how they interact, and maybe an open source project with example code from which to start your own.

Continue reading