- Data models change and evolve with your application.
- There’s plenty of tools that keep track of database schemas and automatically generate scripts to upgrade or downgrade them.
- It’s common for developers to run a migration at the start of their app before running app code.
- Our author explains two common problems with this approach.
- Modern day production deployments and horizontal scaling can get you into a race condition.
- You start assuming that new code will only ever run with the new schema.
- You can decouple migrations from code changes by disabling parallelism during this time.
- Make it a separate command or lock the database during the upgrade.
- We can easily implement locking ourselves in any language.
- Use Redis locks if you’re ok with something external to the DB.
- Use the DB itself by writing to an extra table to say that you’re upgrading it.
- Plan your deployment appropriately so you can run old code with new by making migrations additive in the short term.
- Using a script at startup that optionally performs the migration based on an environment variable integrates wel with Docker and cloud services.
- Upgrades of both code and data should be part of your testing BEFORE releasing to production.
- It’s up to us to learn how to apply design patterns and industry trends to our projects.
- What’s an Object Relational Mapper (ORM)?
- Why do I want to use one?
- Unexpected quirks of using ORMs. What are the tradeoffs?
- What about scalability?
One of the critical pieces in a build system is the ability to view build and test output. Not only does it track progress as the build transitions through the various phases, it’s also an instrument for debugging.
This chapter in the continuous builds series covers how to build a simple log viewer. You’ll find details on retrieving log entries from Docker containers, serving them through Python, linking from a GitHub pull request, and highlighting the data for easy reading.Continue reading
Continuing the build service discussion from the Designing CI/CD Systems series, we’re now at a good point to look at reporting status as code passes through the system.
At the very minimum, you want to communicate build results to our users, but it’s worth examining other steps in the process that also provide useful information.
The code for reporting status isn’t a major feat. However, using it to enforce build workflows can get complicated when implemented from scratch.Continue reading
9 Organizational Test Practices Guaranteed to Lower Quality and Customer Satisfaction
A contribution to freeCodeCamp.org examining test practices opposite to our usual goals.
The most crucial step in any continuous integration process is the one that executes build instructions and tests their output. There’s an infinite number of ways to implement this step ranging from a simple shell script to a complex task system.
Keeping with the principles of simplicity and practicality, today we’ll look at continuing the series on Designing CI/CD Systems with our implementation of the execution script.Continue reading
As internet standards evolved over the years, the HTTP protocol has become more prevalent. It’s easier to route, simpler to implement and even more reliable. This ubiquity makes it easier for applications that traverse or live on the public internet to communicate with each other. As a result of this, the idea of webhooks came to be as an “event-over-http” mechanism.
With GitHub as the repository management platform, we have the advantage of using their webhook system to communicate user actions over the internet and into our build pipeline.Continue reading
Every code repository is different. The execution environment, the framework, the deliverables, or even the linters, all need some sort of customization. Creating a flexible build system requires a mechanism that specifies the steps to follow at different stages of a pipeline.
As the next chapter in the Comprehensive CI/CD Pipeline and System Design series, this article examines which instructions you’ll want to convey to your custom system and how to parse them. The focus is around a common solution, adding a file into the repository’s root directory that’s read by your execution engine when receiving new webhooks.Continue reading
There are many solutions to building code, some of them are available as cloud services, others run on your own infrastructure, on private clouds or all of the above. They make it easy to create custom pipelines, as well as simple testing and packaging solutions. Some even offer open source feature-limited “community editions” to download and run on-premises for free.
Following is my experience on the important aspects to consider when deciding on a build system for your organization that depends on cloud services. The original intent was to include a detailed review of various online solutions, but I decided to leave that for another time. Instead, I’m speaking more from an enterprise viewpoint, which is closer to reality in a large organization than the usual how-to’s. We’re here to discuss the implications and the practicality of doing so.Continue reading
Continuous integration and delivery is finally becoming a common goal for teams of all sizes. After building a couple of these systems at small and medium scales, I wanted to write down ideas, design choices and lessons learned. This post is the first in a series that explores the design of a custom build system created around common development workflows, using off-the-shelf components where possible. You’ll get an understanding of the basic components, how they interact, and maybe an open source project with example code from which to start your own.Continue reading
I’ve been using some form of a database throughout the entirety of my career. Sometimes single-file databases, sometimes full servers. Sometimes testing them, sometimes designing them, but a lot of times I was optimizing them. Even when learning programming with my dad, most of the apps I built were about storing and managing some type of data.
With all those years of experience, I definitely understand enough to know that I’m by no means an expert at any of it. There are several (an understatement) mechanisms by which folks make the best use of their database servers, almost all of them are tradeoffs in memory usage, space, look-up times, results retrieval, backup mechanisms, etc.
As expected, each database mechanism has their own quirks and optimizations, but the common theme is the language which you use to retrieve information: Structured Query Language (SQL). Different database engines implement different extensions to this language, some of which add powerful functionality, some of which just add confusion. But in general, SQL has been very successful in standardization across the industry.
This next chapter in the Practicality Beats Purity series covers the tradeoffs when using direct SQL queries to a database vs programming language abstractions that do it for you, like ORMs.Continue reading
Ying Li’s PyCon2018 keynote discussed the importance of writing secure software, and the responsibility that we, as developers, have in keeping users safe. While watching, it occurred to me that I haven’t written about the new European Union laws that stem from the General Data Protection Regulation (GDPR) that goes into effect this month.
I was involved in several conversations on this topic lately, so here’s some information on the rules, their implications and responses from businesses, and the reactions as the rest of the world tries to implement similar systems.Continue reading
In recent years there’s a growing trend to move away from large all-in-one applications. These “monoliths”, developed with one codebase and delivered as one large system, are hard to maintain. In their place, the industry now favors splitting-off the component systems into individual services. As separate “microservices”, they perform the smallest functions possible grouped into logical units. They are independent deliverables, deployable, replaceable and upgradeable on their own.
Going further into the Practicality Beats Purity series, this article will cover the implications of transitioning to a microservices architecture.Continue reading
Continuing on the Practicality Beats Purity series, today we’re talking about modularity. While written with python in mind, the discussion here applies to any language that’s highly modular and with a large ecosystem.
As is touted frequently, python is quite famous for being a “batteries included” language with a vast ecosystem of modules and packages that provide almost every possible utility or function you’ll ever need. When building large applications, it’s a great idea to make use of this environment and not reinvent the wheel. This makes rapid development and prototyping real easy.
However, you must keep in mind that every new dependency added is one more variable that you have little to no control over. While you may not write the code yourself, there’s still cost incurred in keeping up with the most recent versions of your dependency and watching for security flaws and their respective fixes. It’s also important to pay attention to the size of the community around those dependencies, their interaction with other modules, responsiveness to reported bugs, and the size of supporting documentation both official (like read-the-docs) and unofficial (like stack overflow).
Following we discuss some of the costs.Continue reading
A few hours later, I find myself sitting in the “comforts” of my cubicle. The discussion replaying over and over in my head: “An interface with this behavior will integrate with most common language libraries, with no special client code”, I said. The response was: “But then it’s not a
I’ve spent many years of my career involved in buzzword dogma discussions. It’s present at all levels of software development, from basic principles, to scheduling, to implementation, its interfaces, its tests, the execution, the infrastructure that runs it and its release mechanisms. Most of the time, people lose track of why or what they are building in favor of claiming they are using some common buzzword, regardless of the effects on architecture, ease of use, customer experience or maintenance costs. My experience shows they don’t even know why the buzzword technology does things a certain way or why someone chose it in the first place. Factual or data-based counterargument results in an almost “religious” discussion and even shaming.
Given today’s ease of communication and the ability to share our experiences, it’s great that we try to educate other folks on the problems we typically face throughout our lives and careers. Especially the principles used in managing their solutions.Continue reading