There are many solutions to building code, some of them are available as cloud services, others run on your own infrastructure, on private clouds or all of the above. They make it easy to create custom pipelines, as well as simple testing and packaging solutions. Some even offer open source feature-limited “community editions” to download and run on-premises for free.
Following is my experience on the important aspects to consider when deciding on a build system for your organization that depends on cloud services. The original intent was to include a detailed review of various online solutions, but I decided to leave that for another time. Instead, I’m speaking more from an enterprise viewpoint, which is closer to reality in a large organization than the usual how-to’s. We’re here to discuss the implications and the practicality of doing so.Continue reading
When joining a new engineering team, one of the first things I do is familiarize myself with the dev and test processes. Especially the tools used to enforce them. In the past 5 years or so, I’ve noticed that a lot of organizations still use older tools that haven’t yet evolved to support modern practices. Even teams that purely develop software can find themselves working around cumbersome systems that hinder instead of enable.
What do I mean by that? Very few of these tools include useful interfaces to leverage integrations with other systems (like REST APIs). Most have no concept of modern dev practices like continuous integration or containerization. Almost all of them want to record pass / fail at a step by step basis as if you’re executing manually. The vast majority are built around a separation between test and dev (some even emphasize it). And a lot of them require the organization to hire “specialists” for the purpose of “customizing” the tool to the team. In my opinion, these types of systems coerce the organization to emphasize blame over quality and team boundaries over productivity.
I’ve been very successful at building long-lived alternatives to these systems in several organizations. I’ve done it enough to know which features are worth including, and which to leave to the test / dev engineers, especially after the advent of continuous integration and delivery.Continue reading
I’ve been using some form of a database throughout the entirety of my career. Sometimes single-file databases, sometimes full servers. Sometimes testing them, sometimes designing them, but a lot of times I was optimizing them. Even when learning programming with my dad, most of the apps I built were about storing and managing some type of data.
With all those years of experience, I definitely understand enough to know that I’m by no means an expert at any of it. There are several (an understatement) mechanisms by which folks make the best use of their database servers, almost all of them are tradeoffs in memory usage, space, look-up times, results retrieval, backup mechanisms, etc.
As expected, each database mechanism has their own quirks and optimizations, but the common theme is the language which you use to retrieve information: Structured Query Language (SQL). Different database engines implement different extensions to this language, some of which add powerful functionality, some of which just add confusion. But in general, SQL has been very successful in standardization across the industry.
This next chapter in the Practicality Beats Purity series covers the tradeoffs when using direct SQL queries to a database vs programming language abstractions that do it for you, like ORMs.Continue reading
PyBites: How Promotions work in Large Corporations
A collaboration with Bob and Julian from PyBites on soft skills.
The release of Python 3.7 introduced a number of changes into the async world. There are a lot of quality-of-life improvements, some affect compatibility across versions and some are new features to help manage concurrency better. This article will go over the changes, their implications and the things you’ll want to watch out for if you’re a maintainer. Some may even affect you even if you don’t use asyncio.Continue reading
In recent years there’s a growing trend to move away from large all-in-one applications. These “monoliths”, developed with one codebase and delivered as one large system, are hard to maintain. In their place, the industry now favors splitting-off the component systems into individual services. As separate “microservices”, they perform the smallest functions possible grouped into logical units. They are independent deliverables, deployable, replaceable and upgradeable on their own.
Going further into the Practicality Beats Purity series, this article will cover the implications of transitioning to a microservices architecture.Continue reading
Continuing on the Practicality Beats Purity series, today we’re talking about modularity. While written with python in mind, the discussion here applies to any language that’s highly modular and with a large ecosystem.
As is touted frequently, python is quite famous for being a “batteries included” language with a vast ecosystem of modules and packages that provide almost every possible utility or function you’ll ever need. When building large applications, it’s a great idea to make use of this environment and not reinvent the wheel. This makes rapid development and prototyping real easy.
However, you must keep in mind that every new dependency added is one more variable that you have little to no control over. While you may not write the code yourself, there’s still cost incurred in keeping up with the most recent versions of your dependency and watching for security flaws and their respective fixes. It’s also important to pay attention to the size of the community around those dependencies, their interaction with other modules, responsiveness to reported bugs, and the size of supporting documentation both official (like read-the-docs) and unofficial (like stack overflow).
Following we discuss some of the costs.Continue reading
A few hours later, I find myself sitting in the “comforts” of my cubicle. The discussion replaying over and over in my head: “An interface with this behavior will integrate with most common language libraries, with no special client code”, I said. The response was: “But then it’s not a
I’ve spent many years of my career involved in buzzword dogma discussions. It’s present at all levels of software development, from basic principles, to scheduling, to implementation, its interfaces, its tests, the execution, the infrastructure that runs it and its release mechanisms. Most of the time, people lose track of why or what they are building in favor of claiming they are using some common buzzword, regardless of the effects on architecture, ease of use, customer experience or maintenance costs. My experience shows they don’t even know why the buzzword technology does things a certain way or why it someone chose it in the first place. Factual or data-based counterargument results in an almost “religious” discussion and even shaming.
Given today’s ease of communication and the ability to share our experiences, it’s great that we try to educate other folks on the problems we typically face throughout our lives and careers. Especially the specific principles used in managing their solutions.Continue reading
I spend most of my day, every day, knee deep in code. Optimizing, building, fixing and thinking through workflows can be taxing. This means that the last thing I want to do when I come home is deal with more programming. But I also like learning new things and communicating my experiences so they can help others. I do that through the posts in this website.
Maintaining a web presence without dealing with code means you get to use as many off-the-shelf components as possible. You consider things like WordPress or static site generators that let you concentrate on content, while handling the user interface for you. Write in markdown, build the website, rinse repeat with updates. It’s all very easy, until you need a little more interactivity, like a comments section, or a newsletter signup.Continue reading
Python added formal asynchronicity in the base language a while ago. It’s fun to play with asyncio
coroutines, the basic constructs that execute almost in parallel. But as you start to integrate more with a regular codebase, you may find that things can get tricky. Especially if you’re forced to interact with synchronous code.
The complication arises when invoking awaitable functions. Doing so requires an
async defined code block or coroutine. A non-issue except that if your caller has to be async, then you can’t call it either unless its caller is async. Which then forces its caller into an async block as well, and so on. This is “async creep”.
Using Sofi and WebSockets to command game engines
A while back I wrote a post on using with game engines as frontend interfaces. I want to enable rich interfaces in the Python ecosystem that are usable in virtual and augmented reality. Since then, I was able to build some basic concepts on top of Sofi and I’m here to share them.
Sofi works as a WebSocket server that can issue commands and handle events from a client. It’s written to simplify the use of frontend web technologies as interfaces to a Python backend. You can even package it as a desktop app with a desktop look and feel.
It functions by sending the user to a webpage with the basics needed to open a WebSocket client. This client then registers handlers that process server commands telling it how to alter the DOM or respond to events. All the logic resides with Python, the webpage is the user interface.Continue reading
Packaging and distributing your app sounds simple in principle. It’s just software. But in practice, it’s quite challenging.
I’ve been working on a Python module called Sofi that generates user interfaces. It can deliver a desktop feel while using standard single-page web technologies. For flexibility, I designed it to work through two methods of distribution: in-browser and executable.
Running in the browser, it functions much like a normal webpage. You can load it by opening a file, or launch it from your shell. I also built an executable that runs as a packaged app, independent and without external requirements.
Over time, as I hacked at code in Atom — my editor of choice these days — I remembered that Atom is actually a browser. It uses Node.js as a back end, and the Electron framework for its user interface.This inspired me to start poking at Electron’s internals, hoping to find examples and best practices on how they solved desktop packaging.Continue reading