Making 3D Interfaces for Python with Unity3D

A while back I wrote a post on using with game engines as frontend interfaces. I want to enable rich interfaces in the Python ecosystem that are usable in virtual and augmented reality. Since then, I was able to build some basic concepts on top of Sofi and I’m here to share them.

Using Sofi and WebSockets to command game engines

Sofi works as a WebSocket server that can issue commands and handle events from a client. It’s written to simplify the use of frontend web technologies as interfaces to a Python backend. You can even package it as a desktop app with a desktop look and feel.

It functions by sending the user to a webpage with the basics needed to open a WebSocket client. This client then registers handlers that process server commands telling it how to alter the DOM or respond to events. All the logic resides with Python, the webpage is the user interface.

I wrote it with enough modularity such that the command and event structure is reusable for any technology. So it wasn’t much of a stretch to think that I could replace the webpage portion with a game engine. It was only a matter of finding one with easy access to WebSocket clients.

The game engine

I looked at Unity3D, Unreal and CryEngine. Of the three, the first two have the most expansive communities, and of those, Unity seems to have the larger ecosystem. While usually a matter of taste, I’m not here to argue which one has better graphics. At this point, I want simplicity for a proof of concept, and this came in the form of WebSocketSharp.

WebSocketSharp is a C# library available in Unity3D’s asset store. It provides client and server classes usable by any script in your project’s scene. I ran a few tests and it worked well enough for my needs. Mapping back to Sofi, this part performs the function of the JavaScript WebSocket client library that runs on your browser. Now, let’s wrap it so it understands how to talk to the Sofi WebSocket server, and make the equivalent of sofi.js.

Making the frontend

If you haven’t used Unity3D before, or any game engine for that matter, getting to “hello world” can be confusing. While not required to understand this implementation, I’ll go over some of the main concepts here. Please note that Unity has their own extensive documentation available for more details.

First thing to do when making a “game” — which is what I’m doing for all intensive purposes — is to create a project. Think of this as a package that contains all the assets you’ll need for rendering a world and its interactions. Almost everything in a project is an asset: textures, shaders, scripts, objects, etc.

Within a project there are scenes. Think of these like game levels, or scenes in a movie. It’s a collection of objects, scripts and camera views that render into an experience with a common objective.

Once there’s a scene, I can add the WebSocketSharp library as an asset. This makes it possible to import its WebSocket class into a script in which to do some magic. You can assign scripts to objects in the scene and trigger execution through the engine’s event loop.

Every scene has a default camera. Cameras, among other things, determine how to view the scene. This includes viewing angles, distance, render settings, lighting, etc. But since they’re objects themselves, they can also hold scripts. Which means I can stick a WebSocketClient script on the camera that loads immediately with the scene.

Unity scripts can schedule handlers in the main event loop. There are many game events for physical collisions or user inputs detailed in the docs. But I’m interested in those that run at initialization and on a regular basis. In our case, I’m looking for Update and Start. The former runs when rendering every frame, while the latter executes after loading. I also register a handler for OnDestroy to clean things up when exiting.

To configure the client, instantiate a WebSocket during the engine’s Start event. This requires the address and port of the server, as well as its own set of event handlers. I use OnOpen when a connection completes, OnMessage when receiving new data, and OnClose for disconnections.

It’s important to think about the event loop when working with Start. Message processing cannot happen here because it can block the scene from loading. So I made a command queue in the form of a List object that lives within the client class. The idea is to review that queue on every Update event — once every frame — and handle the commands then.

Besides message processing, the Update event is a good point at which to check for a connection. It’s easy to trigger a reconnect, allowing the scene to wait for a server to spin up if needed.

The engine gathers user inputs and queues them up, much like the earlier socket messages. So I add checks for those here as well, using the sendAsync() method to communicate them back to the server.

Communicating with Sofi

To talk a little bit about protocol, Sofi’s communication mechanism is very extensible. The socket server listens on a particular port expecting JSON data. Event messages have an 'event' key set with the name which the server uses to check for callbacks. If one is present, execution passes to that function, which can in turn use the dispatch() method to send a command. UI commands have a 'command' key to identify them and their other parameters. Except for the basics, I don’t hardcode any of this, so it’s very easy to add custom events and commands by making up new names without modifying Sofi itself.

The WebSocketClient can handle commands for spawning new objects, updating object properties, removing objects and subscribing to events. The types of objects are the very basic shapes: cubes, spheres, cylinders, capsules, planes and text. There are quite a few settings for each, including color, size, position and rotation.

Packaging

The other neat part is that Unity3D can produce binaries for all major platforms. This includes Windows, Linux, MacOS, Android, iOS, and otherx. So not only does this add 3D capabilities to Python, but it also includes multi-platform support. The build process itself is simple and directed by the game engine itself. Couple it with PyInstaller like in my previous article, and now you have only one executable for distribution, Python environment and all.

What does it look like?

For a demo of what it can do, have a look at my PyCaribbean presentation below. I run it at about 20:09.

I’ve posted the Python code on GitHub under the sofi-unity3d repo. The WebSocketClient implementation is located in the engine/Assets folder. It also includes the Unity3D project along with it. The full code will run through my slides for PyCaribbean and starts a listener for any #python tweet.

I also went off and built the binaries for Linux, Windows and MacOS, so you can download and play with it yourselves. When you run the engine, it will load an empty sky and grassy plane scene that sits waiting for a WebSocket server. This could be the Sofi instance in server.py or something of your own making that listens on port 9000.

What’s next?

Today we’ve explored the possibilities around creating game engine based “widget libraries”. Demonstrating that it’s very possible for a Python backend to provide the logic, while leaving the UI heavy lifting to the engine. It’s also portable cross-platform and distributable with little effort.

The next step on the game engine side is to provide practical widgets and assets. With this framework we can spawn any pre-existing game objects, meshes and scripts. It’s up to us to define what makes for good reusable components and interactions, and package them with the built binaries.

On the Python side, I’d like to provide an HTTP-based asset server. This allows folks to create custom assets in something like Blender, which is also Python-enabled, and load them into any scene. No need to touch the game engine.

I also want to build a world with that enables VR or AR on hardware like the Rift, Vibe, Hololens, etc. I have a DK2 at home, but haven’t looked into the making any projects with it yet.

For additional reading on how I made Sofi, have a look at the A Python Ate My GUI series.