When I started making The Moonstone Equation it was just a test-bed for ideas and a place to try our new things; it didn’t even have a name. It was years into development before I considered making it a commercial product and because of this it’s ended up being created from the ground up on a custom engine, written in C++ using minimal external libraries. If you’re the sort of developer that thinks this is a good way of doing things then this page should serve as some sort of warning to you. Or at the very least give you some indication of what you might be getting into.
You should also be warned that this is a really long and occasionally technical article. But, if you can’t get to the bottom of this page then you’ve got very little hope of actually doing the work it describes.
Where to begin
So you’ve decided to write a game and engine from scratch without using any of the hundreds of great libraries or dozens of existing engines. Good for you. Like me you’re probably doing this to learn more about game development (and programming principles) along with the various bits of code and technology required to make everything work. First thing to decide is where you’re going to draw your distinction for ‘no external libraries’ because unless you intend to start writing drivers you’re going to have to use at least a few libraries. In my case I chose to use some of the C standard libraries like stdio.lib (because I’m old) along with Direct X, XInput, XAudio and a few helpful windows libs like PSAPI and WinInet but that’s about it. Obviously I could have chosen a different minimal set and the inclusion of all that Direct X already represents quite gap between me and the actual hardware but it’s also a very very long way from what’s required to make a game engine (and game); as you’ll see below.
So off we go.
The very basic stuff
Now it’s clear that you’ve decided to do things the hard way then you’ve definitely decided not to use any of the helpful C++ functional libraries or any of C++ standard templates and containers. This sort of thinking is actually very common in the games industry. In most cases we decide our additional context or domain knowledge and specific target platform decisions mean we can do a better job for the smaller set of cases we actually have to cover. This isn’t always completely crazy. For instance, the standard templates are very generic, very well tested and often very useful but they’re also pretty ugly, difficult to debug and often do things that make games developers teeth itch; like allocate memory in unpredictable (or annoying) ways. That’s okay for you though, you’re making everything yourself. So what will you actually need?
Bear in mind here that I tried to just make the things I actually had a use for but occasionally some interesting shiny problem would catch my attention and I’d spend a few days implementing something obscure just to see how it worked. You might spot a few of these as you read.
Basic global functions
You’re not using those helpful libraries so you’re going to need to implement some basic universally useful functions in order to implement anything even a little bit complex. These are the sorts of functions that you see in computer science courses and you’ll find to be popular choices for interview questions for programming jobs so it’s good stuff to know about anyway.
Bit Twiddling
Lets start with the ones and zeros. There are tons of uses for functions that mess about with individual bits and better still they can all be quickly tested for all possible inputs to make sure they’re working (unless you’re doing everything in 64Bit). They’re also an endless source of distraction; my inner micro-optimiser loves nothing more than to stare at a ReverseBits function and wonder if it could be done in less cycles. At the very least you’ll need a bit swapper, counter, setter and getter along with functions calculate the next power of two and calculate log in base 2 of a given integer. You might also want to implement grey code conversions and and integer crushing while you’re at it.
Hash Functions
Even though I’m not talking about the cryptographically secure hashing functions you might use to protect your bank account here it’s worth noting that this field is crammed with people that can and have definitely done a better job than you could. These things should be approached for study, don’t go building big bits of your engine on top of your own hand-rolled hashing function unless you want to deal with some super obscure bugs in later life. Understand and implement an existing (non-patented) algorithm and stick with it.
Strings
You’re going to be asking lots of questions about parts of pieces of text so it’s good to get a strong (and well tested) set of string handling functions written. String processing is so common that the more different functions you can think of the better. You can even get distracted and spend days optimising and tweaking it all too if you like. I think these might be my inner micro-optimisers second favourite source files. Also, I’d suggest you stick with Unicode with occasional char conversions as needed (because some old function) you’ll thank yourself later. I opted for some obscure stuff along with lots of basics like ToUpper, StringLength, EndsWith and Concat but you can go as crazy as you like.
Data
If you’re not doing maths and logic or looking at strings then you’re probably moving blocks of data about. You’ll need some classes to contain and manipulate contiguous blocks of bytes. Initially you’ll just need to make them, fill them empty them and copy them about. Later on you’ll be doing a lot more. At this point you should also create the functions that allow you to convert ints, floats bools and all that into steams of data and back again.
Timers
A general purpose set of classes or functions for measuring the number of clock-cycles between two points in time is essential; either because you want to check that those high performance functions you’ve written really are or to control how much time different parts of your engine spend doing work. We’ve had high resolution timers available on our PCs for a long time now so they’re definitely the place to start. However, it’s worth doing a little reading to make sure you understand the effect of using such timers on modern chips that have multiple cores that may or may not all be running at the same speed as each other.
Basic maths
If like me you’ve opted to include Math.h then you already have most of the simple maths functions you’re going to need; many of which talk directly to the CPU and have very little scope for improvement or variation. There are a few places you might want to extend on the basics or write some more complex functions to solve specific problems. I’ve opted to write a quadratic and cubic solution finder (using the obvious methods) as well as domain specific high performance approximation functions for things like ExpBelowZero (which can calculated about 10 times faster than standard exp) or SinCos (calculating both at once within -PI to PI can be done very quickly)
Random numbers
Most games need lots of (pseudo) random numbers and your’s is probably going to be no different. You could just use the rand function but that’s really not in the spirit of writing everything yourself (it’s also pretty terrible). Much like hashing functions this is not an area to approach lightly. Lots has been (and still is) written about generating random numbers so you might find it much more useful to base this critical system on an existing (non-patented) algorithm instead of rolling your own. I’d also suggest avoiding the really big ones like the mersenne twister unless you really have a huge need for quality numbers over performance. Obviously you could write more than one. It’s also worth making sure that when needed you can control how your RNG is seeded; not only will this help you hunting bugs it will also make it easier to write procedural, network or replay code.
Containers
Humans make mistakes. Despite occasional evidence to the contrary, programmers are human. So I always try to write my code in a way that defends my future self from my own errors (or potential errors). I achieve this in a few ways. First there’s the testing, simply writing something and walking away assuming it’s all fine usually ends badly. Then there’s things like exceptions and asserts to make it clear when I’ve violated my original design assumptions or my past self did something stupid; he’s done that a lot. Finally, the part that’s relevant here, I try and make things that avoid the possibility for me to make stupid mistakes. If you’re going to spend years making your new engine and game then you might consider doing the same and a good place to start is your nice new container classes.
Arrays
Having at least two array classes, one pre-allocated and one that auto re-allocates is a good place to start. Your future self will thank you for having a simple array that clear’s up after itself and handles range checking instead of leaking or overflowing. Obviously you can spend as much time as you like considering important things like cache alignment, thread safety or what strategy to use for array growth; I used 1.5 but some people like doubling and others prefer a little bit of fibonacci, it all depends on expected usage patterns.
Lists
I love the intrusive linked list. It’s gone out of fashion in the last 20 years but for me it’s useful properties far out weigh it’s negative points (terrible cache behaviour or reliance on things being derived from a base list-node class often come top of that list). If this thing is new to you then great, that’s why you’re writing everything yourself right? In it’s simplest form an intrusive list is made up of classes that contain the list referencing machinery inside them; usually in the form of a next (and if you like previous) pointer. Once you’ve allocated a pile of list-node derived classes then these containers come with constant time insertion and removal, linear time traversal and zero extra allocations; all good stuff.
Dictionary and PriorityQueue
Now you’ve got the simple containers you can move on to the more tricky containers. The design of these is sufficiently complex that people continue to write research papers about their implementation. Once you’ve done some reading and gone straight to implementation you might find you have to re-write these a few times until you’re happy with them. The main benefit of doing these yourself is the huge amount of understanding you gain about their implementation. You also get to impose constraints on their design that allow for some domain specific optimisation.
Strings
Everyone needs a string class (probably). Now you’ve written all those helpful string functions and containers this is probably the first point where you can mix a few together to create a useful new class. The first thing you might want to consider in your new class that significantly differs from the standard strings is their immutability and behaviour for small units of data (a common use case). In my case I opted to make my string class use memory from a pool of 64 byte blocks unless it’s contents required more than about 28 letters (leaving some bytes to store length and hash) I also made then mutable (allowed their contents to be changed in place). This meant for 95% of my use cases there was no heap allocation when working them and simple work was nice and fast. Obviously you’re making your own game and engine so you can do what ever you fancy.
Magical pointers
A fun place where many programmers like to make their lives difficult is in the handling of references to locations in memory. If you like you can just rely on your own fleshy brain to remember all the places you stored pointers to objects with potentially dynamic life-times; or, you could do the sensible thing and implement some wrapper classes for common pointer behaviours. At the very least I’d suggest you need an unique-pointer (something that only one object can own) and a shared-pointer (something many objects can own). As with many things you should do some reading here and about 300% more testing than you think is appropriate.
The less basic bit still universal stuff
So now you’ve got the critical things written you’re on your way to developing your engine. The next thing you might consider doing is the other universally useful stuff that’s not necessarily game orientated but forms the back-bone of many game engine systems. We’re starting to move very distinctly into territory that’s coloured by the decisions I made when developing my engine. You can (and should) approach these things in any number of different ways.
Threading
Almost everyone that plays your game is going to do so on a device (in my case PC) that has more than one processor running on an operating system that supports multi-threading. So making some effort to use all that other processing power is probably a good idea. As threading makes everything just a little bit more complicated there are a few things you’re going to need.
Atomics
Adding values, assigning things and comparing values are all open to complex interdependence when threads get involved. To reduce the risk of thread bugs driving you insane I’d suggest some wrappers for basic atomic operations. For instance I have some atomic counters and classes to call functions under conditions defined by atomic operations along with a few atomic math functions.
Synchronisation
There are some times when you need to do a little more than assign and test an integer. Thread synchronisation primitives are there for those occasions when you simply have to talk between threads and as such need some way to mediate what’s going (so everything doesn’t just explode). This is another area where you should take significant caution. Even after years of working with threads they still occasionally catch me out. In the end I only needed to implement simple wrappers for the critical section and semaphore.
Job Handler
For threading bonus points you might find it helpful to make a generic handler for work you want doing on another thread. There are lots of occasions you might want to just throw work onto another processor and get the result when it’s done. For that you’d benefit from a general purpose threaded job handling system capable of picking work up from a pending queue, handing it over to a worker thread to deal with and informing the caller when it’s done. You’ll probably find it useful to allow for jobs made of many sub-jobs with potential for dependency between them. Even if you are an experienced software developer with lots of threading experience you should expect to find bugs in such a complex piece of software for the next 5 years (despite any amount of testing)
Low level file handling
This is the bottom level of the more structured file/resource handling system. The point of contact between the engine and the operating system for gathering data stored on disks either locally or perhaps across a network. It’s also another fun place to worry about performance.
Paths
It’ll not take you long to realise that file paths are like an extra complicated string handling problem. At that point you’ll realise you need a specialist path class to contain all the extra functionality used to manipulate what is basically just a string. Things like RemoveExtension, MakeRelative and Exists. In the end I found my simple path class needed 26 public functions.
Raw file access
You’re doing everything yourself so your only option is the basics fopen, fread, fseek, fwrite and fclose. Actually that’s close but not quite true. Some of those functions have some undesirable security characteristics so you should really be using fopen_s and fclose_s. Once you’ve wrapped these in a nice class and tested it endlessly you can move on to more significant things.
Maths structures
Your game is probably going to depict graphical objects moving around in some sort of space (I hear many do) so you’re going to benefit from some structures that relate to more complex mathematical objects like vectors and matrices. Again, there are definitely very functional, fast, well tested maths libraries out there but at least by creating some yourself you’ll grow to understand why you should usually never create them yourself, at least not entirely.
Vectors
The common approach is to create some 2d, 3d and 4d float and integer vector classes, fill them with tons of operator overloading and helpful functions and leave it at that. That’s exactly what I did because it makes game development easier each day. What it doesn’t do is make high performance code. Worse still, some people then decide the best route to improve each of their classes is to replace the individual members they simply called x,y and z with a SIMD primitive instead. If you want high performance vector operations don’t make tons of little classes. instead understand your problem domain and make big parallel blocks of vectors to suit. Anyway, rant over, back to making your game engine.
Matrix and Quaternion
When it comes to the next level of math classes you need something reliable and understandable. If you can’t verbally explain to another person how a quaternion works then you should probably not be writing a maths library. After spending some significant time writing things myself and finding occasional weird edges cases that didn’t work correctly I opted to throw the whole lot away and wrap the DirectXMath matrix and quaternion functions in a simple interfacing class and walk away; you might want to consider doing the same.
Script Reading
You’re not always going to want to read files stored in a binary format. A great deal of power comes from being able to read and interpret text files. This is where all this string functions get some use. When writing your script reading classes you should bear in mind that structured text files can be much more easily edited by fleshy error prone humans and are therefore much more likely to contain mistakes. Obviously binary files can also contain errors that need to be handled, but the nature of parsing makes the task of error handling in a text file a little bit more complex. As you’ll see when you implement it.
Tokenizer
Step one on the path to understanding text files is the creation of a tokenizer. This turns a stream of characters into a chain of objects with context information about things like it’s type and value. You could even create a general purpose tokenizer that takes some sort of syntax definition file (perhaps in Backus-Naur form if you like). I opted to just break the text file down into tokens using some common rules; the result is a list of tokens marked as int, strings, brackets, operators etc. For maximum flexibility you’ll probably want to read up on the many and various ways in which things like numbers can be described in a text file (because -.1e4f is a valid float syntax in many languages for -1000)
Parser
Once you’ve created your robust tokenizer and are happy it’s capable of dealing with all sorts of input you can move onto process of turning that into data inside the game engine. You could write an XML or JSON or just do as I did and write some helpful functions to read values and understand structure markers like brackets, semicolons and line-ends.
More
As you’ll see later there’s a great deal of power that comes with a reliable script reader.
Compression
I should start by saying that this is an absolutely non-essential feature of your engine (or mine) but it is a source of some very interesting (and very distracting) problems. It’s also an easy way of accidentally violating some patents so you should take a little care if you intend to implement any sort of compression. Interestingly, unlike hash functions and random number this is a great area to attempt to roll your own stuff. There’s very little risk and the worst thing that can happen (assuming you code is still tested and reliable) is that your files are all exactly the same size as they already are (and you wasted a few CPU cycles).
Compress
There’s lots of literature available about all sorts of methods for compressing data and if you want to write your own then you’re also going to have to do a lot of reading. There are lots of options ranging from very simple things like run-length-encoding through huffman encoding to horrors like adaptive arithmetic encoding. Obviously if you choose to implement something you don’t understand your future self will probably not thank you for it.
Encode
I’m mentioning encoding separately from actual compressors because they’re have different jobs. Encoders are primarily tasked with reducing the entropy of a block of data without necessarily making it any smaller. A few good encoding steps can often turn your average compressor into something pretty good. Again this is an area where you can roll your own code (as the results are easy to test and measure) but you’ll also want to implement some classics like the Burrows-Wheeler-Transform and the Move-To-Front transform.
Package files
Knowing that you have a single file location to go to for all your data is quite useful. It also makes deployment a lot easier. First thing you’ll have to do is extend your basic file handling to deal with files that are potentially stored inside larger files (which may now be compressed). Simple implementations of this are just big package files (all the content of all the files bunched together in a single file) with indexes of contents at the start. These can either be read when files are requested or read when your engine starts up. In either case you simply use that information to decide weather to read a file from a package or a loose file when it’s requested based on those indexes. I opted for variation of this method. I knew I didn’t need lots of data so I read all the packages in when the game launches and unpack them as needed, basically creating a file system located in memory that’s checked before disk. Oh, and don’t forget you’ll need to write some tools to create those packages.
Memory tracking
As your engine and game grow in size you are going to start leaking memory. I say that with certainty because I’ve never seen a situation where every programmer perfectly accounts for all their allocations 100% of the time; unless you’re some sort of memory allocation savant. That’s okay, you just need some tools to ensure that you can spot when it happens and fix it. That’s where you need a memory tracker. This is a piece of code that notices every single memory allocation and de-allocation and keeps track of its source and lifetime. It can be quite slow and memory intensive so it’s probably not something you want running in your released code. In my case I start the memory tracking up before everything else (including the initialisation of static objects) then analyse the results as the application closes and display a report in the debug window; simple but effective.
The Engine
Okay you’re doing pretty well; you’ve got some basic globally useful systems and you can start to think about the architecture of your engine. This is so subjective and there are so many ways to approach the problem that my only option is to tell you what would be required to make my engine. So, from now on I’ll assume you’ve curiously decided to make exactly the same design decisions as me, that may or may not be a good thing. Luckily I’ve made a fairly generic PC-centric engine so many of the things I talk about below can apply to other designs.
I’ve defined the distinction between the engine and everything else as the point at which systems and classes are designed to be part of a complete whole that supports the creation of a game rather than separated parts that could be used in lots of different applications. There’s much more cross-dependency within the engine.
Resources
You’re not going to get very far without data, usually in the form of graphics and audio or scripts/binaries. You’ll quickly realise that simply calling a simple open-read-close style ‘loadBlah’ function when you want something is going to be horrible to use and add lots of stalls to your game when ever it happens. Instead you could make use of all that thread code you wrote earlier along with some of those containers to create a streaming resource system that supports lots of different type of file. Obviously you’ll need to handle the possibility that some resources rely on others and most need some sort of fix-up function once all that loading is complete, that should also be threaded. You should also make sure it’s easy to add new resource types as your engine grows, at the moment you’ve not got a renderer or a sound system but you’ll need to feed them both with data once they’re created. So what’s the minimum set of file types to support? That depends what you’re making. I didn’t need to load 3d models but I did need textures, audio, shaders, general scripts and binary data. The file handling for each of these is detailed in the sections below that relate to the systems they feed.
Windows
This is it. The point at which you’ll be able to compile your code, press run and see a window appear. Well it will be once you’ve understood all the weirdness of the windows message pump and spent at least a few hours cursing the smell of 20+ years of legacy code its functionality is clearly encumbered by.
Startup
As with everything else here I’m not going to detail the step by step process you’ll need go through to open a dialog window. There’s tons of exciting documentation littered around the rest of the internet for all that. However there is one very important thing you should ensure when you’re writing your window code. It needs to be responsive; all the time. It should appear as soon as absolutely possible, it should handle size changes quickly and it should close as soon as the player requests the game to quit. If you’ve ever been annoyed by how long a game takes to open its window or how long an application takes to quit you’ll understand how important this is. Don’t just open the window and present a black screen either, that’s just as bad.
Message Pump
Prepare to scour the internet for the best way to handle the windows message pump. Also prepare to find a lot of bad information. This is one of the more esoteric parts of the window handling process and you’ll probably be fiddling with it for years. Don’t forget about suspend and resume either. In these modern times more people have a laptop that they just fold shut when they want to pause for a while. That’ll suspend your game and unless you handle the resume you’ll find that everything is very very broken.
Mouse and Keyboard
It’s within your windows message pump that you’ll receive a lot of the information you might want about the activity of the mouse and keyboard. You’ve not yet written your input handling but now would be a good time to add the functions you need to transmit that mouse and keyboard info to other parts of your engine. You might also consider disabling the windows key while your game is running (and has focus), there are not many gamers that enjoy accidentally pressing the left windows key at some critical. Also, If you intend to make lots of use of the mouse you should also spend some time implementing high resolution data capture from the mouse device because getting an event only when the mouse pointer moves from one pixel to another is actually terrible for camera controls.
Audio
Even though you’re making everything yourself you still have to use some significant libraries if you want to use some devices. XAudio is about as low as you could go without being an audio engineer. Even then you still to have to do a lot of work yourself.
Busses
Once you’ve done lots of reading and have a second type of ‘bus’ in your vocabulary (as in, not the vehicles) you can start to create the framework classes for getting raw audio data from some source, through some mixers and effects to some configuration of speakers. This is another great place to make use of all that threading code you wrote earlier, because unless you want lots of pops and pauses you’re going to have to feed those busses constantly using threads. This will give you a new appreciation of the efforts that go into sound engine design.
Simple Audio
Time to make use of that threaded resource loading. Initially you’ll want to read uncompressed WAV files as these are much simpler than any of the alternatives. You can find lots of documentation on the RIFF format that it’s based on and from there you can quickly get some sounds on your bus.
Music
You’ve heard sound great music in other games and you want to have some of that emotional punch in yours. You’ve got two choices; write a sequencer or read compressed audio. I opted for the second using MP3 format. A fun thing to remember about MP3 format is that its creation and decoding is protected by patents and licensing. Luckily windows comes bundled with a licensed decoder. So, once you’ve read up on (and implemented) the acmDriverEnum function and all it’s friends and located some accurate information about the internal format of MP3 files you’ll have some data and a working decoder to feed. If you get everything right you’ll also have a set of classes to turn disk files into music streams. You’ll probably find the whole music handling part of your audio so time consuming and tedious that you’ll neglect things like smooth switching between different music tracks until you realise your game needs it.
3D
You and many other people playing your game may have gone to the effort and expense of purchasing and installing arrangements of speakers to simulate spatially located audio. It’s up to you how far you want to push your support; 2.1, 5.1, 7.1 maybe even 11.1. In any case you’ll need the notion of an audio listener (or potentially many audio listeners if you’re making a split screen game) within your engine. This will need to handle additional things like environmental effects like reverb and filters.
Graphics
Graphics system design is sufficiently large and complex that some people mix and confuse ‘graphics engine’ with ‘game engine’. In this section we’re just considering the creation, manipulation and transference of geometric and textural data from somewhere in main memory (or disk) through the graphics devices onto the screen. That’s already a lot of stuff.
The Device
I’ve opted to use Direct X (something I occasionally regret). Unless you writing a software renderer you’re going to have a lot of pieces of data that you want to manipulate and send to some graphics device. Some of those pieces of data represent results from work on the graphics device while others represent relatively static data you want the device to use.
Start up
Lets start by detecting and selecting the device you’d like to handle all your graphics work. There’s often more than one to choose from and very regularly the least capable one is presented first (looking at you on-board intel cards). This whole job got a lot easier with the more recent versions of Direct X. It was ugly in DirectX 9 and terrible in the versions before then.
Shaders
Next, it’s time to make use of that resource loader you created earlier. If you’re using Visual Studio for your coding then you get the benefit of built in shader compilers so the only thing you need to do is handle the loading of the resultant binaries. Even if you don’t write your pixel and vertex shaders in pairs you’ll probably want to support some sort of pairing to create ‘material’ objects. You could even include details about other aspects of surfaces like the depth read/write modes or culling order. Before long you’ll find you’ve created a workable material format and all the loaders required to use it. You should also spend some time considering how you’ll be passing parameters into your shaders when it comes to actually rendering everything. That includes semi-global parameters like viewport values through per-pass or per-surface parameters all the way down to per-instance parameters like world-matrix.
Pipeline
It’s also very likely that your game is going to make use of some of those fancy modern pixel shader effects (like bloom) or need to be rendered in multiple passes. For that you’re going to need to implement something to keep track of all the shaders and render targets used in multi-pass rendering. You’ll also need to reuse things and make sure everything gets done in the right order. Also, don’t forget to handle the screen resolution changing, lots of those targets will need suddenly to be recreated when your player presses Alt-Enter.
Textures/Images
DirectX actually support loading and converting a huge number of different file graphics file formats but we’ll just consider creating some raw image loading. You’ll be surprised how often you just want some image data but don’t want (or have) a whole graphics device setup just to get it loaded. Luckily a long time ago the TGA format was created. It’s well supported by image software and it’s super easy to read. In it’s 24 or 32 bit uncompressed form (without palette) it’s just a 18 byte header followed by interleaved image data followed by a 16 byte footer; easy.
Vertex Data
All that image data is going to need something to hang onto. You could go for a very flexible vertex system that allows for easy broadening of vertex data as your requirements change but as you’re already spending years making something small scale I suggest that you’ll probably be just find with a few simple vertex declarations for common vertex formats. I use three.
Then there’s the separate (but related) issue of vertex buffer management. As we’re assuming you’re making the same design choices as me, we’ll also assume that you’re creating a game with lots of dynamic geometry made up of lots of sprites. Unless you really love the idea of GPU memory fragmentation bugs I would strongly recommend not creating thousands of tiny vertex buffers. Instead you’ll want to allocate some nice big vertex buffers and handle the progressive filling and reuse of them as you go. Each large vertex buffer should act like a slightly dithered queue with new geometry being added at one end with the other end becoming progressively unused as dynamic sprites end their life. Once a vertex buffer has no more live sprites in it you can simply reset your write position to zero and start again. You’ll have to spend a little bit of time choosing the right balance of size for your big buffers to to avoid excessive wastage and keep a reasonable amount of vertex data ‘live’ at any one time.
Draw Commands
Yet another area of engine design where people continue to write articles and get excited and emotional about exactly the best way of feeding data and commands to the GPU in order to produce the final results on screen. I opted for a simple method that’s reasonably easy to adjust and debug. Each frame a list of jobs is created that represent geometry to be drawn to a specific surface (or surfaces); each containing details of vertex buffer and index buffer sources along with state, texture and shader requirements. These job lists are sorted to retain final results while reducing state changes or expensive resource movements. Then the lists of jobs are sorted by surface dependencies. Finally these lists are iterated through and applied. It works well but there are many better/faster/more-flexible approaches that have different time/bugs/complexity characteristics that you’ll probably be distracted by for a few weeks or months.
UI
You’re going to need some UI and it’s going to be more complicated than you expect so it’s worth considering how you intend to graphically describe, create and manipulate it. Ignoring (at least partially) how you’ll actually interact with it. UI has some interesting rendering characteristics that can complicate your handling of draw commands; it’s usually very draw order dependant and it usually uses lots of different textures. I’m ignoring any desire to have some parts of the UI mask out other parts (scroll windows for instance) because a lot can be achieved without it and if you really feel like you need it then you’ll just have to include render target switching in the generated draw commands. I opted to describe my UI using horizontal and vertical lists of elements that are either of fixed or flexible size arranged in a hierarchy defined in script files. This hierarchy is then traversed and the draw commands that represent it are built and marked such that they are never re-ordered. We’ll consider how UI is interacted with in a later section for now you can concentrate on problems like word wrapping and odd space distribution in a pixel aligned lists of boxes.
Camera
Beyond the actual rendering of your geometry there’s the more subtle aspects of camera motion and placement itself. This includes lots of things from cinema about framing and composition as well as lots of things learnt from games about motion and interaction. The first and best thing you can do here is look at every other game. There are lots of very good (and very bad) examples of game cameras and it’s important to be able to spot what makes one good and another bad. There’s also lots of room for subjectivity here so it’s important to put your camera in front of plenty of different people to see if their response is the same as you expect. Pro-tip; vomiting testers is bad.
Interpolation
At the root of any reasonable camera motion system is some robust interpolation code. That means something beyond move-halfway-towards-the-target maths and into critical damping. Once you’ve got some good interpolation you’ll find yourself using it all over the place. Even outside the camera code.
Framing
The code that handles this needs to balance what looks nice with what feels nice. There’s lots of literature on aesthetically pleasing composition and framing so I’ll just assume you’ve read, digested and implemented all of that. The other thing to handle is the motion of the camera as your character (or whatever) moves around. To start with you should obviously aim to keep your character on screen (almost) all the time. After that you can consider the extra marks stuff like giving a view ahead so the player isn’t so surprised when that pit comes into view or subtly include points of interest in the frame to guide the players eye. If everything goes well your player will never notice all the code you’ve written to move the camera; it’ll just work. This is probably a good reason in itself never to become a game camera programmer.
Shake
This covers anything you want to do to the camera that’s short lived and in response to some in-game event. Everyone loves camera shake, right? Well people love the idea of camera shake but it’s effectiveness is in the details of it’s implementation. If you want to implement good and convincing camera shake you should probably do a little research on the different sources and effects of cameras shaking. Attaching your phone camera to stuff and recording a few videos works quite well. Also, while were talking about shake you should also understand the difference between physical camera shake and cinematic frame shake.
Input
So now you’ve got your engine outputting graphics and sounds you might want to interact with it. You’ve already written the code to gather and transmit the keyboard and mouse events, the next step is to include gamepad input and bring it all together in a single system that can be used by your game. Lets call it the input system.
XInput
The first step is to gather the state of any attached gamepads (I ignored joysticks but you can include those if you like) and create events that can be handled along with those from the keyboard and mouse. As a general rule I try to avoid polling if I can do something using events but when it comes to gamepads at the very basic XInput interface you have to poll through the XInputGetState function. I turn these state polls into events by comparing against the state at the previous poll interval and informing higher functions about the differences. Once you’ve implemented all this you’ll be able to add listener functions to the input system from higher level game objects and have those functions called evenly from any connected device.
Filters
Unless you want multiple systems responding to the same input you’ll have to implement some code to filter input listeners in and out based on all sorts of complex constraints. This is useful all over the place but it’s critical when you start to implement UI interactivity.
UI
Now you’ve got a full compliment of input devices and signalling it’s time to make yourself some interactive UI. This is another of those subtle and complex areas of code design but you’ll find a lot less documentation around the internet about exactly how to handle certain things or respond in certain situations. That doesn’t mean you shouldn’t look for information, just expect it to be harder. I think the main reason for this lack of reading material is that unlike other areas where it’s clear what good results look like, in UI and UX design the better it’s working, the less you notice it. There are a few subtle details of interaction that you can see in windows itself (you probably use them without noticing). For instance button clicks are handled on release to allow you to cancel the click by moving your mouse pointer off the button before letting go. Or, when the mouse buttons are pressed other UI buttons cannot be highlighted (gain focus) even by using keyboard navigation; to avoid confusion. There are lot of them (books full in fact) and if you’re going to implement a good UI system you’ll need to understand them. You’ll also have to integrate all that mouse based UI design with the gamepad and keyboard. Good luck.
Game States
Your game is going to need to move between different macro states. Loading screens, main menu, in game, etc. Writing some code to handle this stuff will make your life easier. I opted for a simple state stack. Moving on.
Internet
Being able to interact with any sort of online service is almost essential in modern game engines. Once you’ve got some basic system to handle basic HTTP/HTTPS requests and the generally asynchronous (occasionally unreliable) nature of network communication you can do all sorts of fun things. Don’t forget that you’re also going to need to learn PHP (or Ruby or what ever other language you fancy) and implement your server side code. This is a great way to learn about how all this works but it’s also a great way of opening yourself up to all sorts of security and privacy issues. In short, use an existing service unless you are an internet security expert.
Scoreboards
These are usually just simple sets of name-number pairs arranged in groups and sorted. super easy to implement and super easy to use.
Analytics
Game developers love analytics but players are often suspicious of it. It allows developers to get clear information about the impact of their design decisions and potentially pick up on (and fix) issues that evaded testing but it’s obviously open to abuse. I suggest that if you do implement analytics you should gather the minimum amount of data needed to answer only the key questions you might have. It’s tempting to generate events and value for every single tiny little player action. This just results in a swamp of data. I opted to use analytics during testing and then scale it right back for release.
Anything Else?
You’re almost there but the things listed above are just a summary of the big stuff. I’ve skipped some of the simple (but often critical) things that glue it all together. Things like Exception handling, crash-dump generation, release versioning or debug logging. you’ll need these ‘glue’ systems all over the place.
The Game
That’s it, you’ve built a primitive engine capable of handling a simple game on a single platform (PC). The work in the previous section probably constitutes a solid few months of programming if you’re able to spend all day doing it, but If you’ve only got a little bit of spare time to work with then you could expect all that to take many times longer; potentially years. At least it’s done now. You can move on to the fun part where you actually create your game.
Obviously every game is different so there’s very little point in me attempting to detail the steps you’ll need to take to create yours. Instead I’ll detail the work that went into making the significant systems within The Moonstone Equation. Some are applicable to other games, like platform handling and save game support, other are very specific to my game. Even though it might not be directly applicable, it might give you some sense of the scale of the workload you’ll have to undertake.
Physics and collision detection
This doesn’t sound like making a game. Surely this is part of the engine? In most cases, yes, but as I’ve opted to make a platform game with tiles arranged in a grid I can avoid creating a generic collision detection and physics system and just handle the simpler case. There’s a lot of very heavy duty maths required to create a full scale physics engine but non-rotating axis aligned boxes is all I needed. Once I’d created some basic box penetration handling, simple gravity and buoyancy I had all I’d ever need. I also excluded traditional friction. Nothing needed vertical friction and things stop moving horizontally on surfaces based on game rules and simple damping. On top of this there are a few extra bits of code to improve the player experience, like changes to how box penetration is handled for the player when they jump upwards near corners.
Platformer Handling
When it comes to understanding how to make decent platform game handling It’s lucky that there’s so much reference material. By that I mean there are lots of great platform games that demonstrate how handling should feel. They also demonstrate that there’s still tons of scope for difference. If you play Super Mario 3 followed by Ray-Man Legends and you’ll see what I mean. Aside from the essential stuff like responsiveness and predictability there there are lots of differences in how they feel. On top of that there’s subtle (almost invisible) things these game do to make the player feel more in control. For instance, Mario allows to to successfully jump upwards from a ledge for a few frames after you might actually have walked off of it and Ray-Man subtly alters jump arks to guide the character between tricky platforms. These give the player a sense that they’re better and more accurate than they actually are and they make the game feel more enjoyable as a whole.
With all this in mind and an understanding of how I wanted The Moonstone Equation to feel when it was played I set about implementing my variation of platform handling.
Sprite based level editor
If I was going to get anything completed I needed some tools that made the process of creating my game world as quick and fluid as possible. For that reason I opted to stick with 16×16 pixel tiles and a level editor built into the game. This is one of the areas where all that UI system wok really payed off. Making and changing editor dialogs is reasonably easy.
Procedural tile selection
To further speed things up I implemented some procedural tile selection for particular sprite sets. Things like selection based on tile neighbour or selection from a random pool.
Decoration
As this isn’t actually the last century there’s nothing to stop me from breaking a few rules for aesthetics when I want to. I’ve opted for an aligned grid of tiles to aid the primary game rules and give a particular look to the but sometimes I might want to place a decorative sprite in a very specific place. It’s all just polygons in the end so I made sure the editor allowed me to choose between grid-snapping and free placement.
Lighting
Ambient occlusion and light flow are not things I think I’ve ever seen in a 2d tile based game. So I implemented it. It allows me to better control the mood of the levels as well as helps differentiate The Moonstone Equation from all the thousands of other games it has to complete with in the wider world. Someday I’ll write a long article about how it’s done. For now I’ll summarise by saying; lots of maths that it took some time to get it looking just right.
Virtual Machine
The Moonstone Equation contains a few dozen computer terminals for the player to interact with. I wanted them to have rich functionality and really feel like computers (rather than the elaborate text boxes and menus used in many other games) so I basically implemented them as virtual computers; between them they’re running over 50 different little programs from games to email and services. To make all this stuff work I approached the problem from two sides with the aim of meeting in the middle.
Machine definition
The first step was to define the virtual computer I wanted to use. I didn’t need it to be an accurate simulation of the inner workings of actual hardware, I needed it to be convincing (and interesting). I started with the terminal display. It’s handled through a pixel shader referencing a 32×32 pixel texture representing screen memory. Each pixel can be one of 256 colours corresponding to one of 256 code symbols arranged in a grid on a larger texture. This basically means that I can use a 1024 byte array as traditional terminal screen memory with byte values representing letters (just like ASCII). It might sound archaic but it’s actually very refreshing to use. Once that was all working I wrote hundreds of simple helper functions to act as the basic terminal tool-set. These included things to write text, scroll the screen contents, read keyboard input, draw lines and so on. As this isn’t a real simulation, these were all just written in C++.
The next step was to define the virtual processor that was going to actually handle the running of my little programs. Again because it’s not a simulation I could cheat, so I opted for a simple stack machine with 36 basic instructions that could operate on a range of types (ints, floats, bools and strings). This instruction set also included the ability to call specifically formatted C++ functions so once I’d created some glue code the virtual processor was interfacing directly with the entire terminal tool-set (along with any other functions I wanted to expose).
Compiler
At this point I switched to the other end of the problem. The tools to allow me to write code for these virtual machines. I wanted to write code using something that was enough like C/C++ to make it easy for me to write. I can write in C++ much faster than I can write pascal because I do it most of every day (I would also mean I could use code colouring in Visual Studio). It also needed to be simple, I didn’t want to extend the virtual machine functionality to accommodate it. At this point there was a bit of a pause while I did lots of reading. I obviously needed to do research for many of the other areas I’ve already covered but I think this is the only place I stopped and read nearly a whole book on the subject. If you ever feel the need to write a compiler I recommend reading Compilers: Principles, Techniques, and Tools, it’s basically got everything you need to get started all in one place, which is nice. In the end I opted to support C style syntax for functions and statements along with simple static allocation of basic variables, no structs and definitely nothing like classes. This was more than enough.
With my language defined I set about creating the abstract syntax tree representation of it followed by converting it all into reverse polish notation (Code parsing is full of some great terminology). I didn’t find a need go beyond this to an intermediate language and I didn’t implement any optimisation steps aside from pre-evaluating constant statements. I’d opted to implement a stack machine (on purpose) So the result was something I could simply traverse to generate the instructions required by the virtual machine. The machine instructions could just be saved as a block of unsigned integers in a binary file.
The result
Doing all this extra work for the in game terminals meant I could implement a much richer experience for the player. Instead of simple boxes of text I’ve created full games to play, email systems, research terminals, and machine monitors that contain relevant and dynamic content. With a bit more work I went on to allow the virtual characters in the world to make use of all the terminals as a connected research network; sending messages to each other (and the player), logging in (and out) daily to create reports or write journals and just generally adding another layer of believably to the world that I couldn’t have done without it.
Persistent real-time
It’s been done in a few games but I don’t know of any puzzle-platformers containing a world that persists and changes while you’re away (a little bit like Animal Crossing).
It would be trivial to just write some code that read the clock and made decisions based on it but (obviously) I wanted something a little more interesting. The first step was to make sure that in the places within The Moonstone Equation world where you could see the sky it (and the lighting) reflected the time of day. I already had a dynamic lighting system so this was the simpler part. The next step was to make sure that the characters and many of the objects in the world developed over real-time as well as in response to the actions of the player. This itself can be broken down into two parts. First there’s the simple but laborious task of making sure that the characters take account of any passed time when they say things to the player. So if someone hasn’t seen you for weeks they might comment on it, especially if they expected to see you sooner; or if they always see you early in the morning they might ask about you sleeping habits. This was basically achieved by a combination of heuristics and branching conversation trees that can persist or change over time; this also produces an absolutely huge number of possible ways to experience the story elements of the game, which is nice. The Second and more complicated part was making the world feel like it’s a little more alive. Characters are supposed to be doing research, sending emails, writing reports and notes, resting, going to sleep and so on. Things like the position of commonly used objects like books, mugs, small machines, boxes and so on should change over time. You’ll even see some plants grow if you play for long enough. For the characters this was achieved using a simple AI simulation that gives everyone a rough schedule and simple goals. This simulation is updated for the passage of real-time and when you return after being away it just catches up when the game loads (which is why it needs to be very simple). The objects in the world that change do so in only a few ways. Plants and such progressively develop from an initial form to a final (usually more interesting) state. Renewables like the contents of the vending machines cycle through a pattern with occasionally interesting ‘special’ states. Variable objects like the books or boxes on shelves are subject to semi-random changes based on how often they’re used. Finally there are some special objects develop in specific ways depending on things like plot and player action.
When it’s all considered together, the areas of the game world where people live and work hopefully feel like that’s happening all the time. It also allows for some contrast against the areas of the game where it (intentionally) looks a little bit more like time stands still.
Other Bits
Just like in the engine there are lots of little parts of the game that are essential glue between the larger parts that are not large enough for a whole section of their own. Here are a few.
Controller Switching
Many less people than you might expect actually own a gamepad for their PC (about 12%-16%) so making sure your game is fun and playable on keyboard is critical stuff. The Moonstone Equation feels as good on game-pad as it does on keyboard. It’s also important to allow your players to switch from one to the other as they choose while the game is running, including unplugging or plugging in a controller. This should all happen seamlessly and without fuss. I’ve seen plenty of (sometimes big) games fail at this. Oh, and don’t forget that all those on screen controller hints will need to match the input device.
I opted to make The Moonstone Equation so it can be played to completion using either game-pad or a keyboard but there are some occasions where you have to type on the keyboard if you want to experience everything; I think the computer terminals feel more authentic if you get to type simple commands into them. I hope that very few people find this design choice odd or inconvenient. I expect that everyone has a keyboard within easy reach and I have a very very low expectation of ever porting my game to a games console.
Game Saves
The earlier work on the engine to support structured files and versioning made most of the task of supporting game saves file a lot simpler. The hardest part was making sure that I could reliably iterate through all the world data and apply the changes the player had made since they started playing without missing anything new I’d added to the game since the save file was created.
Effects
There’s lots of different types of dynamic moments within the game world that call for some sort of audio-visual response. I didn’t write a particle system within the main engine because I didn’t want to spend the time writing the tools that would be required to turn a general purpose system into specific effects for specific occasions. Instead I opted to implement a simple code based system that make use of the dynamic vertex/index buffer handling to continuously generate particles when needed. It’s not the most optimal approach but I only need a few thousand particles at a time; PC hardware has been capable of those sorts of numbers for a long time. Along with the general particle stuff there’s also a few parts of the game that use pixel shader effects. These are always much easier to write directly as needed; especially when you’ve already written the rest of the engine.
Conclusion
Congratulations, You’ve battled through ten thousand words of my rambling warning about the risks and effort involved in making everything yourself (or you’ve just scrolled to the bottom to read this bit). If you’re still considering starting from some very basic point and writing an engine as well as a game then great, you’ve obviously got some serious determination.
However, there are a few significant things I’ve left out (along with many small things).
For a start I’ve only focused on the technical aspects of game creation, There’s a lot more to game production than just that. I’ve not mentioned the thousands of (occasionally animated) sprites that went into the Moonstone Equation, or all the other artwork and UI. On top of that there’s the creation and balancing of hundreds of sound effects and the composition and production of a dozen or so musical scores. If you opt for branching narrative and puzzles as I have you can expect that to take up a significant amount of time too.
Once all that’s done (actually before all that’s done) you’ve also got to consider how you’re going to tell everyone/anyone about your amazing creation; that means marketing. So now you’ll also need to create screenshots, videos, trailers and other promotional materials along with a website, social media accounts and forums to put it all on to allow you to communicate with the fans and supporters you hope to find.
All this effort might also lead you to write a long rambling article warning others about the perils (and potential folly) of your protracted development methods.
Good Luck.