Depth Ordering in GameMaker 2

At some point during your usage of GameMaker 2 you will very likely encounter a very classical issue in game development, how do I ensure my sprites are drawn in the right order? Anything 2D with movement will still have a sense of hierarchy, some things are in front of others, but how do you do that with GameMaker?

When you will look into your trusty search engine, you will most likely find either this blog about using a technique called z-tilting, and if the following doesn't satisfy you, definitely check it out, or you may find the YouTube channel FriendlyCosmonaut with this tutorial video, which is a bit older but still teaches some valid techniques. As someone with more background in different engines and other areas of software techniques, I want to elaborate on some of my strategies to solve this issue, including the one I used in my student project game fairy-strategy, a turn based 2D strategy game.

The classic depth = -y

The oldest and easiest way to get this working. Things lower on the screen will be before things higher on the screen, which covers like 99.9% of all use-cases. This has been used, as far as I heard, since the earliest days of GameMaker, and you will find many texts about it, especially related to GameMaker 1.

However, this bears some terrible secrets I fear. Reading the documentation on the depth property you will find this line, which concerns me:

but it may be that you want to change the depth of an instance using this value, in which case a “temporary layer” will be created specifically for the instance at the given depth

This sounds like a very heavy operation to me, including the fact that this one instance is now in it's own render-batch, which will significantly increase overhead with every instance we do this with.
My takeaway from this is, don't do this when you have a good amount of instances, especially when targeting weaker hardware.

Fine, I'll do it myself

Now, if putting instances on dedicated layers isn't cutting it, the next step of course is sorting instances yourself. Just like depending on which rectangle you draw first in the same area in paint, instances that are drawn first stay behind instances drawn later, by virtue of the new pixels replacing the old. GameMaker draws layers depending on their depth to leverage this effect, but if there is more than one instance on a layer, they will just be drawn in the order they are created, which may be the creation order, but it's not that reliable actually, and it's also static, so anything that moves will not get updated in it's order.

So let's do it ourselves? Make an object, a depth sorter, find all the instances in need of ordered drawing, we can for example give them all the same parent object, and then sort them based on their x and y. Next we overwrite their draw event so they stop drawing themselves and instead let our depth sorter draw them using with(instance) { draw_self(); }. Straight forward, effective, no extra layers, one draw batch. However, we got a couple issues here, some we can solve, some we cannot.

1. Drawing anything more complex than a sprite
   The Issue with draw_self() is, that it only instructs the instance to draw it's sprite. If you want to do anything more fancy with it, you are out of luck. However, there is a solution.
   Instead of killing the draw event, we set the instance to be visible: false in the before_draw event, and set it back to it's initial value afterwards, so the step events can still rely on it. Then, inside our depth sorter, we call event_perform(ev_draw, ev_draw_normal); on that instance, if the visible value we remembered before is actually true. I actually used this in fairy-strategy, but more on that later.

2. That's a lot of CPU load
   Well, if you sort all your instances every step, you are gonna hit your CPU hard, especially hard with more instances. Step runs on your central processing unit and it may be very weak, especially on older hardware, which might cost you your 60 ticks even. First, one should think really hard if you want to sort your instances every step, sorting it every couple steps should already help a lot and not be too much of a hassle. But maybe you can only sort on specific occasions, like on a turn based game, you only need to consider a new order when something with a turn starts moving, right? And on that note, be careful with actual sorting. Depending on how you do it, that's gonna be a lot of loops. If you can evaluate just the instances that might have changed position, you want to instead look at their neighbors in your collection, and swap them around in place if possible. This last step, while not very trivial, will give you some high gains again.

Fancy Shaders

First some setup.

We need to first get all the instances we need to render “sorted”:

// Somewhere sensible, like step_beginn:

self._instances = instance_find(obj_par_depthsorted, all);

This codes' location depends on your use case, but step_beginn would be the most reliable and expensive.

Then, when we start rendering, we want these instances to not render themselves, so we need to set visible on each to false, but still remember what it was set to before, so we can reverse that later:

// draw_begin

for (var _i = 0; _i < array_length(self._instances); _i++) {
    var _instance = self._instances[_i];
    _instance._shadow_visible = _instance.visible;
    _instance.visible = false;
}

Keep in mind that for this to work, the object doing this needs to be as early as possible in the creation order.

On the end of the draw, we reverse our previous visibility changes:

// draw_end

for (var _i = 0; _i < array_length(self._instances); _i++) {
    var _instance = self._instances[_i];
    _instance.visible = _instance._shadow_visible;
}

Now, since some version of GameMaker 2 we got access to shaders. It's just fragment and geometry, but it will do for the following, which is what I did in fairy-strategy! (sort off)
As someone with some experience in other engines, my first instinct was “how can I use the depth buffer”, which should be what's used by layers, maybe. Or maybe they really do just draw them in order. In any way, GameMaker 2 allows you to activate z-testing, so you want to call this in your depth sorters draw event:

gpu_set_zwriteenable(true);
gpu_set_ztestenable(true);
shader_set(sh_zsort);

The last line is setting our shader to handle future draw calls. This shader is leveraging our new depth testing. Why? Because we need to write the z-value somehow, as GameMaker doesn't use depth for that but instead does it's layer shenanigans. But let's have a look at our shader:

attribute vec3 in_Position;                  // (x,y,z)
//attribute vec3 in_Normal;                  // (x,y,z)     unused in this shader.
attribute vec4 in_Colour;                    // (r,g,b,a)
attribute vec2 in_TextureCoord;              // (u,v)

varying vec2 v_vTexcoord;
varying vec4 v_vColour;

void main() {
    vec4 object_space_pos = vec4(in_Position.xy, -in_Position.y, 1.0); // Setting the Z
    gl_Position = gm_Matrices[MATRIX_WORLD_VIEW_PROJECTION] * object_space_pos;
    
    v_vColour = in_Colour;
    v_vTexcoord = in_TextureCoord;
}

most of this is what gets auto-generated when you just add a new shader object. The important part is the line where we calculate our position in object space, we first feed in in_Position.xy as we do, and then for z, we give it our -y. We basically cycled back to the old way, just this time, it's the GPU doing it. The last parameter needs to be 1.0, because math (something something normalized matrices something). And now in our draw event, as we have enabled our shader, we just dispatch each objects draw event:

for (var _i = 0; _i < array_length(self._instances); _i++) {
    var _instance = self._instances[_i];
    if (!_instance._shadow_visible) continue;
    
    with (_instance) {
        event_perform(ev_draw, ev_draw_normal);
    }
}

We can obtained these instances using a common parent object in our create event (but if you can create more instances at runtime, you need to keep this updated). Also keep in mind you want to remove dead instances, since calling anything on them will crash your game.
In the end, we clean up:

shader_reset();
gpu_set_ztestenable(false);

And now, we are drawing all of the instances every draw event, but without having to sort them CPU-side, as the z-value will allow the GPU to check if a pixel should truly be drawn, or if it's in the background and thus hidden.

In my opinion, this will be the fastest way to do this, since GPUs are good at this sorta thing and this should also not increase the number of render batches, unlike previous solutions. If you no also find a way to update the collection of instances holding everything you need to draw this way, this really should be the fastest solution (for platforms with good shader support).

Some last words or something

One would wish, that the engine could use the z value more directly itself, as even a 2D game usually has a sense of depth, but it hasn't been long since shaders released, so who knows when they might realize that creating temporary layers depending on depth is a really shady hack /personalOpinion. Of course, using z sorting could be a bad idea in some scenarios, I do not know how well it works in a browser, or on a phone. But for desktop it should really be the goto if you have more than, say, 30 instances in need of sorted rendering. Also, there are many people better at shaders than me, that can do more creative stuff, like the blog I mentioned in the intro that does z-tilting, in which you can have sprites partially hide other sprites based on their position, something my implementation does not account for, so really have a look at that and FriendlyCosmonaut too if you are not sure if this is the best way for you and your game.