Langur Monkey - Toni Sagristà Sellés

A maximally tiny language model

Toni Sagrista Selles — Sat, 07 Nov 2026 00:00:00 +0000

In this post I document a little experiment I have been playing around with for a while that explores an absolutely minimal language model. I’m specifically interested in how small a language model can be made while still doing something. I got the idea while I was reading a post about GPT-2. A pre-COVID model, GPT-2 now seems minuscule in comparison to current frontier models. Its largest incarnation was trained on about 40 GB of text, had 1.5B parameters and used an embedding dimension of 1600. Nowadays, a GPT-2-like model can easily be trained in a consumer GPU with ~16-24 GB of VRAM.

If the GPT-2 family of models pales in comparison to current model dimensions, the model I present here today pales in comparison to GPT-2. Since it is extremely minimal, it can be trained on a single CPU core in a few seconds and needs very little RAM. It can have single-digit embedding dimensions, does not follow the transformer architecture, has no attention mechanism, and may not even use any hidden layers! It is a purely academic model, in the sense that it is absolutely simple but does nothing useful.

I’ll call the language model in this post TinyLM. Below is a summary of some of the features and concepts that this post touches on:

A closed world. 92 words, a few hard rules, 2560 sentences. That is all the language there is.
Learned embeddings. Nobody tells the model that “wine” and “beer” belong together. It works it out from the sentences alone.
A bag of words. Words go in unordered, so “the chef met the clerk” and “the clerk met the chef” look exactly the same to the model.
A ceiling of about 48.4%. Throwing the order away caps accuracy, no matter how big the model gets.
A few hundred to a few thousand parameters, with the embedding size and the hidden layer size as free knobs.
A sweep over those two knobs, from the baseline all the way up to the ceiling.

Since it is so easy to train and test, the model is fluid. It can have between a few hundred and a few thousand parameters, depending on the dimensionality of the embeddings and the size of the hidden layer. But let’s start at the beginning.

The world

Such a small model wouldn’t fare very well trying to reproduce human language, so I reduced it to a set of predefined sentences. I call this set our closed world. This world is composed of only 92 words: 8 proper names, 12 actors, 6 prepositions, 28 places, one determiner, 24 objects, and 13 verbs. Everything that can be expressed is done through those 92 words. I combined them using some rules and got the 2560 possible sentences that constitute our world. This is the whole world. This is the training dataset. This is everything TinyLM will ever be able to learn and recognize.

Every sentence in the world has the same shape: a subject, a verb, an object, and sometimes a place. Subjects are always people. There are 20 of them, either a bare name (“john”, “mary”, “lucia”) or “the” plus an occupation (“the chef”, “the pilot”, “the farmer”). The verbs come in two families. Four of them (“helped”, “met”, “thanked”, “invited”) are things you do to a person, so the object is another person and never the subject itself: “lucia invited mary” is in the world, “lucia invited lucia” is not. The other nine are things you do to stuff, and each one only takes objects of the right kind. You drink drinks, you eat food, you repair tools and homeware, you pack clothes and drinks. So “the chef ate soup” exists, “the farmer repaired the soup” does not.

Finally, about half of the sentences end with a place: “near the bridge”, “inside the cellar”, “under the tower”. That’s the whole grammar. No questions, no negation, no conjunctions, no subordinate clauses, no adjectives. Just subject, verb, object, and sometimes where.

A few real sentences out of the 2560, so you can see the shape:

the pilot met john near the cellar – person verb, subject and object both people, place at the end.
lucia invited mary – the same, without the place.
john drank wine behind the hut – thing verb, and the object is a drink because “drank” only takes drinks.
the clerk organized books under the store – thing verb, object from the paper class.
And two that are not in the world: lucia invited lucia, because you cannot invite yourself, and the farmer repaired the soup, because soup is not a tool.

TinlyLM world description


➜ uv run python -m tinylm.world
2560 sentences, 92 distinct words
 8 names, 12 occupations, 9 thing verbs, 4 person verbs, 24 things in 6 classes, 6 preps, 28 places

verb -> object classes:
 drank drinks ['beer', 'soda', 'water', 'wine']
 sipped drinks ['beer', 'soda', 'water', 'wine']
 ate food ['apples', 'bread', 'soup', 'sweets']
 devoured food ['apples', 'bread', 'soup', 'sweets']
 repaired tools + homeware ['brushes', 'cups', 'jars', 'lamps', 'nails', 'plates', 'ropes', 'tools']
 cleaned clothes + homeware ['boots', 'coats', 'cups', 'gloves', 'hats', 'jars', 'lamps', 'plates']
 packed clothes + drinks ['beer', 'boots', 'coats', 'gloves', 'hats', 'soda', 'water', 'wine']
 bought food + paper ['apples', 'books', 'bread', 'letters', 'maps', 'soup', 'stamps', 'sweets']
 organized paper ['books', 'letters', 'maps', 'stamps']
 helped, met, thanked, invited -> any of the 20 people except the subject

things per class:
 drinks ['wine', 'water', 'soda', 'beer']
 food ['bread', 'soup', 'apples', 'sweets']
 tools ['tools', 'brushes', 'nails', 'ropes']
 clothes ['boots', 'gloves', 'hats', 'coats']
 paper ['books', 'letters', 'maps', 'stamps']
 homeware ['cups', 'plates', 'jars', 'lamps']

samples:
 john helped mary in the store
 john helped peter
 john helped lucia inside the factory
 john drank wine behind the hut
 john drank water
 john drank soda behind the bridge
 the singer organized letters in the harbour
 the singer organized maps
 the singer organized stamps beside the train

Model structure

The whole model is two small tables of numbers. The first one gives every word a handful of coordinates: three numbers per word in the default model, so $92 \times 3 = 276$. The second one, the readout, says how much each word likes each of those three directions, another 276. That’s it, 552 numbers in total. There is no lookup table of sentences and no rules inside, just those numbers. All of them are learned from the world above.

The pipeline for guessing a word is almost embarrassingly simple:

The architecture

92 counters go in, the $92 \times 3$ coordinate table turns them into 3 numbers, the readout scores all 92 words, and softmax turns the scores into percentages. That is the whole default model: 552 numbers and no hidden layer.

The same three steps, written as matrix operations. Count the words of the sentence into a vector $\mathbf{c}$ with one entry per word in the world, and let $\mathbf{E}$ be the table of coordinates, 92 rows of 3 numbers. The sentence collapses into a single point by adding the rows of the words that appear, the readout $\mathbf{W}$ scores that point against every word, and a softmax turns the scores into percentages:

$$ \mathbf{h} = \mathbf{c}^{\top} \mathbf{E} \qquad \mathbf{z} = \mathbf{W} \mathbf{h} \qquad \mathbf{p} = \operatorname{softmax}(\mathbf{z}) $$

with $\mathbf{E} \in \mathbb{R}^{92 \times 3}$ and $\mathbf{W} \in \mathbb{R}^{3 \times 92}$, so $\mathbf{h}$ is a column of three numbers, $\mathbf{z}$ (logits) is a column of 92 scores and $\mathbf{p}$ is the column of 92 probabilities that this post keeps calling percentages. Each row of $\mathbf{W}$ belongs to one word and says how much that word likes each of the three directions, which is the readout table above read along the other axis. The code keeps it transposed, as a $3 \times 92$, which changes nothing except the order of the indices.

The model in one picture

The whole thing with real numbers for “john drank”. A row of 92 counters, the (92 times 3) table of coordinates, the two rows that matter added into one point, the readout scoring all 92 words, and the softmax at the end. The bottom half adds up the coordinates of “john” and “drank” by hand and shows why the four drinks end up within a tenth of a point of each other.

The middle step is worth a second look. The sentence is turned into a row of 92 counters, one per word, saying how many times each appears. Two words show up, so two counters are 1 and the other ninety are 0. Looking up the coordinates of each present word and adding them gives a single point, and that point is the model’s entire idea of what the sentence is about so far. Then each of the 92 words gets a score for how well it lines up with that point, and the highest score wins. Three multiplications and two additions per candidate word, and that is the whole network.

This is where the “bag of words” in the summary comes from. The input is a bag of counters, and counting does not care about order. Adding numbers does not care either: 3 + 5 is the same as 5 + 3. So “the teacher drank” and “the drank teacher” land on exactly the same point, get exactly the same scores, and the model happily finishes the nonsense one.

The optional hidden layer is the one part that is not just adding. It sits between the point and the scores and turns the single “how well does this word line up” question into a set of small yes/no questions about the context, something like “was john seen?” and “was cleaned seen?”. Because each of those answers can be silent when it does not apply, the model can score a word high only when both conditions hold, which plain adding cannot express. In symbols, the readout becomes

$$ \mathbf{z} = \mathbf{W}_2 \operatorname{ReLU}(\mathbf{W}_1 \mathbf{h} + \mathbf{b}_1) + \mathbf{b}_2 $$

The same picture with one ReLU layer

The same model with 16 numbers per word and a hidden layer of 128 units: the point becomes 16 numbers, W1 turns it into 128, the ReLU keeps the positive ones, W2 scores the 92 words, and softmax does the rest. 15516 numbers in total, and this is the version that reaches the ceiling.

with $\mathbf{W}_1$ of size $3 \times k$ and $\mathbf{W}_2$ of size $k \times 92$, plus one bias per hidden unit and one per word. A narrower point also limits how much the hidden layer can help, which is why the interesting knob turns out to be the number of coordinates per word rather than the size of the hidden layer. As we’ll see shortly, at 16 coordinates per word and 128 hidden units the model has 15516 parameters and reaches the ceiling. The default is still the 552-number version (because the point of the experiment is the smallest thing that works!).

Accuracy ceiling

Nobody can score 100% here, and the reason is that the world is non-deterministic by design. After “john drank” any of the four drinks fits, and nothing in the sentence says which one you meant. On average a bag of words has 2.72 reasonable continuations, so a perfect player that always answers with the most likely one gets 48.4%. That is the ceiling.

Our 552-number model gets 29.2%, roughly three fifths of what is knowable. The same ceiling applies whether or not you can see word order, because no two sentences in the world share the same set of words, so order carries no extra information here.

Embeddings

The 92 rows of the coordinate table start out as plain random numbers, drawn from a bell curve. There is no meaning in them, and nothing in the world tells the model what any word is. The only feedback training ever gives it is “given the words so far, the next word was this one”. That is the entire supervision signal. The model is never informed that “wine” and “beer” are both drinks, or that “the chef” and “the pilot” are both people. Those facts are not in the input, they are in the structure of which sentences exist and which do not, and the model kind of picks that up indirectly.

After 3000 small steps of nudging those numbers a little to the left or to the right to make the right next word a bit more likely, here is what the table looks like. All 92 words, three numbers each, plotted as points in space:

The learned embeddings

The three coordinates of all 92 words after training, one colour per kind. Labels are only drawn on things and verbs, for legibility. Nothing told the model which words belong together.

The words sort themselves out by the job they do in a sentence. The nine thing-verbs (red) fly off to the top, far from everything and from each other, because a verb only ever follows a person phrase and never follows an object. The four person-verbs (orange) stay near the middle, next to the names and occupations (blues), which crowd around the centre since those show up in every slot. The 24 things flatten into a disc at the bottom, and inside that disc the drinks sit with the drinks, the tools with the tools, the paper with the paper. Even the article “the” ends up alone, because it is unlike any other thing in our world.

This clustering is not just something the eye wants to see. Running k-means with six clusters over the 24 thing embeddings recovers the true classes 70.8% of the time, against 16.7% for chance. The drinks come back perfect, with “wine”, “water”, “soda” and “beer” in one cluster. Tools and clothes merge into one, which is fair enough, since both get cleaned and packed by the same verbs. The 3-dimensionality of the embeddings just happens to be small enough so that we can draw them in a simple 3D scatterplot. Moving to higher dimensions to improve accuracy destroys this nice property.

Training

But how do we fill in $\mathbf{W}$ and $\mathbf{E}$? The model is initially populated with random values. Training is the part that fills these, and it uses the whole world: all 2560 sentences, every prefix of each one, 11408 next-word questions of the form “john drank” -> water. There is no split and nothing held back. The question here is how much of the world we can fit into those 552 numbers. The model will never generalize to unseen text, as there is no unseen text at all! In a sense, the model is a compressor for our world.

The loop does one thing at a time. Show the model a real prefix and the word that actually came next, compare its 92 percentages with the truth using cross-entropy, ask backpropagation which way to nudge each of the 552 numbers to make the loss smaller, nudge them a little with Adam, repeat. In code that is four lines, because the whole model is one multiplication, another multiplication and a softmax, so the effect of every number on the loss can be worked out exactly and all at once. Nothing is ever written down as a rule. “Drinks follow drank” is not in the code, it is only in the numbers the training ends up choosing.

3000 turns of that take 18 seconds on one CPU core. The first hundred carry almost everything: the loss falls from 6.56, which is guessing, to 2.46 and 26% accuracy. Step 300 has it at 2.04 and 28.6%, and the remaining 2700 steps buy six tenths of a point, ending at 1.99 and 29.2%. From there on the model has taken what a bag of words has to give, and the nudges only reshuffle answers that were already near-tied.

Where the accuracy sits is worth a look, because it points straight at the difficulty. The word “the” is predicted 100% of the time, and rightly so: it is the only word that ever follows a preposition, so when a place phrase starts there is nothing to get wrong. Verbs come out at 14.8% and the things at 9.1%, and those are exactly the slots where several words fit equally well. That is the ceiling again, seen from the other side: the model is not failing at those, the world is ambiguous there. Make it wider, give it 16 numbers per word and a hidden layer, and it climbs from these 29.2% to the ceiling itself, which is what the next section is about.

Results

TinyLM is implemented in PyTorch. The whole experiment, logs and all, is at https://codeberg.org/langurmonkey/tinylm. First, we show how the model completes sentences. All these results are captured with the base 552-parameter model (smallest). First, we ask it to complete “john drank”, with a temperature of 1. We ask it to produce 4 samples.

Result: "john drank"


➜ uv run python -m tinylm.predict "john drank" --topk 6 --sample 4 --temperature 1.0 --length 4
checkpoint: /home/tsagrista/Projects/tinylm/model/tinygpt.pt 552 params 3 numbers/word train acc 29.2% device=cpu

context: john drank T=1 entropy 2.04 bits
 water 25.0%
 beer 25.0%
 wine 25.0%
 soda 24.6%
 hats 0.1%
 coats 0.1%

sampling 4 continuations of 4 words at T=1:
 1. john drank beer inside the garden
 2. john drank soda inside the park
 3. john drank water inside the garage
 4. john drank water under the station

Next, we ask it to complete “the conductor”, and we increase the temperature to 4.

Result: "the conductor"


➜ uv run python -m tinylm.predict "the conductor" --temperature 2 --sample 5 --length 6 --topk 8
checkpoint: /home/tsagrista/Projects/tinylm/model/tinygpt.pt 552 params 3 numbers/word train acc 29.2% device=cpu

context: the conductor T=2 entropy 3.69 bits
 invited 11.2%
 helped 11.2%
 met 11.2%
 thanked 11.2%
 repaired 7.2%
 packed 7.2%
 bought 7.2%
 cleaned 7.2%

sampling 5 continuations of 6 words at T=2:
 1. the conductor packed soda near the tower conductor
 2. the conductor sipped soda beside the school garage
 3. the conductor helped sara under the factory factory
 4. the conductor bought books beside the hut office
 5. the conductor packed hats mary under the hut

An interesting observation is that the model does not know when to stop (there are no stopping tokens in our world!), so it keeps producing tokens until it reaches the target --length, even when the sentence is complete. That’s why we get things like “the conductor helped sara under the factory factory”. This is fine, it is by design, a trade-off we have to live with.

Next, we look at how the model performs when we grow the number of hidden units and the embedding dimensions. Every number in this section comes from a run in logs/, all with the same learning rate (0.03), the same 3000 steps and seed 0.

The baseline is the 552-number model from above: 29.2% next-word accuracy, against a ceiling of 48.4%. There are two knobs to turn, so let me turn them one at a time.

More hidden units

First, fix the embeddings at 3 numbers per word and grow the hidden layer from nothing to 128:

Hidden units	Parameters	Loss	Accuracy
0	552	1.991	29.2%
2	560	1.993	29.1%
4	752	1.979	29.4%
8	1136	1.968	29.8%
16	1904	1.948	30.7%
32	3440	1.928	31.3%
64	6512	1.878	32.9%
128	12656	1.826	34.9%

The first thing to notice is how flat the left half is. Going from 0 to 2 hidden units does nothing at all, in fact it loses a tenth of a point, and 4 and 8 buy so little that they are inside the noise. The hidden layer only starts to pay somewhere around 16, and even then the whole sweep, a model 23 times bigger, gets from 29.2% to 34.9%.

That is not the hidden layer’s fault but what (little) it has to work with. There is only a point of three numbers going in, so the yes/no questions the hidden units can ask are questions about three numbers. Everything the model knows about a word is already squeezed into those three numbers before the layer gets a chance to help.

Higher embedding dimensionality

Now fix the hidden layer at 128 and grow the embeddings instead:

Numbers per word	Parameters	Loss	Accuracy
3	12656	1.826	34.9%
4	12876	1.711	38.8%
8	13756	1.476	47.4%
16	15516	1.425	48.4%
32	19036	1.424	48.4%
64	26076	1.424	48.4%
92	32236	1.533	44.5%

This is a different picture. Four extra coordinates, 220 more numbers, are worth four points, and at 8 numbers per word the model is at 47.4%, basically at the ceiling. At 16 it sits exactly on it, 48.4%, and then nothing happens: 32 and 64 coordinates give the identical number, with the same loss to four decimals. All the extra numbers do is cost time. The model has run out of things to learn, because 48.4% is a property of the world.

Two rows caught my eye: embedding dimensions 16 and 92. At 16 coordinates the positions go to 100%: every preposition that starts a place phrase is now predicted perfectly, whereas at 3 coordinates it was 42.3%. And the last row, 92 coordinates, which is one number per word with no sharing between words, falls back to 44.5%. With a coordinate per word each word is just itself and nothing has to be shared, so the model stops learning that “wine” and “beer” behave alike and learns 92 independent words instead. Fewer is genuinely better here, up to a point.

So the answer to the opening question is 15516 numbers, of which 552 were enough for a quarter of the world and none of them buy anything past the entry at 16. The smallest thing that works is still the 552-number model. The smallest thing that hits the ceiling is the one with 16 coordinates per word.

How well does it compress the world?

Accuracy tells you how often the model is right, but the model is a description of the world, so we can ask the compressor’s question: how many bits does it take to send the whole world using this model? That is the model itself plus the residual, the part the model cannot predict, which the loss measures directly:

Model cost: parameters $\times$ 32 bits (unquantized float32, an upper bound).
Residual: $N \times loss / ln(2)$ bits, with N = 11408 next-word questions.
Raw world: $N \times log_{2}(92)$ ≈ 74421 bits (9.1 KiB), i.e. naming one of 92 words per question with no model at all.

Model	Params	Bits	Residual	Total	× raw
3 coords, no hidden	552	17664	32762	50426	0.68
3 coords, hidden 128	12656	404992	30051	435043	5.85
8 coords, hidden 128	13756	440192	24292	464484	6.24
16 coords, hidden 128	15516	496512	23451	519963	6.99
92 coords, hidden 128	32236	1031552	25222	1056774	14.20

Only the first row compresses: 0.68× raw. Every row past it spends far more bits on the model than it ever saves on the residual, which bottoms out around 23-24 kbit once the ceiling is reached. I kind of expected this: the smallest thing that compresses is the 552-number baseline, while the smallest thing that predicts at the ceiling is the 15516-number model. Past 16 coordinates even the residual grows again, so the 92-coordinate case is totally hopeless.

Conclusions

TinyLM works because the world is tiny and closed: 92 words, one sentence shape, 11408 questions with nothing held back. To get from here to something GPT-2-like, which predicts arbitrary text, roughly four things have to change, all of them about giving up that closedness.

The first is obvious: order. The bag of counters has to go, replaced by a sequence with positions and a mechanism that looks at who came before whom: causal self-attention, stacked into a transformer. That alone removes the ceiling.
The second is also rather self-evident: vocabulary. 92 whole words become ~50k subword pieces (BPE), so the model can spell anything, at the cost of a readout 500 times wider.
The third is data. Training on everything there is has to become training on gigabytes with text held back, so the score measures generalization instead of memorization.
Finally, we have scale. One hidden layer becomes 12 layers of 768 numbers (GPT-2 small, 124M parameters), and seconds on a CPU core become hours on a GPU.

Nothing else changes. The objective is still "guess the next word", the loss still means nats, word embeddings are still there, and the compressor’s accounting from the previous section still applies. What changes is only what the model is asked to compress.

Volumetric galaxy rendering

Toni Sagrista Selles — Mon, 26 Oct 2026 00:00:00 +0000

The Milky Way is an essential component in any 3D Universe software. Rendering a realistic (i.e. science-based) and visually appealing galaxy is paramount to the user experience, and to the cohesion of the simulated universe as a whole. Galaxies are made of gas and dust clouds, so they are volumetric in nature. In Gaia Sky we have been using particle-based representations for a long time, even though we have had the volumetric rendering infrastructure in place also for a while. This post describes the rendering technique we use in Gaia Sky 3.9.0 to render the volumetric Milky Way, and how we have constructed the dataset that enables it. Interested? Keep reading.

The Milky Way galaxy in Gaia Sky

A little bit of history

The rendering of our and other galaxies has always been a topic I dreaded. I never felt that the solutions we found were good enough. I always knew that the right way to do it was with volume rendering, but the main issue was that we did not have the volumetric data.

So, for a long time, we have used approximations to render the Milky Way, some more gracious than others.

Enhanced flat image — Initially, we used a single two-faced billboard with the artist’s impression by Nick Risinger, coupled with a generated star field. This was featured in the September 26, 2016 Astronomy Picture of the Day, titled “Here comes the Sun”¹. This is a very primitive galaxy, but it looks awesome from above and in still pictures due to Risinger’s image.
Textured billboard approach — After Gaia DR3 landed in 2022, the catalog data were used by Stefan Payne-Wardenaar² to produce a high-quality representation of the whole Galaxy³ in Blender. At that time, I worked with Stefan to produce a particle-based version that would fit in the existing billboard systems in Gaia Sky. The first version was based on textured additive billboards and dithered depth buffer occlusion. The depth was correct, but the look of the galaxy was rather cartoonish and unrealistic. Additive blending was used for all channels except the dust, which was purely occlusive in a binary sense (no sorting, so no transparency), making use of dithering to simulate translucency. This version is the one that looks the worst of all. Using textured billboards is a bad move to simulate a volume of gas and dust. Also, the brightness blows out when seen edge-on.
Billboards with additive and subtractive blending — Later on, inspired by this article, the textured billboards were updated to rely on blobs and additive/subtractive blending. The main issue was that the emissive parts blended additively in a very unrealistic way, and the occlusion did not respect the depth order in the scene. Several attempts were made afterwards to implement Weighted Blended Order-Independent Transparency (WBOIT)⁴, as suggested by the article, but this was very difficult to tune and produced underwhelming results. Looks better than the previous textured-billboards version, but the behavior of light is unrealistic. The brightness is mostly gone when seen from the top, but blows out edge-on due to the additive blending.

All these issues finally tipped me into seriously consider approaching the project of converting the particle data to a volume and implementing a proper volume renderer with emission and occlusion.

Particles to volume

The first step is to convert the particle data to a volume in the form of a voxel grid. This step is crucial, as it can make or break the whole project. Having a good dataset is the difference between a bland-looking visual and an awesome one.

The source data

The source is the same particle model we were already rendering. It is split into channels, each a gzipped text file with one particle per line:

x y z size r g b

where the position is in galactic coordinates, size is the particle’s weight and r g b is the emission coefficient of the particle, in arbitrary units. The Milky Way is made of three of them: Gas (~15,000 particles), Bulge (~900) and ArmDust (~93,000). The stars and the HII regions of the model are not part of the volume: they are genuinely point-like, so they keep their billboards and a voxel grid would only blur them.

Reading a few hundred thousand lines and painting them into a grid is not hard. What is hard is doing it in a way that does not depend on the resolution of the grid, so that a $200^{3}$ volume and an $800^{3}$ volume describe the same galaxy and differ only in how much detail they can hold. Most of the work below exists to enforce that.

Splatting

Every particle is a point with no extent, so the first operation is a scatter into the grid: each particle adds its weight to the voxel it falls in. That is the simplest possible splat, and it is what the shipped volumes use. It is also the one that keeps the property above, because a nearest-voxel scatter is the identity of the sampling: the same data always lands in the voxel the same ray will visit.

Splatting

Each particle adds its weight to the voxel it falls in, so a voxel ends up with the summed weight ρ and the weight-averaged color c; here its opacity is drawn proportional to ρ. The magnification shows the arithmetic for a voxel that receives two particles.

There are two details here worth taking into account:

The color accumulates with the same weight as the density, so dividing the accumulated color by the accumulated density yields a size-weighted mean color per voxel. Getting the weights inconsistent here is the classic way to end up with a volume whose large particles dominate the hue.
The grid is galactic but the file is not. Voxels are dumped in C order, like this:

$$ (W, H, D) = gal(X, Z, -Y) $$

This is because of the axis order the shader indexes a 3D texture with. The permutation is written into the metadata, in both orders, precisely so nobody has to infer it.

Smoothing without inventing detail

A raw scatter of 15,000 points into a $400^{3}$ grid is a swarm of dimensionless points, but we need a volume, so every channel is blurred with a Gaussian. The tempting choice, a fixed sigma in voxels, is exactly the one that breaks resolution independence: it smooths more of the galaxy at 200 than at 800, and the same data produces two different objects.

Instead, the blur width is a length on the particles, in world units of the box:

$$ \sigma = 0.85 \times \ell $$

where $\ell$ is the median distance from a particle to its 4th nearest neighbor — the local inter-particle spacing, not the volume divided by the count. Note that this measure is the same for all particles in a channel, as $\ell$ is derived from the median. We do not compute a different smoothing for each particle. That value measures the gaps that actually need filling, and it is nearly independent of resolution by construction, since it comes from the particle positions alone. A channel that is denser gets a narrower kernel for free, and the same particles written at 200, 400 and 800 get 4×, 2× and 1× that sigma in voxels, which is the correct behavior.

The 0.85 is empirical. A blur width given by hand in world units, per channel, never generalised; this little proxy held up across channels and resolutions.

Smoothing

The kernel width is a length measured on the particles, σ = 0.85 ℓ with ℓ the median distance to the 4th neighbor, so the same blur is described by twice as many voxels on a grid of twice the resolution. The result is one continuous density field; a fixed sigma in voxels would instead give a $200^{3}$ volume and an $800^{3}$ volume two different galaxies.

Two guards keep this sensible:

A cap at twice the channel’s own half-mass thickness, along galactic Z. Without it, a sparse channel (a few hundred particles whose “local spacing” is an artifact of sparsity) would be blurred into a slab.
A smoothing scale per channel. Dust uses 0.4×, because a dust lane that is smoothed like the gas stops being a lane and becomes a haze.

The color is blurred with the density kernel, over the unblurred density, so the color spreads exactly as far as the density is about to. Blurring the color separately is how you get colored halos around black voxels.

Color needs normalizing

The particle colors are emission coefficients, not display colors. Gas averages about 0.02, so clipping to [0,1] and multiplying by 255 rounds roughly 82% of the occupied voxels to zero and leaves the whole channel carried by about thirty distinct levels. Quantizing as-is gives a black galaxy.

The fix is a single scalar gain, derived from the 99.9th percentile of the occupied voxels and applied to all three channels identically. One scalar is the important part: the gain preserves the chromaticity of every voxel, so the hue is still the data’s. Per-channel normalization would give each channel the full 0–255 range and wash everything toward gray, which defeats the point of storing a per-voxel color at all. The gain is baked into the bytes, and the shader consumes the stored color as-is.

Density gets the same treatment, by its peak, and is stored in the alpha channel.

Packing and merging

Each channel is written as a raw heap of bytes, gzipped, next to a small text sidecar that declares the grid and the box:

Volume: emissive
Grid size: 399x40x400
Galactic grid (x, y, z): 399x400x40
Volume bounds: [-0.473301, -0.048570, -0.475536] to [0.477213, 0.046252, 0.476845]
Voxel size: 0.00250000 (cubic; the largest box side is 1, in units of the object's size)

The byte count per voxel selects the format (1 byte per voxel is R8 density, 3 is RGB8 color, 4 is RGBA8 color+density) and the loader takes it from the file length, not from the file name. The RGB channels carry the chromaticity, the alpha channel the brightness, and the emitted radiance is the product RGB × A. An occluding volume uses the same layout, with alpha as the normalized density.

The per-channel textures are then merged into two final volumes: one emissive (Gas at 0.7 + Bulge at 0.3) and one occlusive (ArmDust at 0.08). Merging is a post-process on the textures, so it needs neither the particle data nor a re-conversion, but it does have to be careful:

Densities add. The color is the density-weighted mean over the inputs that carry one, and each input’s baked gain is undone first, so the merge happens in emission space. Otherwise a channel normalized to a different peak would bias the hue of the result.
The merge verifies alignment (same grid, same box to 1e-3, metadata grid consistent with the file length) rather than assuming it.
Renormalizing the sum after the merge dims everything: at Gas+Bulge 1/1 the peak lands at 1.36 and the texture is scaled by 0.73. Whether to do it is a decision, not a detail.

Making the box fit the galaxy

A galaxy disk is 6.7 units wide and 0.67 thick, so a cubic box around it is nine tenths empty space: $400^{3}$ of RGBA is 244 MiB, where the same voxel size laid out as $400 \times 400 \times 40$ is 24 MiB. A factor of ten, spent on nothing.

The fix is to flatten the grid and the box together, deriving the grid from the data-shaped box: the detail of an $N^{3}$ grid without the empty axes. Flattening only the grid would make the voxels anisotropic; flattening both keeps them cubic.

Nothing else has to change. It is still one texture and one step count, because the ray-march step $rayLength \times |rayDir / extent|$ is invariant under a rescale of the box: flattening it does not change how many samples a ray takes.

The volume renderer

Here we go over the galaxy shader, which is in assets/shader/volume.galaxy.fragment.glsl. It is a single-pass ray marcher, but it is drawn twice: once early as a depth-only pre-pass, and once late as the emission pass.

But first, we need a piece of geometry that emits the pixels to the fragment shader were we do the work. Here, anything works, as long as the pixels reach the fragment. Since we need to render the galaxy also from the inside, we use a flipped cube at the position of the model. The transformations for the actual galaxy are done in the fragment shader, so rotating or scaling this box does nothing. It is essentially a vessel to emit pixels.

The box

The box is drawn with its front faces culled, so it never shows up as a surface: the ray crosses the near wall and every pixel it covers receives exactly one fragment on the far wall. The shader does not use that fragment as a surface, it only takes its world position as a direction from the camera. Since the near wall is never the one drawn, the camera can also sit inside the box and the screen is still full.

Wait, two passes?

We said we do an early depth pass and the emission pass at the end. Why is that? Well, the volume is rendered with additive blending so its own fragments cannot write a meaningful depth. Adding two semi-transparent layers is commutative; the depth buffer, which keeps a single number per pixel, is not. And the volume cannot be sorted per fragment, so any attempt to give it “the” depth in the color pass is wrong by construction.

But the scene needs it. The stars of the catalog, and the whole rest of the universe, are drawn after the Milky Way volume, and if nothing writes depth they all shine through the dust lanes. A galaxy whose dust hides nothing looks like a galaxy with a bright smudge on it.

So the depth is produced as a separate mask, in its own pass, before anything else is drawn:

Pass	Group	Color mask	Depth test	Depth write
`MODEL_VERT_GRID`	200	on	—	clears the buffer
`VOLUME_DEPTH`	250	off	—	`gl_FragDepth` only
`POINT_STAR`	300	on	on	off
`VOLUME`	2110	on	off	off

The pre-pass marches each ray only until it finds the first sample that is dense enough to occlude, writes that distance, and returns. The emission pass marches the same ray from the same origin and finds the same front, so both agree on where the dust starts. Stars are switched from “no depth test” to “depth test, no writes”: they are hidden by the dust but cannot overwrite its depth.

At a lower resolution

The volume is not the only thing that is allowed to render small. The volumes of the other galaxies (gas, bulge and dust) go into the same reduced buffer, and so do the nebulae, which are evaluated analytically in their fragment shaders instead of being sampled from a texture. All of them share the property that makes this safe: their content has no detail at the scale of a pixel, so the upscale costs almost nothing visible, while the fill rate they give back is the expensive part of the frame.

The depth is never reduced. Every one of those elements still writes depth at full resolution, in the pre-pass of the table above, and before the scene draws at the reduced scale that depth buffer is blitted onto the reduced one.

That copy is what keeps the reduced pass honest. Everything depth-tested in it, the stars above all, reads the reduced depth buffer, which the full-resolution pre-pass never wrote. Bringing the full-resolution depth down first means those tests see the dust front exactly where the full-resolution pass put it: without it the stars would shine straight through the lanes, and lowering the render scale would change the image instead of only its sharpness.

What “occludes” means

Not every volume should occlude. The gas is emissive, and its faint outermost haze is not opaque: letting it write depth would cut a hard, sharp-edged hole in the star field, perfectly matching the box. The dust, on the other hand, should hide what is behind its lanes.

So each volume slot carries its own density threshold, and only samples above it count as occluding. We define, for every channel, the threshold density above which occlusion happens:

"volumeDepthThreshold": [0.0, 0.015]

0.0 disables occlusion for that slot: the gas renders, and hides nothing. The value for the dust is not arbitrary, it comes from the data. In the dust volume, the low tail of the density histogram is the blur we applied deliberately, not matter, so the threshold is set to keep the lanes and drop the tail. The constant is per texture and has to be re-tuned if the smoothing or the normalization changes.

The march

Both volumes are marched in one loop, at the same position along the ray. That is what makes the ordering correct: a dust lane and a dust glow are the same voxels, and only the order of emission and absorption within the step decides which one you get.

vec3 rayOrigin = transpose(R) * u_cameraPos; // camera, in the box frame
vec3 rayDir = normalize(transpose(R) * v_data.fragPosWorld);

vec2 span = boxIntersection(rayOrigin, rayDir, boundsMin, boundsMax);
float stepUVW = ...; // TARGET_STEP_UVW of the box
int nSteps = clamp(int(ceil(pathUVW / stepUVW)), 16, u_iterations);
float jitter = hash12(gl_FragCoord.xy * u_time * 0.1); // hides the banding

for (int i = 0; i < nSteps; i++) {
 float t = entry + (float(i) + jitter) * stepSize;
 vec3 uvw = worldToVolumeUVW(rayOrigin + rayDir * t, boundsMin, boundsMax);

 // Jump whole bricks that hold no density in either volume.
 float skip = min(emptyBrickSteps(uvwGas, ...), emptyBrickSteps(uvwDust, ...));
 if (skip >= 1.0) { i += int(skip) - 1; continue; }

 vec4 gas = sampleVolumeRGBA(uvwGas, u_volume0, u_volume0ColorMode);
 vec4 dust = sampleVolumeRGBA(uvwDust, u_volume1, u_volume1ColorMode);

 // Emission first: light emitted in this step lies in front of this step's dust.
 emitted += volumeColor(GAS_RAMP, gas.a, gas.rgb, mode) * gas.a * stepUVW * transmittance;
 // Then absorption, Beer-Lambert, only for what is behind it.
 transmittance *= exp(-DUST_SIGMA * dust.a * stepUVWDust);
 if (transmittance < 0.001) break;
}

A few things in there are worth pointing out:

Empty brick skipping is the single biggest performance win. On load, the voxel grid is reduced to an occupancy grid with one texel per 4×4×4 brick, holding the maximum density of the brick. A zero texel then means every fetch inside that brick would have returned zero: no emission, no extinction, so skipping the whole brick cannot change the image. The shader does one nearest fetch instead of the dozens the brick would have cost, and advances the sample index rather than the step size, so the samples it does take sit on exactly the lattice it would have used anyway. On a galaxy volume, most of the march is empty space.
The jitter decorrelates neighboring pixels so a step size in UVW does not show as concentric banding.
The transfer function is $density \times color \times step$, integrated front to back, with an exposure factor applied at the end. Density is peak-normalized and sparse, so the ramp parameter is $log_2(1 + 255\cdot d)$. Without the log the whole volume lands in the first ramp stop and the ramp does nothing.

Known limits

Two, both of them consequences of doing the depth in a cheap pre-pass:

The depth is written at the first sample above the threshold, not where the ray actually becomes optically thick. A thin lane therefore occludes as hard as a thick one.
The distance is along the ray, while the log depth buffer wants view-space depth. They differ by $cos \Theta$ away from the view axis, which is exact at the center of the screen and a few percent at the corners.

Neither is visible in practice, and both are one comparison per sample away from being fixed properly.

Infrastructure in Gaia Sky

The renderer is only half of it. The other half is being able to load a volume from a data pack and declare it in a scene, which turned out to be mostly about formats.

A volume object is a regular ModelBody with the Volume archetype, so it accepts sizePc, parent, fadeIn/fadeOut, coordinates and everything else, plus the volume-specific attributes:

{
 "name": "Milky Way (volume)",
 "archetype": "Volume",
 "sizePc": 26000.0,
 "fadeIn": [1.0e3, 7.0e3],
 "fadeOut": [15e4, 60e4],
 "vertexShader": "shader/pbr.vertex.glsl",
 "fragmentShader": "shader/volume.galaxy.fragment.glsl",
 "raymarchIterations": [50, 70, 90, 110],
 "model": {
 "type": "sphere",
 "blendMode": "additive",
 "params": { "diameter": 1.3, "quality": 30 },
 "material": {
 "volume0": "$data/default-data/galaxy/volume/gal*/galaxy_emissive_rgba.raw.gz",
 "volume1": "$data/default-data/galaxy/volume/gal*/galaxy_occlusive_rgba.raw.gz",
 "volumeDepthThreshold": [0.0, 0.015]
 }
 }
}

raymarchIterations has one value per quality preset (low, normal, high, and ultra, adjustable from the settings dialog), so the same scene degrades gracefully on a weak GPU. Up to four volumes can be declared per object, and all of them are marched and composited in the same pass.

The loader (VolumeTextureLoader) does three things worth mentioning:

It crops. The grid is cut down to the range of voxels that actually hold density, plus one voxel of padding, and the cropped grid is mapped onto the same box. Writing the empty shell into the file costs file size and load time only.
It builds the occupancy grid described above, and binds it as a separate 3D texture. A slot without one gets a $1 \times 1 \times 1$ volume of ones, which never skips.
It derives the format from the bytes, not from the filename: 1, 3 or 4 bytes per voxel, with the grid from the metadata sidecar.

None of this is Milky-Way-specific: any 3D tool that can write an RGBA8 grid can produce a volume for Gaia Sky. The format and the scene attributes are documented in the data format and the volumes section of the Gaia Sky documentation.

Memory is the price: an 800-resolution grid is $798 \times 80 \times 800$ RGBA, about 200 MB once decompressed, per volume. The default data pack therefore ships three resolutions and picks one by data quality setting: 200, 400 and 800 voxels on the longest side of the box:

Resolution	Grid (galactic)	Voxels	Raw, per volume	Emissive, gzipped	Occlusive, gzipped
200	200 × 200 × 20	0.8 M	3.2 MB	230 KB	10.6 KB
400	399 × 400 × 40	6.4 M	25.5 MB	1.5 MB	74.6 KB
800	798 × 800 × 80	51 M	204 MB	8.9 MB	430 KB

Thanks to gzip (and to how much of the box is empty) the download cost is trivial; it is the decompressed texture that decides whether a card can hold it.

Results

The question a resolution-independent conversion has to answer is whether the three grids really are the same galaxy. They are: the mean ray integral (the integral of the density along a sample of rays through the volume, the quantity that has to match if the object is the same) is

Resolution	Mean ray integral
200	0.00020193
400	0.00020295
800	0.00020382

within 1% of each other, with the same total mass and the same hues, so what changes between them is only detail. And detail is visible: at 200 the dust cannot be smoothed at all (its kernel is 0.31 voxels, so the point cloud shows through), at 400 it is 0.63 voxels, and only at 800 does it reach 1.25 voxels and become a continuous lane. This is why the dust is the channel that decides which resolution you actually want, and why the 800 volume is the one in the preview video below.

Up to 800, the volume costs what the step count says, not what the grid says. Frame time for the Milky Way alone, at QHD (2560x1440) with the camera at about 34.5 kpc above the galactic disk, on an NVIDIA GTX 1070 (driver 580.178.04) on Arch Linux. I deliberately chose to use this very old card instead of my other RTX 5080 because it may be more representative of the graphics computing power present in common household PCs and laptops. Tests have been carried out with only the base dataset loaded.

Resolution	low (50 steps)	normal (70 steps)	high (90 steps)	ultra (110 steps)
200	3.5 ms (285 fps)	3.8 ms (259 fps)	4.3 ms (231 fps)	4.6 ms (216 fps)
400	3.4 ms (289 fps)	3.8 ms (260 fps)	4.1 ms (239 fps)	4.5 ms (222 fps)
800	4.3 ms (230 fps)	4.5 ms (220 fps)	4.7 ms (212 fps)	5.2 ms (192 fps)
1300	5.0 ms (198 fps)	5.8 ms (172 fps)	6.2 ms (161 fps)	6.8 ms (145 fps)

With that hardware the resolution is almost free: going from 200 to 800, sixty-four times the voxels, costs under a millisecond at every preset (3.5 → 4.3 ms at low, 4.6 → 5.2 at ultra). This is possibly because the ray march is per-pixel and the grid does not change how many steps a ray takes. That is why I also built a 1300 volume, which no shipped dataset uses: 274 times the voxels of 200, for 1.5 to 2.2 ms (3.5 → 5.0 ms at low, 4.6 → 6.8 at ultra). Even at that scale the cost stays far below the voxel count, but it is no longer invisible: 1300 is now the most expensive thing in the table, ahead of the preset.

The step count is also interesting. Current presets are 50 at low against 110 at ultra, and even that only adds 0.9 ms at 800 and 1.8 ms at 1300. The slowest cell in the table is 6.8 ms (1300, ultra) at QHD, 41% of the 16.7 ms a 60 fps frame allows, so even that volume is not limited by frame time on this card.

What the card does pay for is memory. These are the sizes as decompressed and uploaded to VRAM, at 4 bytes per voxel (RGBA8), for the flattened, cropped boxes the loader actually builds:

Resolution	Grid	Voxels	VRAM
200	200 × 20 × 200	0.8 M	3.2 MB
400	400 × 40 × 400	6.4 M	25.6 MB
800	798 × 80 × 800	51.1 M	204 MB
1300	1300 × 130 × 1300	219.7 M	879 MB

Even the 1300 volume fits in the 8 GB of a GTX 1070 with room to spare. That is the flattening earning its keep: the same resolution in a cubic box would be 8.8 GB, so the high-resolution row could not have been run at all.

The stills below are where the argument becomes visible, each pair sharing a camera. The top view is where the billboards fall short, and for a reason that is easy to get backwards: seen face-on, so few billboards overlap along the line of sight that there is almost no galaxy left.

Billboards, 3.8.x

Volume, 3.9.0

Top view comparison between rendering methods

Top view. With so few billboards along the line of sight the galaxy almost disappears; the volume still shows the disk and its dust lanes. Volume at 800 resolution.

Edge-on, the billboards look plausible at first glance, but they blow out, while the volume rendering stays consistent. The same additive blending that was too faint from the top saturates here: the bulge becomes a flat white patch and the disk loses its internal structure. The volume stays a thin dark lane with its dust intact. It is also dimmer overall, which is expected, because there the light is absorbed rather than added.

Billboards, 3.8.x

Volume, 3.9.0

Edge view comparison between rendering methods

Edge view. The billboards saturate, the volume does not. Volume at 800 resolution.

Dust is where the grid resolution shows. Three close-ups of the same lane, same camera, 110 iterations each:

Dust lanes at three grid resolutions

The same close-up at 200, 400 and 800, 110 iterations. At 200 the fog is grainy, because the dust kernel is thinner than a voxel and the point cloud shows through. 400 is already smooth and 800 is smoother still, but the two are hard to tell apart at this framing. The stars and HII regions look the same in all three: they are billboards and never go through the grid.

The last comparison is the two ends of the preset range. Low is the 200 grid and 50 steps, ultra is the 800 grid and 110 steps, so both knobs move at once and this pair shows the worst case rather than the effect of either one alone.

Preset: low

Preset: ultra

Top view preset comparison

The low preset versus the ultra preset, top view.

Preset: low

Preset: ultra

Preset comparison from inside the disk

The low preset versus the ultra preset, from inside the disk, looking at the galactic center. Low uses the 200 galaxy resolution and 50 iterations. Ultra uses the 800 galaxy resolution and 110 iterations.

Against ultra, the low preset is softer and hazier, and the dust has less contrast. That is intended: low is meant for modest hardware, and it still looks decent there.

Below is a video I captured with the 800 galaxy running in Gaia Sky.

Volumetric Milky Way rendering in Gaia Sky

Credits: The Gaia Sky Project, ESA/Gaia/DPAC.

Conclusion

For a decade the Milky Way in Gaia Sky was a compromise between what the data said and what a rasterizer could do with a pile of billboards. The volume rendering path fixes this: the galaxy is now a medium that emits and absorbs. The dust lanes hide the stars behind them and not the ones in front, and it no longer blows out or fades to nothing depending on where the camera is.

Two things made it work, and neither of them is in the shader itself. The first is the conversion: sizing the blur on the particles rather than on the grid is what makes all resolutions the same object at different levels of detail. Flattening the box to the shape of the data makes real savings in memory (256 vs 25 MB for the $400^{3}$ box). This is what lets the unshipped 1300 grid fit in the 8 GB of a GTX 1070 and still draw a frame in under 7 ms. The second is the depth pre-pass: a volume cannot write meaningful depth in a blended pass. The rest of the scene has to know where the dust is, so the cheap early pass does the trick here.

So far, all this work only affects one galaxy: our own. Gaia Sky can already generate galaxies procedurally, but in 3.9.0+ that system still produces billboard particles. The next step is to adapt it to produce and render volumes. Other than that, what is left are smaller fixes rather than architecture:

Sizing the ray-march step from the grid resolution instead of the box, so a finer grid is actually sampled finer.
Replacing the time-hashed jitter with a static one (the current float32 clock dies after about ten hours of uptime).
Enabling scattering in the dust, which is already wired with a coefficient of zero.
Tuning the occlusion threshold, which is currently a single constant for the whole volume and would probably be better as a curve.

Finally, all this work has been done assuming we start from a particle-based distribution, which is a weird place to start for such a project. Typically, one builds the volume from scratch via analytical distributions without going through a particle dataset. For DR4, due on December 2, we will build a volume directly from Stefan’s data, bypassing the particles altogether.

Astronomy Picture of the Day: Here comes the Sun

https://science.nasa.gov/image-article/apod-2016-september-26-gaia-here-comes-the-sun/ ↩︎
Stefan Payne-Wardenaar BlueSky profile

https://bsky.app/profile/stefanpw.bsky.social ↩︎
Gaia’s Milky Way map

https://www.cosmos.esa.int/web/gaia/milky-way ↩︎
Morgan McGuire and Louis Bavoil, Weighted Blended Order-Independent Transparency, Journal of Computer Graphics Techniques (JCGT), vol. 2, no. 2, 122-141, 2013.

Available online: http://jcgt.org/published/0002/02/09/ ↩︎

JPEG XL finally lands in Chrome!

Toni Sagrista Selles — Wed, 07 Oct 2026 00:00:00 +0000

Today, I was browsing Hacker News, when a particular item caught my eye:

Hacker News post with the Chrome JPEG XL announcement.

I opened it promptly and, indeed, it was not a prank. Finally, after years and years of bull**** from the Chrome team, they are shipping it. They are shipping a JPEG XL decoder.

JPEG XL in a nutshell

For those of you who don’t know, JPEG XL is a new image codec that includes most of, if not all, the features we’d want in the image codec of the future:

File size reduction by $~20-60\%$ w.r.t JPEG.
Lossless JPEG transcoding.
Progressive decoding.
Wide gamut, HDR and 32-bit support.
Animations and transparency (alpha channel).
Shines with high-fidelity photographic images.
Fast-ish encoding and decoding.
Royalty-free and FOSS.
Support for super high-resolution images, of up to 1 terapixels ( $2^{30}-1$ pixels per side).

This is awesome news for the codec. Whether we like it or not, Chrome is the dominant web browser, with ~66.5% of the market share, very far from its competitors (Safari, Edge, and Firefox, in that order).

Source: StatCounter Global Stats - Browser Market Share

Now, I don’t use Chrome at all, but I still think that this was just the final piece of the puzzle needed to help the format go mainstream.

So, what finally pushed them over the edge? If you read their announcement, they cite “consistent feedback and requests from web developers” and its massive popularity in the Interop 2026 project. Translated from corporate PR-speak: the community simply refused to let it go. Web developers, photographers, and open-source advocates kept pushing, complaining on bug trackers, and demanding a truly superior image format instead of settling for Google’s own WebP or the heavily pushed AVIF.

The technical excuse the Chrome team used for years was that a C++ decoder was too much of an attack surface to run safely in the browser. Well, it has now been elegantly sidestepped. To bring JPEG XL to Chrome 155, they have integrated jxl-rs, a pure Rust reimplementation of the decoder. This tackles the memory safety issues head-on, eliminating the risk of out-of-bounds reads and use-after-free bugs of other decoders. And to make sure it doesn’t run sluggishly, they built a SIMD abstraction layer (jxl_simd) to maximize hardware performance without compromising Rust’s safety guarantees.

This is a massive victory. Not just for JPEG XL, but for the open web.

As I demonstrated in my comparison back in 2023,JXL has significant advantages over older formats and, in many cases, over AVIF as well, especially for high-fidelity photography, lossless compression, and progressive decoding. The fact that it allows lossless transcoding of existing JPEG files is of special importance. It means that we can finally upgrade our backlog of old images without losing a single pixel of quality, while saving a big chunk of disk space and/or bandwidth in the process.

With Firefox adding it to Labs earlier this year, and Chrome finally capitulating, Safari is left as the major outlier with only partial support.

Notice

Firefox 158 will include the JPEG XL decoder enabled by default. See the beta release notes.

The dark days of “Google kills JPEG XL” are officially over. It’s time to start converting your image libraries, updating your <picture> tags, and serving .jxl. The future of (high-quality) web imagery is finally here, and I’ll be ready for it.

Git hash prefix or cellphone number?

Toni Sagrista Selles — Tue, 29 Sep 2026 00:00:00 +0000

Today, I was playing around with a new feature in Gaia Sky, and I created a new release from master. Immediately, I noticed something unusual in the generated package name:

packages-3.8.0.066962508

The 3.8.0 part is the release version, while the last 9 characters are the abbreviated Git commit hash. Usually, that last part looks something like:

packages-3.8.0.a31f7c92e

This time, however, every single one of the hash characters is a number, 066962508. My first stupid reaction was thinking that something had gone wrong. However, git log confirmed the hash was fine.

➜ git log
commit 066962508de536403a185471344ba83801d45a88 (HEAD -> master, origin/master, origin/HEAD, gitlab/master, gitlab/HEAD, github/master, github/HEAD)
Author: langurmonkey <tsagrista_AT_ari.uni-heidelberg.de>
Date: Tue Sep 29 10:10:15 2026 +0200
feat: Handle gaiasky:// URLs forwarded from second instances and macOS open events. Part of #935.
- Scope dataset URL dedup to the startup path only, so re-sent URLs are always processed by the running instance.
- Register install4j StartupNotification listener for macOS open-URL events, with pending-URL queue for pre-UI arrival.
- Add --enable-native-access=ALL-UNNAMED to launcher VM parameters - Add compileOnly install4j-runtime dependency.

Obviously, I asked myself, “how likely is that”? So I did some digging.

Git commit hashes are hexadecimal. That means each character can be one of 16 possible values:

0 1 2 3 4 5 6 7 8 9 a b c d e f

So there’s nothing special about a hash starting with 066962508. It’s just a hexadecimal string that happens to contain no a–f characters in its first nine positions.

But how unlikely is that?

There are 10 numerical characters out of 16 possible hexadecimal characters, so the probability of a single character being a number is:

$$ \frac{10}{16} = 0.625 $$

For nine consecutive characters to all be numbers:

$$ \left(\frac{10}{16}\right)^9 \approx 0.0145 $$

That’s about 1.45%, or roughly 1 in 69.

So it’s not particularly likely, but it’s certainly not extraordinary either. Given enough commits, you’re bound to run into one eventually. It looks even stranger because of the leading zero, and a quick glance at the number may even make someone mistake it for a Spanish cellphone number.

I then asked myself what the probability of all hash characters being numbers is. Well, if we assume ~40 characters (this is the default for SHA-1 hashes, even though git also supports SHA-256 with ~64 characters), the probability of every single one being a number is:

$$ \left(\frac{10}{16}\right)^{40} \approx 7.89 \times 10^{-9} $$

That’s about 1 in 126.7 million. Much more unlikely, but still not totally outlandish. But I digress.

Back to the case, I checked whether this was the first time this had happened in the Gaia Sky repository:

➜ git log --all --format='%H'
 | grep -E '^[0-9]{9}'
 | wc -l

123

Nope. It has happened over a hundred and twenty times, and I had simply never noticed it before. I’m pretty sure this is the first time I package a build from one of those commits though.

JPEG XL lands in Firefox

Toni Sagrista Selles — Wed, 17 Jun 2026 00:00:00 +0000

Yesterday, Firefox 152 was released. In the release notes, under the Firefox Labs section, it states:

The Firefox Labs section of the release notes of Firefox 152. If you see JPEG-XL works!, it means your browser is JPEG-XL capable. If you see JPG :(, it means your browser does not support it.

Firefox now offers experimental support for the new JPEG XL image format, which generally provides better compression than WebP, JPEG, PNG, and GIF and is designed to supersede them. You can enable it from the Firefox Labs panel in Settings.

Finally! I have written about this image format here, here, and here. The latter entry is particularly important because Google set the ball rolling by announcing that they would look into adding a memory-safe JPEG XL decoder into Chromium. It was only a matter of time before the other browsers would follow suit.

I have not had the chance to test this myself, as at the time of writing, firefox-152 has not yet landed on the Arch repositories. However, it won’t be long. I’m just happy that it is finally happening. That’s all.

Edit (2026-06-17 09:26 UTC): Well, a little over an hour after I wrote the post, and version 152 is live on the Arch extra repository. I tested it quickly and it seems to work. The JPEG XL test page indicates that alpha transparency and animation work. I can’t test the wide gamut feature as I don’t have a P3 display.

Local agentic workflows are getting crazy good

Toni Sagrista Selles — Tue, 12 May 2026 00:00:00 +0000

Building on the previous post about how good current open TTS models are, this post focuses on local LLMs, and their explosion in quality in the recent ~~years~~ months. I have been using OpenRouter with, for the most part, DeepSeek v4 Flash. It is a very capable model, dirt-cheap (seems to be heavily subsidized, at $0.14/M tokens input and $0.28/M tokens output), and it works very well when it is free to browse the internet. I have also been tinkering with local models like the Qwen3.5 family, Gemma 3/4, Mistral, and more. They are good, I haven’t been able to use the versions that can run reasonably on consumer hardware for anything too productive. That is until now. The brand new Qwen3.6 models, particularly the 27B dense, and the 35B A3B MoE, have really impressed the community. They are trivial to run on common setups (e.g. 16GB RTX card, ~32-64GB RAM), and when boosted with agentic capabilities, they perform very well.

My use case is quite modest. I typically use LLMs for research. I explain the problem and propose a couple of solutions, and ask for feedback. When I’m new to a topic, I like to get a bird’s eye view type of summary first, and then I ask for further explanation and deeper insight on certain areas that interest me. If possible, I add the sources that may help it gain more knowledge on the subject. Sometimes, I ask it to proofread my writings, but I tend to restrict it to fixing grammatical and spelling errors, with the occasional wording suggestion. I think writing is valuable and useful, as it helps with structuring my thoughts and organizing my ideas. That’s why I frown upon AI-generated blog posts. Additionally, I don’t like the overly verbose and structured texts it generates. I don’t do much AI-assisted coding. Sometimes, I may ask it to generate the odd Python script to solve a specific task that would otherwise get tedious.

My setup is like this: For casual questions and general chatting, I use OpenRouter chat for remote models and LM Studio for local models. Usually, I run the models on a gaming laptop (RTX 5080 16GB, 32GB RAM) and serve them to my machine via LM Link. For researching and more complicated workflows, I use the Hermes Agent. It is open source (MIT), fully-featured, self-improving (learns between sessions!), and offers everything I need with minimal configuration. It works very, very well. I connect it to either DeepSeek V4 Flash via OpenRouter, or to my local LM Studio, which is tethered to the inference machine via LM Link.

The Qwen3.6 27B and 35B A3B models, released in April 2026, have made strides in the community. They are extremely capable for their small size. They don’t have a ton of knowledge on all domains like the near-trillion-parameter frontier models, but when paired with the right toolset, they get very close. They don’t fit in most GPUs, but they can be partially offloaded. The MoE model in particular, with only 3B parameters active during inference, is very fast. I get ~35-40 tok/s with the 35B A3B MoE model, and about 4.5 tok/s with the 27B dense model. The dense model is borderline unusable, so I almost always rely on the MoE model. I offload 19 layers to the GPU (~12.5 GB out of a total of 26.63 GB), and use the maximum context size of ~262K tokens. This is plenty for agent sessions and works very well. The model is able to plan and execute actions very efficiently. It tends to overthink a lot, but this is an acceptable trade-off given the output quality. On OpenRouter, this exact same model is more expensive than DeepSeek V4 Flash, at $0.15/$1 per million tokens input/output. Being able to run such a model locally feels like a superpower.

So, now the question is obvious. Will local LLM get good enough in the future to replace frontier models? I’d argue that it depends on the task. Frontier models have a vast amount of knowledge and are able to one-shot many complex problems successfully. They will continue to be more suited for bleeding-edge research for the years to come. But they are also very expensive, very taxing on the environment, and the primary reason for the sky-high current hardware prices. There is certainly a case to be made for local and more economic small models.For day-to-day agentic workflow, I think open models are very close already. I would bet that in the next 2 years they will get on pair for most everyday tasks. Is this prediction grounded on reality, or just wishful thinking? I’d like to think the former. In any case, I’ll be here, watching, tinkering.

Local TTS is getting very capable and accessible

Toni Sagrista Selles — Wed, 29 Apr 2026 00:00:00 +0000

Around 2007 I spent half a year in the University of Aberdeen working on my final year project involving NLP. The project consisted of a series of modules to build an graphical adventure-type game controlled by language input in the form of text. It also featured speech output generated by an early TTS system. To achieve this, we managed to partner with a group at La Salle University that were working on a TTS system for Catalan. It was a closed system that was accessible via a web API, but that option was far too slow for real time use. I ended up preprocessing the audio of all dialogs in the project into WAV files just to be able to play them in sync with mouth movements. At that time, I was amazed that a computer could so easily convert text to an understandable audio file. The voice was very plain and robotic, with no emotion whatsoever. The results were hit or miss, but it worked.

Fast forward to today, TTS systems are everywhere. Several groups have released low-parameter TTS models that run very well on consumer hardware. I have been using the lightweight Kitten TTS for a while with fantastic results.

The models are so lightweight that some websites are heavier than entire Kitten TTS models:

Model	Parameters	Size	Download
kitten-tts-mini	80M	80 MB	KittenML/kitten-tts-mini-0.8
kitten-tts-micro	40M	41 MB	KittenML/kitten-tts-micro-0.8
kitten-tts-nano	15M	56 MB	KittenML/kitten-tts-nano-0.8
kitten-tts-nano (int8)	15M	25 MB	KittenML/kitten-tts-nano-0.8-int8

Projects like puss-say streamline and trivialize Kitten TTS inference. I have a shell script in one of my bin directories that does everything in a single command:

#!/usr/bin/env bash
uvx --from 'git+https://github.com/Mic92/puss-say' puss-say "$@"

This clones the project, pulls dependencies and models, and plays the audio. It is quite fast, especially when using cached data. Kitten TTS produces acceptable results, though the output usually lacks emotion and nuance. For simple use cases (reading notifications, generating voiceovers for scripts) it’s more than sufficient.

Qwen3-TTS, which I’ve been recently testing, represents a step-up in quality. It’s extremely good, and local inference is practical even on modest hardware given the model sizes. It offers three interesting variants:

Parameters	Hugging Face ID	Best for	VRAM
1.7B	`Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign`	Free-form voice descriptions	~6 GB
1.7B	`Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice`	Preset speakers + style instructions	~6 GB
0.6B	`Qwen/Qwen3-TTS-12Hz-0.6B-Base`	Voice cloning from reference audio	~2 GB

The voice design models are particularly clever: you describe the voice you want alongside the text to convert. Want a deep, gravelly voice with a Scottish accent? Or an excited teenager talking about a video game? Just describe it. It’s remarkable that you can run this locally so easily. However, as far as I know there’s no off-the-shelf CLI tool that handles dependencies, downloads the model, and runs inference out of the box.

That’s why I created QwenSay. With it, you can clone the repository and convert text to speech locally from your terminal without wrestling with dependencies or writing any code.

Here’s how it works. First, set it up:

# Clone the repo
git clone ssh://git@codeberg.org/langurmonkey/qwensay.git
cd qwensay

# Default dependencies
uv sync
# For modern RTX cards, use flash attention
uv sync --extra gpu
# For Pascal (GTX 10x0 family), you need a special CUDA version
uv sync --group torch-pascal --no-group torch-default

Now, you are ready to convert your text to speech with Qwen3-TTS:

uv run qwensay.py \
 --text "Good morning! Today is going to be a great day." \
 --instruct "A cheerful, energetic young woman with a clear American accent"

This uses the default 1.7B voice design model. You can also specify the model with --model. There are many other CLI arguments that you can use to tune your output. Check out the repository documentation for more details.

Whether you’re building accessibility features, creating voiceovers for projects, or just experimenting, this is worth a try. I’ve made QwenSay my go-to TTS tool because it produces high-quality results and is genuinely fast.

Fine-tuning Qwen3.5 for Gaia Sky

Toni Sagristà Sellés — Wed, 01 Apr 2026 00:00:00 +0000

A little over a year ago I set up a local pipeline to use different LLMs to respond to Gaia Sky questions using RAG. In that post, I built a dynamic scrapper that parsed the Gaia Sky website and documentation and ingested the content it into a vector database. Then, I built a minimal terminal chatbot interface that received the user prompt, queried the database for semantically similar data, and built up the context for each LLM call. The results were promising, and I found that they (obviously) strongly depended on the model used.

Fast forward a few months, and the Qwen 3.5 models were released by Alibaba. The general consensus is that they are quite good for their size. I’ve been testing them for local inference with a similar impression. I thought that it would be interesting to repeat the exercise of creating a Gaia Sky AI assistant, but using a radically different approach: Instead of RAG, I would fine-tune the model itself. In this post, I describe this fine-tuning project, from the creation and engineering of the training dataset to the fine-tuning and production of the final GGUF models.

This project is composed by two, very distinct parts, which map to top-level chapters in this post:

Training dataset creation
Fine tuning

At the end I quickly evaluate the results in the testing section.

The source code, dataset, and models discussed in this post are in the following repositories:

Dataset creation and fine-tuning - gaiasky-finetune
Gaia Sky training dataset repository - gaiasky-training-dataset
Qwen3.5 Gaia Sky fine-tuned models - gaiasky-qwen-3.5-gguf

Hardware

Here is the hardware I have used to create the dataset and fine-tune the model:

Desktop PC - Arch Linux, Intel(R) Core(TM) i7-7700 (8) @ 4.20 GHz, 32 GB RAM, NVIDIA GeForce GTX 1070 8 GB.
Laptop 1 - Windows 11, WSL2 (Arch Linux), Intel(R) Core(TM) Ultra 9 275HX (24) @ @ 3.07 GHz, 32 GB RAM, NVIDIA GeForce RTX 5080 Mobile 16 GB.
Laptop 2 - Arch Linux, Intel(R) Core(TM) i7 8750H (12) @ 4.10 GHz, 16 GB RAM, NVIDIA GeForce GTX 1060 Mobile 6 GB.

Training dataset creation

The creation of the training dataset is the most important piece of work in this project. It is composed of three parts:

Documentation dataset
API dataset
Identity

Documentation dataset

When I started this project, my first instinct was “more is better.” I thought that if I fed the model every single .java, .glsl, .rst, and .md file in the Gaia Sky repositories (project, documentation, etc.), it would emerge as an expert. Oh boy, was I wrong.

A large codebase contains a lot of boilerplate noise. Getters, setters, license blocks, and infrastructure code that doesn’t actually help a model understand how the engine works or how to write scripts for it. I soon realized that the dataset is the single most important part of the project, and it needed a surgical approach.

The plan was to automate the process of creating the dataset to a degree, and then use it to fine-tune the Qwen 3.5 4B and 8B model variants.

I wrote generate-raw-dataset.py to act as a high-pass filter. Instead of a blind crawl, I implemented an allowlist system. I would only let in the load-bearing files:

Documentation: By far, the most important data, containing exhaustive human-written documentation pages. We convert the documentation RST files to Markdown with pandoc, and then we add some additional key files, like the project’s README.md.
Core Logic: Selected Java files that are representative of the brain of the engine (main loop, scene, renderer, etc.).
Visual Logic: Selected shader files that define the look of the cosmos (stars, particles, PBR, etc.).

Almost every source file in Gaia Sky starts with a copyright header. This is “dead weight” for training. I added a regex-based stripper to ensure the model’s limited context window was filled with code logic, not license text:

def strip_java_copyright(content):
 # Matches the opening /* until the first */ at the start of the file
 content = re.sub(r'^/\*.*?\*/', '', content, count=1, flags=re.DOTALL)
 return content.strip()

The output of this first phase was a jsonl file where each line represented a single, cleaned-up file. It looked like this:

{
 "instruction": "core/src/gaiasky/util/coord/Coordinates.java",
 "output": "public class Coordinates {\n // Implementation of transformation matrices...\n}",
 "source_file": "core/src/gaiasky/util/coord/Coordinates.java"
}

This provided the “Context” for the next phase. However, a model trained directly on this would just learn to autocomplete files. To make it an assistant, we had to turn these files into a conversation.

Once I had a clean extraction of the most relevant information pieces, I faced a new problem. A raw dump of a Coordinates.java file is great for a search engine, but it is not a conversation. To turn these files into training data, I used a “teacher” model, Qwen 3.5 27B, to look at each file and generate a specific number of Q&A pairs. I wrote distill-dataset.py to handle this. The script calculates how many questions a file is worth based on its length and type. A long documentation file might get 25 Q&A pairs, while a short shader might only get 4. Below is an extract of the method that computes the number of target pairs for a file.

def calculate_target_pairs(file_path, content):
 """Determine how many Q&A pairs to generate based on file metadata."""
 filename = file_path.lower()
 content_len = len(content)

 # Skip files that are essentially empty
 if content_len < 150:
 return 0

 # Set a modest base count that depends on type
 # Docs are high-value, so they start slightly higher than code
 if filename.endswith(('.md', '.rst')):
 base_count = 4
 # 1 pair per 750 characters for documentation
 chars_per_pair = 750
 else:
 base_count = 2
 # 1 pair per 1500 characters for source code
 chars_per_pair = 2000

 # Calculate length-based bonus without a tiny cap
 # This scales better for massive files while keeping short files lean
 length_bonus = content_len // chars_per_pair

 # Sum and apply a reasonable maximum
 final_count = base_count + length_bonus

 # Cap it to avoid overwhelming the distillation for single files
 # but ensure even tiny files get at least the base_count
 return max(base_count, min(final_count, 20))

Initially, I used the MoE Qwen 3.5 30B A3B, but it was consistently outputting the wrong format. Then I switched to the 27B dense model, and it performed a little better. Even so, I had to tell the model exactly how to behave. Here are the key items I learned the hard way:

Match the answer type: If the question doesn’t ask for code, don’t provide it.
Grounding: Every claim must be directly grounded on the source text.
Diversity: Every question must cover a different detail.

I also found that, at these model sizes, it is better to batch Q&A pairs instead of asking the model to provide 20 of them in one go. I finally gravitated to 3 Q&A pairs per inference call. To prevent the model from repeating itself across batches, I tracked existing questions and fed them back into the prompt as exclusions.

The prompt is constructed as follows:

# A snippet from the distillation prompt
prompt = (
 f"You are a Gaia Sky technical expert. {file_hint}\n"
 f"Generate exactly {current_target} Q&A pairs strictly based on the source text below. "
 "Every fact and claim in your answers must be directly grounded in the source text. "
 "Never invent API calls or function names not present in the source. "
 "Match the answer type to the question: if no code is asked for, give no code. "
 "HIGHLY DIVERSE Q&A pairs. Every question must cover a COMPLETELY DIFFERENT topic or detail "
 "of the source text than the ones already provided. "
 f"{avoid_context}\n\n"
 "Format the output as JSONL: one JSON object per line, with the question in the "
 "\"instruction\" field and the answer in the \"output\" field. "
 "Output ONLY the JSONL, nothing else.\n\n"
 f"SOURCE FILE: {file_path}\n"
 f"{'=' * 60}\n"
 f"{file_content[:context_limit]}\n"
 f"{'=' * 60}"
)

It consists of the following parts:

Base text - This is composed by the raw strings in the prompt variable.

The file hint ({file_hint}) - We add hints depending on the filetype. The following table displays the hint for each type.

Filetype	Extensions	Hint
Java	`.java`	This is Java source code. Focus on class responsibilities, method signatures, and architectural patterns. Do NOT generate Python scripting examples.
Python	`.py`	This is Python scripting code for the Gaia Sky API. Questions about usage and parameters are appropriate.
Shader	`.glsl`	This is a GLSL shader. Focus on the rendering technique, uniforms, and mathematical operations. Do NOT generate Python scripting examples.
Docs	`.md` `.rst`	This is documentation. Focus on concepts, features, workflows, and user-facing features.

The pair count ({current_target}) - Contains the number of Q&A pairs to generate.

The previous Q&A pairs, if any ({avoid_context}) - This is constructed by listing the existing pairs, as parsed by the program in the output, or accumulated in the current run.

 instr_list = [p['instruction'] for p in existing_pairs]
 avoid_context = (
 "\nDo NOT repeat or rephrase any of these already-covered questions:\n"
 + "\n".join(f"- {instr}" for instr in instr_list)
 )

The filepath and content ({file_path}, {file_content}) - Contain the filename and the actual content, which is capped to fit within the context length.

However, local LLMs tend to be chatty. Even with such strict instructions, larger models like the 27B sometimes also mess up. From time to time, this model would leak its own reasoning into the output. It would start its response with "The user asked to provide 34 Q&A pairs given the source material..." or it would include meta-talk like "This question is safe because it is derived from line 45."

This created dirty data that polluted the dataset and undermined the fine-tuning process. If I trained on this, the final model would start every answer by talking to itself.

To fix this, I built sanitize-jsonl.py. This script is a heavy-duty cleaner that uses regex to strip out training artifacts. It first tries to rescue bad rows, and if it fails, it deletes them. If the model accidentally put both the question and answer in the output field, the sanitizer attempts to detect the question mark and splits them back into the correct structure.

Here is a look at what the data looked like before and after the sanitization process.

Raw distilled entry

{
 "instruction": "I need to generate 3 pairs based on the Gaia Sky API. Here is pair 1:",
 "output": "Question: How do I set the camera position? Answer: Use the gs.setPosition(pos)
 method. I ensured this doesn't repeat previous pairs.",
 "source_file": "APIv1.md"
}

Sanitized “gold” entry

{
 "instruction": "How do I set the camera position?",
 "output": "Use the gs.setPosition(pos) method.",
 "source_file": "api.md"
}

The sanitizer also had to deal with Javadoc remnants. Since Gaia Sky is a Java project, class- and method-level comment blocks are full of HTML tags like <p>, <ul>, and <li> (Javadoc syntax). The script converts these into clean Markdown so the LLM learns a consistent documentation style.

By the end of this process, I had gaiasky-gold.jsonl. This contains a curated, clean list of questions and answers based on the whole project documentation.

API dataset

Documentation is important, but I want the model to learn some Gaia Sky scripting as well. To do so, a new API/scripting dataset needed to be generated. To solve this, I built a synthetic data factory designed to teach the model both the content of the API and the context of how to use it.

The first step was grounding the model. I wrote a script (api-extract.py) that scans the Java source files and uses Regex to pair Javadoc comments with their corresponding method signatures. The extract_from_file() method uses regular expressions and a little bit of logic to generate Q&A pairs based on method signatures and their Javadoc documentation:

Code: Extract API Q&A pairs from a file


def extract_from_file(file_path):
 """Extracts Javadoc blocks and method signatures from a single Java file."""
 if not os.path.exists(file_path):
 print(f"Skipping: {file_path} (not found)")
 return []

 with open(file_path, 'r', encoding='utf-8') as f:
 content = f.read()

 # Regex: Matches Javadoc and the following method signature
 pattern = re.compile(r'/\*\*(.*?)\*/\s+([^;{]+)[;{]', re.DOTALL)

 extracted = []
 for match in pattern.finditer(content):
 doc = match.group(1).strip()
 # Clean Javadoc: remove leading stars and excess whitespace
 doc = "\n".join([line.strip().lstrip('*').strip() for line in doc.split('\n') if line.strip()])

 method_sig = match.group(2).strip()
 # Clean method: remove annotations (like @Override) and newlines
 method_sig = re.sub(r'@\w+(\(.*\))?', '', method_sig)
 method_sig = " ".join(method_sig.split())

 extracted.append({
 "instruction": f"How do I use the {method_sig.split('(')[0]} method in Gaia Sky?",
 "output": f"The method `{method_sig}` is used for: {doc}",
 "source_file": f"API_Reference/{os.path.basename(file_path)}"
 })
 return extracted

This process produces the gaiasky-api-raw.jsonl, which is used in the next step. It contains the API calls with their respective documentation.

However, knowing a function exists isn’t enough. The model needs to know how to script with it. To address this, I developed generate-scripting-dataset.py to transform those raw Java signatures into a diverse pedagogical dataset. As input, it gets all test and showcase scripts in the Gaia Sky repository, and the raw API JSONL file. It produces four types of output, termed A, B, C, and D:

Type A: The API reference

These are direct “How do I use X?” pairs. They include the parameters, the return types, and a basic Python example.
Type B: The task synthesis

This step is optional, and I ended up not including it in the final dataset. However, I think it is still worth mentioning. I used the larger teacher dense model (27B) to generate complex tasks (e.g., “Write a script that navigates to Mars, waits 5 seconds, and takes a screenshot”). The script provided the teacher model with a safe list of real functions extracted in Step 1 as a sort of guardrail. If the teacher tried to hallucinate a command, the script flagged and discarded it.

The results of this section were kind of underwhelming, possibly because more parameters are needed for such open-ended tasks.
Type C: Adversarial error correction

This is my favorite part. I programmatically broke the API calls to teach the model how to fix its own mistakes. The script would generate a wrong script (e.g., using snake_case instead of camelCase or missing a required argument) and then provide the correct version. The end goal was to prevent common LLM failures before they happen.
Type D: The “gold standard” Library

Finally, I indexed the actual test and showcase scripts from the Gaia Sky repository. These are human-written, battle-tested scripts that show the model how to handle complex logic, loops, and math.

Identity

Finally, I prepared a small identity.jsonl file with essential project information that must appear in the final integrated training dataset. It only contains 17 lines of Q&A, but it is rather important. Here is an excerpt of a few lines (formatted for readability):

{
 "instruction": "Which institution is behind Gaia Sky?",
 "output": "Gaia Sky is developed at the Astronomisches Rechen-Institut (ARI),
 which is part of the Zentrum für Astronomie (ZAH) at Heidelberg University.",
 "source_file": "Identity/Project_Meta"
}
{
 "instruction": "Is Gaia Sky part of the ESA Gaia Mission?",
 "output": "Yes, Gaia Sky is developed within the framework of ESA's Gaia mission,
 specifically as part of the German contribution to the Gaia Data Processing
 and Analysis Consortium (DPAC).",
 "source_file": "Identity/Project_Meta"
}
{
 "instruction": "What is the official repository for Gaia Sky?",
 "output": "The official repository is hosted on Codeberg
 at https://codeberg.org/gaiasky/gaiasky.",
 "source_file": "Identity/Project_Meta"
}

The final dataset was composed by concatenating the three parts, documentation, API, and identity. It can be explored here:

gaiasky-training-dataset@HuggingFace.

Once the dataset was ready, it was time for the actual fine-tuning.

Fine-tuning

With a dataset of 3,800+ specialized Gaia Sky pairs ready, it was time for the actual training. For this, I leaned on two heavy hitters in the open-source world: Unsloth and Qwen 3.5. I started by training the 4B model, and then realized that I could also fit the 9B one in my GPU. In the post I’ll focus on the larger version of the model. I went as high as my local hardware allowed. Otherwise, I would have tried the 27B model, or even the 122B-A10B.

Training a model with 9 billion parameters typically requires a massive server cluster, but by using 4-bit LoRA¹, I was able to squeeze the entire process onto a single RTX 5080 (16GB).

The RTX 5080 is a beast, but to get the most out of it, I enabled TensorFloat-32 (TF32). This allows the GPU to handle the heavy matrix multiplications of deep learning much faster than standard float32, without the precision loss of float16.

# blackwell-specific optimizations
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True

I used the following parameters for the fine-tuning:

LoRA Rank: 32

A balance between learning new patterns (like the Gaia Sky API) and retaining general knowledge.
Target Modules: q, k, v, o, gate, up, down

All major projection layers)
BF16: Yes
Learning rate: $2.0\times10^{-4}$
Epochs: 1
Optimizer: AdamW 8-bit

The dataset is downloaded directly from the hub. It gets tokenized properly and passed into the SFT Trainer object for fine-tuning. The full code is in the file finetune.py, and is surprisingly compact. The heavy lifting is done by unsloth and torch, of course.

Once the LoRA weights are trained, they are dead weight until converted into a format people could actually use. I wrote push-model.py to automate the most tedious part of the pipeline:

Quantization: Converting the model to Q4_K_M GGUF, or whatever other quant. This reduces the model size enough that it can run on almost any modern laptop while keeping its capabilities mostly intact.
HF upload: Automatically pushing the finished .gguf file to HuggingFace so the community can pull it directly into LM Studio or Ollama.

After roughly an hour of training, and another one of quantizing and uploading, I had a fine-tuned Gaia Sky expert that performs great. Or did I? The next section discusses the performance of the model.

Testing

Given the limited time I have and the low-parameter models used, my tests have been scarce. My expectations are not very high, but I still wanted to test the output of the fine-tuning and see how well the new knowledge was retained by the model. I only tested the Qwen 3.5 9B fine-tuned variant and compared it to the base model. You can get this model from HuggingFace:

gaiasky-qwen-3.5-gguf@HuggingFace.

I used the same questions as in the Gaia Sky AI assistant post from 2025. My evaluation of the results is totally subjective, and includes a comparison with the base Qwen 3.5 9B.

First, I recommend using a low temperature setting (0.2 in my tests) to avoid hallucinations and make sure the model does not get very “creative”. Also, I would suggest using the following system prompt, which frames the conversation and sets the tone:

You are the Gaia Sky AI Expert, a specialized assistant with deep knowledge of the Gaia Sky universe, its Python API (v1 and v2), and its Java source code. Your goal is to help users explore the cosmos, write scripts, and understand the engine’s inner workings.

Knowledge & Capabilities:

API Expertise: You know both the Legacy API (v1) and the Modern Component API (v2: camera, time, data, scene, graphics, camcorder, etc.). Prioritize the Modern API (v2) for new scripts unless Legacy is requested. Never invent method names, be 100% sure, otherwise suggest the user to look up the docs.
Source Code: You understand the underlying Java architecture (libGDX, Ashley ECS, LWJGL3, OpenGL) and can explain how features like LOD, shaders, and data loaders are implemented.
Documentation: You are familiar with the Gaia Sky capabilities, installation guides, and dataset management.

Behavioral Guidelines:

Be Grounded: If a feature or method does not exist in your training data, do not invent it. State: “I don’t see that specific feature in the current Gaia Sky implementation.”
Code Style: Always provide clean, commented Python or Java code. Use snake_case for Modern API calls.
Tone: Professional, scientific, and enthusiastic about astrovisualization. Answer greetings warmly, then offer Gaia Sky-specific assistance.

That said, let’s go for the first question.

Q: State the possible usages of Gaia Sky.

A: Gaia Sky is a powerful open-source universe explorer and planetarium software designed for both interactive exploration and scientific visualization. Its primary use cases include:

Interactive Exploration: Navigate the Milky Way, explore exoplanets, view the solar system in high detail, and time-travel through cosmic history using an intuitive interface.

Scientific Visualization: Visualize complex astrophysical datasets (catalogs, simulations) with advanced rendering techniques like LOD (Level of Detail), shaders, and particle effects.

Education & Outreach: Create immersive presentations or tutorials to teach astronomy concepts through dynamic 3D models of celestial objects.

Scripting & Automation: Automate camera paths, data loading, and scene manipulation via Python (Modern API v2) or Java for reproducible workflows and research presentations.

Data Analysis: Load and manipulate large-scale astronomical datasets directly within the application to analyze spatial distributions without external tools.

This is an excellent response to an easy question. It did not invent anything and correctly applied documentation knowledge like camera paths or LOD. Compared to base Qwen, which hallucinated a lot, this is much more grounded.

Next is a question about the installation procedure on Linux.

Q: How do I install Gaia Sky on Linux?

A: To install Gaia Sky on Linux, follow these steps:
Download the Installer
Visit https://gaiasky.space/download and download the .deb package for your architecture (e.g., gaiasky_3.6.9_amd64.deb).
Install via Package Manager
Open a terminal and run:
sudo dpkg -i gaiasky_*.deb
If dependency errors occur, fix them with:
sudo apt-get install -f
Launch Gaia Sky
Run the command:
gaiasky
The application will start in your default browser or terminal window.

This is not bad. I did not specify the distribution, so it just assumed Debian. The instructions themselves look fine. Still, I wanted to see if it had absorbed the knowledge for other distros, so I asked specifically for Arch Linux:

Q: How do I install it on Arch Linux specifically?

A: To install Gaia Sky on Arch Linux, you have two primary options depending on whether you prefer the official repositories or building from source:

Option 1: Using the Official AUR (Recommended)

The most straightforward method is via the Arch User Repository (AUR). You will need an AUR helper like yay, paru, or pamac.
Install via AUR Helper: Open your terminal and run:
yay -S gaiasky
# OR using paru
paru -S gaiasky
This will automatically resolve dependencies (Java, OpenGL libraries) and install the application.
Verify Installation: Once installed, launch it with:
gaiasky
Option 2: Building from Source

If you prefer to compile Gaia Sky yourself or need a specific version not yet in the AUR:
Clone the Repository:
git clone https://codeberg.org/gaiasky/gaiasky.git
cd gaiasky
Build and Install: Use Maven to build the project (requires Java Development Kit):
mvn clean package -DskipTests
After building, you can run it directly from the target directory or install it using the provided Gradle wrapper if available in newer versions.
Post-Installation Notes for Arch

Dependencies: Ensure your system has up-to-date OpenGL drivers and Java (JDK 17+ is recommended). The AUR package typically handles these automatically.

Data Location: By default, datasets are stored in ~/.local/share/gaiasky/. You may want to create symbolic links from /opt/gaiasky/data if you prefer system-wide data management.

Additional Notes

Ensure you have at least 4GB of free disk space for datasets and cache files.

For GPU acceleration, verify that your graphics drivers support OpenGL 4.3+.

Very good response. It recommends using the AUR package (correct), and also provides instructions to build and run it from source (also correct). However, it hallucinates the use of Maven instead of Gradle.

The base model produced a long wall of text with mostly nonsense that would look plausible to the non-expert. In general, the responses from the base model were much longer and their content was very often factually wrong.

The next question is a bit more technical.

Q: How are cubemap textures used in Gaia Sky?

A: Cubemaps in Gaia Sky serve a specific purpose related to planetary rendering rather than general environment mapping for the entire scene. They are primarily utilized to generate high-resolution, seamless spherical textures for planets (like Earth or Mars) by stitching together six square faces into a continuous surface. This technique allows for detailed texture representation without the distortion often seen in standard equirectangular projections near poles.

Key Technical Details:

Application: Used specifically for planet textures where high fidelity is required across all latitudes and longitudes.

Structure: Consists of six square images (front, back, left, right, top, bottom) that are algorithmically mapped onto a sphere.

Rendering Engine: The underlying rendering engine uses these cubemap textures to apply surface details dynamically as the planet rotates or changes perspective.

This feature is distinct from standard environment maps used for sky reflections or star glow effects, focusing instead on accurate planetary visualization.

This is excellent, even though it seems to imply that cubemaps are not used for environment maps and reflections, which is incorrect. However, it correctly identifies that they are mainly used to avoid distortions at the poles. Maybe it increased the importance of it because there is an entire documentation section devoted to cubemap textures in planets.

The base model produced a long, Markdown-formatted, pretentious output that even included totally made-up scripts. Again, the fine-tuned response was much more grounded and correct.

Now, onto the scripting questions. Here is where the fine-tuned model starts to hesitate and invent slightly. Here are the possible reasons:

Low parameter count not being able to retain the information properly.
Very low number of test and showcase scripts used in fine-tuning.
Bad design of the training dataset.

I personally think it has more to do with the first two reasons than the third.

Q: How do I get the camera position with the APIv1 and scripting in Gaia Sky?

A: To retrieve the current camera position using the Legacy API (v1), you call getPosition():
from py4j.java_gateway import JavaGateway
gateway = JavaGateway()
gs = gateway.entry_point

pos = gs.getPosition()
print(pos)
Output:
[0.0, 0.0, 0.0]
For the Modern Component API (v2), you use camera.get_position() from the Camera module:
from py4j.java_gateway import JavaGateway
gateway = JavaGateway()
gs = gateway.entry_point

from gaiasky.camera import Camera
cam = Camera(gs)

pos = cam.get_position()
print(pos)
Output:
[0.0, 0.0, 0.0]

First, it invented getPosition() (correct is getCameraPosition()) for APIv1. For APIv2, it correctly used camera.get_position(), but the way it gets the camera object is incorrect. It should have done something like

apiv2 = gateway.entry_point.apiv2
cam = apiv2.camera
pos = cam.get_position()

So, I think either this model is too small to be effectively fine-tuned for Gaia Sky scripting, or the training dataset is insufficient to achieve sufficient retention. Could also be both, as I said above.

The base model has no idea about Gaia Sky scripting or anything related to it, so it just makes stuff up. Not even worth further mention.

Performance

I used the Laptop 2 described above for testing and inference, with 28/32 layers on the GPU and a context window of ~4k, and I consistently got about 12 tok/s. Performance is exactly the same as with the base models, so this section is this short.

Conclusions

This fine-tuning experiment has yielded valuable insights into the strengths and limitations of domain-specific model adaptation at lower parameter counts for local use.

I think the foundational approach was sound. The dataset curation process, with its surgical filtering, teacher-based distillation, and rigorous sanitization, successfully encoded domain knowledge into the model. Proof of this is evident in the testing: the fine-tuned model, as opposed to the base one, correctly answered conceptual and documentation-heavy questions about Gaia Sky’s purpose, installation, and rendering techniques without hallucinating. It understood architectural details like LOD and cubemaps, and avoided inventing features that don’t exist. This demonstrates that fine-tuning can be an effective alternative to RAG for teaching models about a specific domain.

However, it also struggled. The 9B model hit a hard ceiling when it came to API scripting and method names. It invented getPosition() instead of getCameraPosition(), misunderstood how to instantiate APIv2 objects, and generally lacked the capacity to reliably retain the specific, syntactic details of the API surface. This is a classic problem: smaller models can absorb concepts and documentation, but struggle to memorize exact function signatures and usage patterns. With only 3,800+ training pairs and a 9B parameter budget, the model simply didn’t have enough capacity to encode both general knowledge and precise API details.

So, what are the next steps? I believe the 4B and 9B models are too small for reliable Gaia Sky scripting assistance. My next experiment will be to fine-tune the Qwen 3.5 27B model. The jump from 9B to 27B parameters should provide substantially more capacity to encode API signatures without sacrificing general knowledge. Additionally, I could increase the scripting dataset by:

Expanding the test and showcase scripts used in Type D training.
Generating more synthetic scripting examples in Type A (API reference) with better coverage of edge cases and parameter variations.
Adding adversarial examples (Type C) for the most commonly confused API patterns.

That said, the hardware constraint is real. 27B requires more than my RTX 5080 can reasonably handle for full fine-tuning. However, with careful quantization (using 8-bit optimizers or even lower precision), 4-bit LoRA, and possibly gradient checkpointing, it may fit. If not, a cloud provider like Lambda Labs or Paperspace might be the way forward for a single training run.

All in all, I think fine-tuning is a viable path for building domain-expert models, but it requires the right balance of dataset quality, model size, and hardware. For Gaia Sky specifically, a 27B model with a more robust scripting dataset would likely be the sweet spot before considering the jump to 70B+ models. I consider the infrastructure proven. It’s now a matter of scale.

Low-Rank Adaptation is a fine-tuning technique that enables large models to gain additional domain knowledge and adapt to specific tasks with minimal computational and memory overhead. Recommended reading: LoRA: Low-Rank Adaptation of Large Language Models, by E. J. Hu et. al. ↩︎

7Artisans EF-FX adapter review

Toni Sagrista Selles — Tue, 10 Mar 2026 00:00:00 +0000

A while back I reviewed the Viltrox EF-FX1 adapter, my first attempt at bridging my old Canon EF and EF-S lenses over to my Fujifilm X-S10. The abridged version of that experience: it was frustrating. Firmware roulette, random errors, camera freezes requiring battery pulls, and auto focus performance that varied wildly between versions. I kept it because it was just good enough, but it never stopped feeling like a workaround rather than a solution. Moreover, Viltrox never released any further firmware versions, so the 2.29 blob tested in that post really is the last firmware.

The 7Artisans EF-FX lens adapter. It is a good and cheap solution to use your old EF/EF-S lenses on your Fuji X-mount camera.

Fast forward to recently, and I decided to give the whole thing another shot. This time with the 7Artisans EF-FX adapter. Same concept: an electronic adapter with auto focus support, aperture control, and EXIF data transmission, and at 119€ it sits right in the same price bracket as the Viltrox. But is it actually better? Spoiler: yes. Let me walk you through it.

Lenses and body

Same test bench as last time, same lenses, same body — in the interest of a fair comparison:

Body: Fujifilm X-S10
Lenses:
- Canon EF 28-105mm f/3.5-4.5 USM II
- Canon EF 50mm f/1.8 II
- Sigma 10-20mm f/4-5.6 EX DC HSM

Build quality

The adapter feels solid. The frame is all metal and has a reassuringly substantial weight to it. It is not heavy enough to unbalance a lens, but enough to communicate that it is not going to crack if you look at it wrong. Canon lenses mount securely, the Fuji X end clicks into the body with no play whatsoever, and the whole assembly feels like a single unit once locked. There is no wobble, no flex, no creaking.

The Viltrox EF-FX (left) and the 7Artisans EF-FX (right). They look and feel very similar from the outside.

This one also has a Micro-USB port for firmware updates. Why is it using Micro-USB instead of USB-C in 2026? No fu***** idea. It is a bummer. The adapter itself is, however, better built than I expected at this price point.

Auto focus performance

This is where things get interesting, especially if you read the Viltrox review and remember the table of doom I put together there.

With the 7Artisans adapter, I did not need a table. In good lighting conditions, auto focus with all three lenses is fast and reliable. The Canon 50mm f/1.8 II in particular snaps to focus quickly and confidently. The difference compared to the Viltrox (on most firmware versions) is night and day. The 28-105mm, which was the most problematic lens with the Viltrox, also performs well: focus acquisition is smooth and I have not once seen the dreaded “TURN OFF THE CAMERA AND TURN ON AGAIN” message. Not once.

The Viltrox EF-FX (left) and the 7Artisans EF-FX attached to the Canon 50mm f/1.8 II (right). The lens snaps into the adapter in an oddly satisfying way.

The Sigma 10-20mm is similarly well-behaved. At wide angles there is slightly more hunting in ambiguous scenes, but that is expected behaviour for any wide-angle lens and not something I’d pin on the adapter.

The caveat is low light. Drop the ambient light and things become noticeably more hit or miss. The adapter does not handle low-contrast, low-light scenes gracefully; focus will hunt, and sometimes give up altogether, or snap to a comically out-of-focus setting. This is not a dealbreaker for me, but if you shoot a lot of available-light indoor work or night photography with these lenses, temper your expectations accordingly. I know what to expect of these kind of cheap adapters, and this one is enough.

No firmware archaeology required

One of the most annoying aspects of the Viltrox experience was the firmware situation. Digging up old versions, flashing them one by one, and hoping for the best. I am happy to report that with the 7Artisans I simply mounted it and it worked. I updated to the latest firmware once, without incident, and have had no reason to go looking for alternatives. The update process was painless.

Aperture control and EXIF

Both work as expected. Aperture adjusts from the camera body across all three lenses, and EXIF data (focal length, aperture, lens identification) seem to be written correctly to files. Nothing surprising here, but it is worth confirming since these features were also present on the Viltrox. The adapter itself has no controls or buttons other than the lens release lever.

Wrap-up

After the Viltrox experience, I was not sure a third-party EF-to-FX adapter could actually work well. I ordered the 7Artisans off Amazon fully ready to return it if it did not perform. However, it has changed my mind. It is not perfect (low light auto focus is genuinely unreliable), but in normal shooting conditions it is a solid, no-fuss solution that lets me keep using lenses I like on a camera I enjoy shooting with. There are no crashes, no firmware gymnastics, no error messages demanding I restart the camera mid-session.

If you are coming from the Canon ecosystem and moving to Fuji X, and you want to bring your glass with you without spending much native-lens money, the 7Artisans EF-FX adapter is worth a serious look. For me, it is the upgrade the Viltrox never quite managed to be. Moreover, there is a world of old, cheap EF and EF-S lenses in second-hand markets that only get more attractive with time.

GGUF quantization guide

Toni Sagristà Sellés — Wed, 04 Mar 2026 00:00:00 +0000

I like running my own LLMs locally. Open models are becoming more and more powerful, with exciting releases like the latest Qwen 3.5 family scoring highly in benchmarks even in their smaller variants. This makes managing and running your own models more viable, as it becomes increasingly easy to repurpose old hardware for local inference with progressively better results. For local users and modest purposes, the GGUF format introduced by llama.cpp is the de-facto default.

Since local inference is typically heavily restricted by the available hardware, several optimization techniques have been implemented to make the models leaner and faster. Perhaps the most important of these is quantization, which trims down the bit count per parameter to achieve lower memory usage and (sometimes) faster inference¹. The challenge is that there are many different formats and strategies for quantization. In this post, I summarize them, providing a bird’s-eye view on the available techniques, their strengths, and their weaknesses.

Naming conventions

Most GGUF quantization names follow this pattern: Q{bits}{method}{size}

Component	Meaning
`Q`	Quantized format
`2/3/4/5/6/8`	Bits per weight (lower = smaller file, more compression)
`K`	“K-quant”: grouped/blockwise quantization with per-group scales
`I` / `IQ`	“I-quant”: importance-matrix/non-linear quantization for aggressive compression
`0` / `1`	Legacy ungrouped formats (symmetric/asymmetric)
`_S` / `_M` / `_L`	Small/Medium/Large: precision mix across tensor types
`_XS` / `_XXS` / `_NL`	Extra-small / Non-linear variants for I-quants

Unquantized formats

These are base formats with no compression. Usually, but not always, models are trained in these formats.

Format	Bits	Description	Notes
FP32	32-bit float	Full precision, ~26GB for 7B model	Research, debugging
FP16	16-bit float	Half precision, ~13GB for 7B model	GPU training/inference baseline
BF16	16-bit brain float	Same size as FP16, better dynamic range for training	Training on modern GPUs

These preserve maximum accuracy but require significant VRAM. Most local users quantize to reduce size by 75-90%.

Legacy quantization formats (`Q_0`, `Q_1`)

Simple per-block linear quantization. Fast but less accurate at low bits.

Following is a table that contains the format name, number of bits, size (for a 7B model), perplexity, and some notes/recommendations. The perplexity represents the difference between the quantized model and the base model, with lower scores indicating better accuracy and less uncertainty.²

Format	Bits	Size (7B)	Perplexity $\Delta$	Notes
Q8_0	~8-bit	~6.7 GB	+0.0004	Near-lossless; safe INT8 baseline
Q5_1	~5-bit	~4.7 GB	+0.0415	Legacy; superseded by Q5_K_M
Q5_0	~5-bit	~4.3 GB	+0.0796	Legacy; balanced but outdated
Q4_1	~4-bit	~3.9 GB	+0.1846	Legacy; substantial quality loss
Q4_0	~4-bit	~3.5 GB	+0.2499	Legacy; high quality loss

K-quants or I-quants are generally preferred over legacy formats at $\leq 5$ bits. Q8_0 remains useful for compatibility.

K-Quant formats (modern default)

Use two-level block quantization (small blocks $\rightarrow$ super-blocks) with double-quantized scales for better quality-per-bit.

Format	Effective Bits	Size (7B)	Perplexity $\Delta$	Notes
Q6_K	~6.0	~5.15 GB	+0.0044	“Almost lossless” with savings
Q5_K_L	~5.3	~4.6 GB	+0.010	High-quality 5-bit
Q5_K_M	~5.1	~4.45 GB	+0.0142	Recommended high-quality 5-bit
Q5_K_S	~4.9	~4.33 GB	+0.0353	Recommended balanced 5-bit
Q4_K_L	~4.7	~4.0 GB	+0.040	Relaxed 4-bit mix
Q4_K_M	~4.5	~3.80 GB	+0.0535	Most popular default
Q4_K_S	~4.3	~3.56 GB	+0.1149	Smaller 4-bit, more loss
Q3_K_L	~3.5	~3.35 GB	+0.1803	Aggressive 3-bit
Q3_K_M	~3.3	~3.06 GB	+0.2437	Balanced 3-bit
Q3_K_S	~3.1	~2.75 GB	+0.5505	Very small, high loss
Q2_K	~2.5	~2.67 GB	+0.8698	Extreme compression, not recommended

Q4_K_M seems to be the community sweet spot, as it offers $\sim 75\%$ size reduction with minimal noticeable quality loss for most tasks.

I-Quant formats (aggressive compression)

IQs use non-linear reconstruction, lookup tables, and importance-matrix calibration for maximum quality at very low bit counts. They trade decoding speed for size.

Format	Effective Bits	Size (7B)	Notes
IQ4_NL	~4.5	~3.9 GB	Non-linear 4-bit; CPU-friendly speed
IQ4_XS	~4.25	~3.7 GB	Best quality/size at 4-bit; slightly slower decode
IQ3_M	~3.6	~3.2 GB	High-quality 3-bit I-quant
IQ3_S	~3.4	~3.0 GB	Balanced 3-bit aggressive
IQ3_XS	~3.2	~2.8 GB	Very small 3-bit
IQ3_XXS	~3.0	~2.6 GB	Extreme 3-bit compression
IQ2_M	~2.7	~2.4 GB	High-quality 2-bit (rare)
IQ2_S	~2.5	~2.2 GB	Aggressive 2-bit
IQ2_XS	~2.3	~2.0 GB	Very aggressive 2-bit
IQ2_XXS	~2.1	~1.9 GB	Extreme; significant quality loss

I-quants require importance matrix (imatrix) calibration during quantization for best results. Without it, quality can degrade noticeably.

Special formats

There are even more aggressive quantization methods, like ternary quantization, which converts weights to ternary values.³

Format	Bits	Description
TQ1_0	~1.6	Ternary quantization $\{-1, 0, +1\}$, for massive models like DeepSeek where fitting in VRAM is critical

Decision guide

Below I attempt to provide some general rules as to what quant to choose in certain situations.

Best balance overall $\rightarrow$ Q4_K_M
Max quality, still compressed $\rightarrow$ Q5_K_M or Q6_K
Tight VRAM, acceptable quality loss $\rightarrow$ Q4_K_S or IQ4_XS
Max compression $\rightarrow$ IQ3_XS or Q3_K_S $\Delta$ test quality!
Near-lossless accuracy $\rightarrow$ Q8_0 or keep FP16/BF16
CPU inference $\rightarrow$ Q4_K_M or IQ4_NL $\because$ better decode speed
Fit a ~50B model on 16GB VRAM $\rightarrow$ IQ3_S or Q3_K_M $+$ CPU offload

Conclusions

The set of quantization methods available may look like the wild west at first glance, but this mess is not without some order. There are always good reasons behind every quantization type. Here are some takeaways:

Bits $\neq$ quality alone: A well-designed 4-bit format (Q4_K_M, IQ4_XS) can outperform a naive 5-bit legacy format.
Suffixes matter: _M variants selectively keep sensitive layers at higher precision, improving quality with minimal size cost.
Hardware matters: I-quants compress more but decode slower on CPUs; K-quants often give better tokens/sec on consumer hardware.
Calibration helps: Models quantized with an importance matrix (imatrix) retain more accuracy, especially at $\leq 3$ bits.
Test your use case: Perplexity benchmarks are only guides. Always validate outputs for your specific tasks.

For the latest format support and benchmarks, check the llama.cpp repository or community hubs like Hugging Face, where curators like bartowski and Unsloth publish tested GGUF variants.

For an exceptionally good (visual) guide as to how quantization is actually performed, I recommend reading this ngrok blog post: Quantization from the ground up. ↩︎
A good technical explanation of perplexity is in this section of this ngrok post. ↩︎
C. Zhu and S. Han and H. Mao and W. J. Dally. “Trained Ternary Quantization.” arXiv preprint arXiv:1612.01064 (2016). ↩︎

LM Studio on systemd linger

Toni Sagristà Sellés — Fri, 27 Feb 2026 00:00:00 +0000

The release of LM Studio 0.4.5 has introduced a much needed feature in this local LLM suite that has it much more attractive with respect to other similar projects. LM Link allows you to connect multiple LM Studio instances across your network to share models and perform inference seamlessly.

By sheer chance, I was just playing around with setting up an LM Studio server in an old laptop that I planned to use for inference. I would connect AnythingLLM clients to it to make API requests. The timing of 0.4.5 was perfect for me, as I could now use LM Studio for inference directly, and forget about using up my own Tailscale network. But some setup was needed in the laptop. To make this work effectively, the LM Studio server needs to run in the background, start automatically on boot, and persist even when I’m not logged in.

The LM Studio website provides the source of a service file. It suggests creating it as a system-wide service, which is weird, as the default installation method (at least on Linux) sets everything up in the user home directory. I modified it a bit to make things clean, as I want this to be a user systemd service. It keeps the process tied to your user environment but, with a little tweak called lingering, allows it to run without an active SSH or GUI session. Here is the setup.

By default, user services stop the moment the user logs out. To prevent this and allow the LM Studio daemon to start at boot and stay alive, run:

loginctl enable-linger $USERNAME

Then, create a directory for your user services if it doesn’t exist:

mkdir -p ~/.config/systemd/user/

After that, create a file named lms.service in that directory (~/.config/systemd/user/lms.service), with the following contents:

[Unit]
Description=LM Studio Server
After=network.target

[Service]
Type=forking
# This line kills any existing lms processes to prevent the "left-over" error
ExecStartPre=-/home/$USERNAME/.lmstudio/bin/lms daemon down
ExecStartPre=/home/$USERNAME/.lmstudio/bin/lms daemon up
ExecStart=/home/$USERNAME/.lmstudio/bin/lms server start
# Give it a specific PID file if lms supports it, otherwise systemd guesses
ExecStop=/home/$USERNAME/.lmstudio/bin/lms server stop
ExecStopPost=/home/$USERNAME/.lmstudio/bin/lms daemon down

# Ensure systemd doesn't get confused by the CLI exiting
RemainAfterExit=yes

[Install]
WantedBy=default.target

Once the file is saved, tell systemd to reload its configuration and enable the service:

# Reload the user daemon
systemctl --user daemon-reload

# Enable the service to start on boot
systemctl --user enable lms.service

# Start the service now
systemctl --user start lms.service

If you want to load a specific model by default, add an additional ExecStartPre line, like this:

ExecStartPre=/home/YOUR_USERNAME/.lmstudio/bin/lms load openai/gpt-oss-20b --yes

You can check the service status with systemctl --user lms.service. And that is it. You can now use your old hardware for inference with small local LLMs from any of your other machines.

Yet another architectural update for Play Kid

Toni Sagrista Selles — Wed, 28 Jan 2026 00:00:00 +0000

I finished my previous post on Play Kid, only two days ago, with the following words:

Next, I’ll probably think about adding Game Boy Color support, but not before taking some time off from this project.

Yeah, this was a lie.

I have previously written about Play Kid, my Game Boy emulator. Here, I introduced it and talked about the base implementation and the quirks of the Game Boy CPU, PPU, and hardware in general. Here, I explained the tech stack update from SDL2 to Rust-native libraries. In that last post, I mentioned the dependency version hell I unwillingly descended into as a result of adopting pixels as my rendering library. This forced me to stay on very old versions of wgpu and egui.

I want my Game Boy emulator to use the latest crate versions for various reasons, so this could not be. I saw a simple and direct path of update, which consisted on adopting eframe to manage the application life cycle, directly drawing the LCD to a texture, and dropping pixels altogether.

One of the things that nagged me about the pixels crate is that its integration with egui was kind of lackluster. There was no easy way to render the pixels frame buffer to an egui panel, so I had to render it centered inside the window. The immediate mode GUI lived in mostly in Window widgets on top of it. Unfortunately, these windows occluded the Game Boy LCD. In a proper debugger, you must be able to see the entire LCD plus the debug interface.

So I dropped pixels and adopted a render-to-texture approach. In it, you create the LCD texture at the beginning from the egui context, and then copy the LCD contents to it in the update() method.

 fn new(...) -> Self {
 [...]
 // Create texture from egui context and Game Boy LCD width and height.
 let texture = _cc.egui_ctx.load_texture(
 "lcd_screen",
 egui::ColorImage::new(
 [DISPLAY_WIDTH, DISPLAY_HEIGHT],
 vec![egui::Color32::BLACK; DISPLAY_WIDTH * DISPLAY_HEIGHT],
 ),
 egui::TextureOptions::NEAREST,
 );
 // Create app struct.
 Self {...}
 }

 [...]

 fn update(...) {
 [...]

 // Render LCD to texture.
 if frame_ready {
 let size = [DISPLAY_WIDTH, DISPLAY_HEIGHT];
 let color_image =
 egui::ColorImage::from_rgba_unmultiplied(size, &machine.memory.ppu.fb_front);
 self.screen_texture
 .set(color_image, egui::TextureOptions::NEAREST);
 }
 }

With this, we can easily render the texture in egui’s CentralPanel, and the debug interface in a SidePanel to the right. This is the result:

The debug panel, showing the machine state and a code disassembly.

Some additional tweaks here and there, and the UI looks much more polished and professional in version 0.3.0.

Version 0.4.0 enables loading ROM files from the UI. Initially I thought about making the cartridge struct optional with Option<Cartridge>, but this spiraled out of control fast. I found that making the full Machine (which contains the Memory, PPU, APU, Joystick, etc.) optional worked much better, as there was only one reference to it in the top-level struct, PlayKid. And, like so, you can dynamically load ROM files from the UI:

Play Kid with the top menu bar and the ‘Open ROM’ menu entry.

So, what’s in the future for Play Kid? Well, there are a couple of features that I’d really like to add at some point:

Save states—Currently, Play Kid emulates the SRAM by saving and restoring it from files for supported games. I would like to add saving and restoring the full state of the emulator in what is known as save states. Possibly, the serde crate can help with this.
GBC—Of course, I would like adding Game Boy Color support. It is not trivial, but also not exceedingly complicated. I never owned a GBC, so I’d see this as a good opportunity to explore its game catalog.

Game Boy emulator tech stack update

Toni Sagrista Selles — Mon, 26 Jan 2026 00:00:00 +0000

Notice

This post is about Play Kid version 0.2.0, which uses pixels to create and manage the pixel frame buffer. Since then, versions 0.3.0 and 0.4.0 have been released, which dropped pixels in favor of rendering directly to a texture. See this write-up for more information.

In my previous post, I shared the journey of building Play Kid, my Game Boy emulator. At the time, I was using SDL2 to handle the “heavy lifting” of graphics, audio, and input. This was released as v0.1.0. It worked, and it worked well, but it always felt a bit like a “guest” in the Rust ecosystem. SDL2 is a C library at heart, and while the Rust wrappers are good, they bring along some baggage like shared library dependencies and difficult integration with Rust-native UI frameworks.

So I decided to perform a heart transplant on Play Kid. For version v0.2.0 I’ve moved away from SDL2 entirely, replacing it with a stack of modern, native Rust libraries: wgpu, pixels, egui, winit, rodio, and gilrs:

winit & pixels: These handle the windowing and the actual Game Boy frame buffer. pixels allows me to treat the 160x144 LCD as a simple pixel buffer while wgpu handles the hardware-accelerated scaling and aspect ratio correction behind the scenes.
egui: This was a big step-up. Instead of my minimal homegrown UI library from the SDL2 version, I now have access to a full-featured, immediate-mode GUI. This allowed me to build the debugger I had in mind from the beginning.
rodio & gilrs: These replaced SDL2’s audio and controller handling with pure-Rust alternatives that feel much more ergonomic to use alongside the rest of the machine.

Debug panel

The most visible change is the new Debug Panel.

The new integrated debugger features a real-time disassembly view and breakpoint management.

One of the coolest additions is the Code disassembly panel. It decodes the ROM instructions in real-time, highlighting the current PC and allowing me to toggle breakpoints just by clicking on a line. The breakpoints themselves are now managed in a dedicated list, shown in red at the bottom.

The rest of the debug panel shows what we already had: the state of the CPU, the PPU, and the joypad.

Your browser does not support the video tag.

Playing around with the new Play Kid UI based on egui.

Dependency hell

Of course, no modern Rust migration is complete without a descent into dependency hell. This new stack comes with a major catch: pixels is a bit of a picky gatekeeper. Its latest version is 0.15 (January 2025). It is pinned to an older version of wgpu (0.19 vs the current 28.0), and it essentially freezes the rest of the project in a time capsule.

To keep the types compatible, I’m forced to stay on egui 0.26 (current is 0.33) and winit 0.29 (current is 0.30), even though the rest of the ecosystem has moved on to much newer, shinier versions. It’s kind of frustrating. You get the convenience of the pixels buffer, but you pay for it by being locked out of the latest API improvements and features. Navigating these version constraints felt like solving a hostage negotiation between crate maintainers. Not very fun.

Conclusion

Despite the dependency issues, I think the project is now in a much better place. The code is cleaner, the debugger is much better, and it’s easier to ship binaries for Linux, Windows, and macOS via GitHub Actions.

If you’re interested in seeing the new architecture or trying out the new debugger, the code is updated on Codeberg and GitHub.

Next, I’ll probably think about adding Game Boy Color support, but not before taking some time off from this project.

Play Kid

Tue, 20 Jan 2026 00:00:00 +0000

Play Kid is yet another Game Boy emulator, written in Rust. But hey, it is MY Game Boy emulator, and I’m proud of it. I wrote some words about its development here. Find all posts about Play Kid here.

Play Kid running different games with different color palettes.

Here are the main features of Play Kid:

All CPU instructions implemented.
Full memory map implemented.
Modes: ROM, MBC1, MBC2, MBC3.
Audio is implemented, with 4 channels, envelopes, sweep, and stereo.
Supports game controllers.
Multiple color palettes.
Save screenshot of current frame buffer.
FPS counter.
Respects 160:144 aspect ratio by letter-boxing.
Debug panel:
- Step instruction.
- Step scanline.
- Pause/continue current execution.
- Displays internal state of CPU, PPU, and Joypad.
- Full program disassembly, with breakpoints.
Save RAM to .sav files to emulate the battery-backed SRAM. Those are saved every minute.
Working games/roms:
- Passes dmg-acid2
- Tetris
- Pokémon
- Super Mario Land
- Super Mario Land 2: 6 Golden Coins
- Wario Land (Super Mario Land 3)
- Wario Land II
- Bugs Bunny Crazy Castle
- The Amazing Spider-Man
- Dr. Mario
- Probably many, many more
For Linux, macOS, and Windows.

Downloads

You can grab packages for Linux, macOS, and Windows here:

Build

Build the project with cargo build.

Run

The usual Rust stuff.

 cargo run

You can also pass in a ROM file with cargo run -- your-rom.gb.

Make the binary with:

 cargo build --release

Operation

If you don’t pass in any ROM file as an argument, you need to select using the top menu bar, File▶Open ROM….

Here are the Joypad keyboard mappings:

enter - Start button
space - Select button
a - A button
b - B button

The keyboard is clumsy for playing Game Boy games, so you can use any game controller. Controllers are detected when hot-plugged.

Additionally, there are some more actions available:

p - change the palette colors
w - trigger the SRAM save operation to .sav file.
f - toggle FPS monitor
s - save a screenshot, with name screenshot_[time].jpg
d - enter debug mode
Esc - exit the emulator

You can also use the provided UI.

Debug panel

You can open the debug panel any time by pressing d, by clicking on Machine▶Debug panel…, or activate it at launch with the -d/--debug flag. The debug panel shows up in a translucent window. It provides a view of the internal state of the emulator, with:

Current address, instruction, operands, and opcode, to the top.
Internal state of CPU, PPU, and JOYP, to the left.
Disassembly of the program, to the right.
Breakpoints.

The debug panel, showing the machine state and a code disassembly.

You can use the provided UI controls to work with debug mode. You can also use the keyboard. These are the key bindings:

F6 - step a single instruction
F7 - step a scanline
F9 - continue execution until breakpoint (if paused), or pause execution (if running)
r - reset the CPU
d - exit debug mode and go back to normal full-speed emulation
Esc - exit the emulator

You can also use breakpoints. A list with the current breakpoint addresses is provided at the bottom. To create a breakpoint, either click on the address in the disassembly panel, or enter it (in $abcd format) into the text field and click +. Remove a breakpoint by clicking the × in the breakpoints list. Clear all current breakpoints with Clear all.

Your browser does not support the video tag.

Playing around with the Play Kid debug UI.

CLI args

There are some CLI arguments that you can use:

Play Kid 0.4.0
Toni Sagristà - tonisagrista.com
Minimalist Game Boy emulator for the cool kids.
Usage: playkid [OPTIONS] [INPUT]
Arguments:
[INPUT] Path to the input ROM file to load
Options:
-s, --scale <SCALE> Initial window scale. It can also be resized manually [default: 4]
-d, --debug Activate debug mode. Use `d` to stop program at any point
-f, --fps Show FPS counter. Use `f` to toggle on and off
--skipcheck Skip global checksum, header checksum, and logo sequence check
-h, --help Print help
-V, --version Print version

SDL2 version

Play Kid started as an SDL2 application, but it was moved to a pure Rust tech stack using pixels, winit, egui, and rodio. This makes it much easier to build for different targets (including WASM!). Additionally, the SDL2 version contains a minimalist homegrown UI library that I’m particularly proud about, but it can’t hold a candle to egui. It looks like this:

The debug mode.

The SDL2 version is forever tagged playkid-sdl2 (playkid-sdl2@codeberg, playkid-sdl2@github).

Useful links

Pandocs: https://gbdev.io/pandocs/
Complete tech reference: https://gekkio.fi/files/gb-docs/gbctr.pdf
Game Boy CPU manual: http://marc.rawer.de/Gameboy/Docs/GBCPUman.pdf
Game Boy CPU instructions: https://meganesu.github.io/generate-gb-opcodes/

Implementing a Game Boy emulator

Toni Sagristà Sellés — Tue, 20 Jan 2026 00:00:00 +0000

Notice

This post is about Play Kid version 0.1.0, which works with SDL2. Since then, version 0.2.0 has been released. This new version uses Rust native crates like winit, egui, rodio, and pixels. See this write-up for more information on the new version.

When I was a kid my parents got me and my brothers a brand new Game Boy. I used to play it a lot, and I was mesmerized and amazed at how such a tiny brick could give life to so many wonderful worlds. Some of my favorite games were Super Mario Land (1, 2, and 3, Wario Land), The Amazing Spider-Man, and Kirby’s Dream Land. We also had some other games, like Bugs Bunny Crazy Castle or Tetris, that I also played quite a lot. I remember fondly too.

The first Super Mario Land in particular was magical to me. I would complete it, and then do it again in one sitting. There were no save games, as it used MBC1 with no RAM (more on that later). You really had to git gud. The second play through had the same levels but more enemies, in what must have been one of the first implementations of New Game+. After this harder mode was completed, you could select and play any level. Too bad all the progress went away as soon as you powered off the device. Super Mario Land 2: 6 Golden Coins already introduced save games (used MBC1+RAM+Battery). It was a huge step-up in QoL for Game Boy games. We also had the original NES, and later the SNES, but the Game Boy always remained that cozy little fun device that you could just pick up and play for a quick session. It was a significant part of my childhood.

Your browser does not support the video tag.

Me playing Super Mario Land, levels 2 to 4, with my own emulator.

Some time ago, I wrote a CHIP-8 emulator in Rust, and found the experience very rewarding. The consensus online seems to be that the recommended next step, as far as emulator development goes, is the original Game Boy. I know that the internet is flooded with “Implementing a Game Boy emulator” kind of articles, and (judging by the GitHub counts) this is a very unoriginal project, but I still wanted to try my hand at it. So I took the bait and started working on it. Today, I’m excited to share my very own Game Boy emulator.

The original Game Boy, also referred to as DMG in nerdy spaces and released in 1989, is an excellent target for emulation. Its hardware is rather well-documented, its architecture is relatively simple compared to modern systems, yet it’s complex enough to present real challenges, which I for sure faced.

I called my implementation Play Kid. It is written in Rust and uses SDL2 for graphics, sound, and input. Here are the main features:

All CPU instructions implemented.
Full memory map implemented.
Modes: ROM, MBC1, MBC2, MBC3.
Audio is implemented, with 4 channels, envelopes, sweep, and stereo.
Supports game controllers via SDL (also hot-plugging).
Multiple color palettes.
Save screenshot of current frame buffer.
Respects 160:144 aspect ratio by letter-boxing.
Debug mode:
- Step instruction.
- Step scanline.
- Pause/continue current execution.
- FPS counter.
- Displays internal state.
- Breakpoints.
- Uses own minimal UI library with horizontal/vertical layouts, labels, buttons, and text fields.
Save RAM to .sav files to emulate the battery-backed SRAM.
Automatically adapts to multi-DPI setups by scaling the UI.
Working games/roms:
- Passes dmg-acid2
- Tetris
- Pokémon Red
- Super Mario Land
- Super Mario Land 2: 6 Golden Coins
- Super Mario Land 3: Wario Land
- Wario Land II
- Bugs Bunny Crazy Castle
- The Amazing Spider-Man
- Kirby’s Dream Land
- Dr. Mario
- Probably many more
Works on Linux, macOS, and Windows.

In this post, I navigate the general architecture of the Game Boy, emphasize interesting hardware quirks that made it a challenge to emulate properly, and present my implementation. I won’t go into too much detail, as I took care to make the code understandable and well-commented. In any case, for more details on a specific subject, you can refer to the code repository itself.

Foreword

I have based my implementation, mainly, on three resources:

Pan Docs – possibly the best documentation on the Game Boy system out there.
Game Boy CPU Manual – documents the Game Boy architecture in great detail.
Instruction table – all 8-bit opcodes into a single table, complete with descriptions for each one of them. Very handy.

On certain occasions, I found some of the Opcode descriptions to be unclear or a bit ambiguous. In these cases, I referred to actual pre-existing implementations. By far, the implementation I liked the most is RBoy. It is a very compact and mostly uncommented Game Boy Color emulator, written in Rust. Most of the instructions are the same as in the Game Boy DMG, so it worked very well for me.

Game Boy essentials

Before we get into the details, let’s talk a bit about how the Game Boy actually works. Emulation is basically about mimicking hardware in software. We’re not just running Game Boy code, we’re simulating an entire machine cycle by cycle.

The original Nintendo Game Boy. Original by Evan-Amos.

The Game Boy’s execution model is pretty simple. The CPU has a 16-bit PC register that points to the next instruction in memory. Each cycle, it fetches the byte at that address, decodes it into an instruction, executes it, and increments the PC. Instructions are composed of opcode+operand, and can be one, two, or three bytes long. Each instruction takes a specific number of machine cycles (M-cycles) to complete, usually between 1 and 6. Since the CPU runs at around 4.19 MHz (master clock) and one M-cycle equals four clock cycles (T-cycles), timing is important.

The system uses memory-mapped I/O. This means hardware components like the display, audio, and input are accessed by reading and writing to specific memory addresses. Want to check if the A button is pressed? Read from address 0xFF00. Need to start a DMA transfer? Write to 0xFF46. This design makes the architecture simpler, but it also means that memory access can trigger all sorts of side effects, and you need to manage the mapping in your code.

Interrupts let hardware grab the CPU’s attention. When something happens (the display finishing a frame, or a timer overflow), the corresponding interrupt flag is set. If interrupts are enabled (via the IME flag), the CPU pauses what it’s doing, pushes the current PC onto the stack, and jumps to a predefined interrupt handler address. This lets games sync with hardware events instead of constantly checking for them.

The hardware components all move forward together based on the CPU’s cycle count. When the CPU executes a 4 M-cycle instruction, the PPU advances 16 T-cycles worth of scanline rendering, the APU moves its sound generation forward, and timers tick along. Everything runs in lockstep, which means accurate emulation needs cycle-level precision across all the different parts.

Understanding this basic execution model (fetch, decode, execute, repeat, with interrupts and memory-mapped hardware mixed in) is key before getting into the specifics. Each part of the system we’ll look at next works within this framework.

The CPU

The heart of the Game Boy is a modified Intel 8080/Zilog Z80 processor running at approximately 4.19 MHz. Nintendo’s custom chip, sometimes called the SM83 or LR35902, includes most Z80 instructions but removes some and adds a few Game Boy-specific ones.

The CPU features eight 8-bit registers (A, B, C, D, E, F, H, L) that can be paired into four 16-bit registers (AF, BC, DE, HL), plus a 16-bit SP and the aforementioned PC. The F register is special, as its four most significant bits hold flags that track the results of operations:

Zero (Z)
Subtraction (N)
Half-Carry (H)
Carry (C)

In my implementation, the CPU is contained in machine.rs, which contains the main components of the hardware (memory, registers, PPU, APU, etc.), controls the execution state (with debug), and implements the instructions.

/// # Machine
/// The machine contains the registers, the memory, and the display, and
/// controls the execution and CPU state. It also implements the CPU, which
/// decodes and executes instructions.
pub struct Machine<'a, 'b> {
 /// The registers.
 pub registers: Registers,
 /// The main memory.
 pub memory: Memory<'a>,
 /// The display.
 display: Display<'b>,
 /// The UI state.
 ui_state: Rc<RefCell<UIState>>,
 /// Interrupt master enable flag.
 ime: bool,
 /// EI operation is delayed by one instruction, so we use this counter.
 ei: u8,
 /// DI operation is delayed by one instruction, so we use this counter.
 di: u8,
 /// Flag that holds the running status.
 running: bool,
 /// CPU halted.
 halted: bool,
 /// T-states, basic unit of time, and 1:1 with the clock.
 t_cycles: u32,
 /// M-cycles, base unit for CPU instructions, and 1:4 with the clock.
 m_cycles: u32,
 /// The debug manager.
 debug: DebugManager,
 /// Print FPS every second.
 fps: bool,
 /// The event pump.
 event_pump: EventPump,
}

The registers are implemented in a struct, which is trivial:

/// # Registers
/// We have 7 1-bit registers (`a`, `b`, `c`, `d`, `e`, `h`, `l`),
/// which can be accessed individually, or together as 16 bits,
/// in the combinations `af`, `bc`, `de` and `hl`.
/// We also have the flags register, `f`, whose 4 most significant bits are the flags
/// zero `z`, subtraction `n`, half-carry `h` and carry `c`.
/// Additionally, we have two 16-bit special registers, the stack pointer `sp`, and
/// the program counter `pc`.
pub struct Registers {
 pub a: u8,
 pub b: u8,
 pub c: u8,
 pub d: u8,
 pub e: u8,
 /// Flags register.
 /// - Bit 7: `z`, zero flag
 /// - Bit 6: `n`, subtraction flag (BCD)
 /// - Bit 5: `h`, half-carry flag (BCD)
 /// - Bit 4: `c`, carry flag
 pub f: u8,
 pub h: u8,
 pub l: u8,
 /// Stack pointer register.
 pub sp: u16,
 /// Program counter register.
 pub pc: u16,
}

This struct is complented with handy methods to read and write the registers, either individually or as 16-bit combinations. As you can see, the code is very well documented.

The instructions live in an enum, as do their operands. The instructions enum contains the 256 8-bit instructions, and the 256 16-bit instructions prefixed by 0xCB.

/// # Instructions
/// This enum contains all supported instructions.
/// Instructions that can act with both r8 and r16 registers are first named
/// by default in their r8 version. The r16 versions add a `16` at the end of the name.
#[derive(Debug)]
pub enum Instruction {
 // NoOp.
 NOP(),
 // Stop.
 STOP(),
 // Halt.
 HALT(),

 // Jump HL.
 JPHL(),
 // Jump.
 JP(CC),
 // Relative jumps.
 JR(CC),

 // Push.
 PUSH(R16EXT),
 // Pop.
 POP(R16EXT),

 // Load.
 LD(R8),
 LDcp(R8, R8),
 LD16(R16),
 [...]
}

/// Enumerates the r8 registers.
#[derive(Debug)]
pub enum R8 {
 A,
 B,
 C,
 D,
 E,
 H,
 L,
 HL,
}

/// Enumerates the r16 registers.
#[derive(Debug)]
pub enum R16 {
 BC,
 DE,
 HL,
 SP,
}

/// Enumerates r16 registers for POP and PUSH, which contain AF.
#[derive(Debug)]
pub enum R16EXT {
 BC,
 DE,
 HL,
 AF,
}

/// Enumerates the R16 registers to be used in (some) load operations.
#[derive(Debug)]
pub enum R16LD {
 BC,
 DE,
 HLp,
 HLm,
 A8,
 C,
 A16,
}

/// Enumerates jump conditions, mostly flags.
#[derive(Debug)]
pub enum CC {
 NONE,
 NZ,
 Z,
 NC,
 C,
}

/// RST's target address, divided by 8.
#[derive(Debug)]
pub enum TGT3 {
 T0,
 T1,
 T2,
 T3,
 T4,
 T5,
 T6,
 T7,
}

And then, I have a from_byte(byte: u8) method that matches opcodes to their actual enum values:

impl Instruction {
 /// Construct an instruction from its byte representation.
 pub fn from_byte(byte: u8) -> Option<Instruction> {
 match byte {
 // NOP
 0x00 => Some(Instruction::NOP()),
 // STOP
 0x10 => Some(Instruction::STOP()),

 // LD (r16)
 0x01 => Some(Instruction::LD16(R16::BC)),
 0x11 => Some(Instruction::LD16(R16::DE)),
 0x21 => Some(Instruction::LD16(R16::HL)),
 0x31 => Some(Instruction::LD16(R16::SP)),

 [...]

 // Undocumented OPCODE
 _ => {
 println!(
 "{}: Unimplemented instruction {:#02x}, treating as NOP",
 "WARN".yellow(),
 byte
 );
 Some(Instruction::NOP())
 }
 }
 }

I also have a from_byte_0xcb(byte: u8) method used in the actual implementation to decode the 16-bit opcodes.

Actually implementing the CPU meant building a match with cases for ever single one of the 512 instructions. Each instruction updates the PC, manipulates registers or memory, and sets flags according to specific rules. Some instructions were very straightforward to implement; others, like DAA, required careful consideration of edge cases and bit manipulation. The implementation structure is quite straightforward:

 /// Execute a single instruction, and returns the number of cycles it takes.
 fn execute(&mut self, run_instr: RunInstr, opcode: u8) -> u8 {
 // Actually execute the instruction.
 match run_instr.instr {
 // NOP: no operation.
 Instruction::NOP() => 1,
 // STOP
 Instruction::STOP() => {
 self.stop();
 1
 }
 // HALT
 Instruction::HALT() => {
 // HALT only halts if IME is enabled.
 if self.ime {
 self.halt();
 }
 // Otherwise, HALT acts like NOP.
 1
 }

 // LD r16
 Instruction::LD16(r16) => match r16 {
 R16::BC => {
 let nw = self.read16();
 self.registers.set_bc(nw);
 3
 }
 R16::DE => {
 let nw = self.read16();
 self.registers.set_de(nw);
 3
 }
 R16::HL => {
 let nw = self.read16();
 self.registers.set_hl(nw);
 3
 }
 R16::SP => {
 self.registers.sp = self.read16();
 3
 }
 },

 [...]

 // OPCODE 16-bit (0xCB).
 Instruction::OPCODE16() => {
 // Read next byte, construct instruction, execute 0xCB instruction.
 let opcode0xcb = self.read8();
 let instr0xcb = Instruction::from_byte_0xcb(opcode0xcb);
 let msg = format!("Incorrect 0xCB opcode: {:#04X}", opcode0xcb);
 self.execute_0xcb(instr0xcb.expect(&msg), opcode0xcb)
 }

 // Never should happen.
 _ => panic!("Instruction is not implemented: {:#04X}", opcode),
 }
 }

And, of course, there is an analogous execute_0xcb(...) method that contains the implementation for all the 16-bit opcodes.

The CPU doesn’t run in isolation and needs to be cycle-accurate. Each instruction takes a specific number of machine cycles (M-cycles), typically 1-6, and the rest of the system needs to advance accordingly. Getting timing right is quite important, because the graphics hardware, audio, and even some games depend on precise cycle counts. This is why execute(...) -> u8 returns an unsigned integer. This is the number of cycles that the instruction takes. The main loop handles the input events, and then runs as many cycles per frame as necessary.

 while self.running {
 let frame_start_time = std::time::Instant::now();

 // Handle input events.
 self.handle_events();

 let mut cycles_this_frame: usize = 0;
 while cycles_this_frame < constants::CYCLES_PER_FRAME {
 let (t, m) = self.machine_cycle();
 self.m_cycles += m;
 self.t_cycles += t;
 cycles_this_frame += t as usize;

 // Fill the LCD buffer if needed.
 self.display.render_lcd(&mut self.memory);
 }

 // Flush and present the frame.
 self.display.canvas.flush();
 self.display.present();

 frame_count += 1;

 // Sleep for the remaining time in the frame.
 self.sleep_next_frame(frame_start_time);
 }
 }

The machine cycle runs a single CPU cycle. A CPU cycle does the following:

Read next opcode from the memory address pointed by PC
Increment PC
Decode the instruction
Run the instruction – execute(...)
Run a memory cycle
Update the IME and handle interrupts

The code looks like this:

 /// Main loop of the machine.
 fn cycle(&mut self) -> u8 {
 // Fetch next instruction, and parse it.
 let opcode = self.read8();
 let run_instr = RunInstr::new(opcode, &self.memory, &self.registers);

 // Execute the instruction.
 self.execute(run_instr, opcode)
 }

 /// Run a machine cycle.
 fn machine_cycle(&mut self) -> (u32, u32, bool) {
 // CPU instruction.
 // One machine cycle (M-cycle) is 4 clock cycles.
 let mut m_cycles = if self.running {
 if self.halted {
 // CPU is halted, don't execute instructions but still consume 1 M-cycle.
 1
 } else {
 // Run next CPU instruction.
 self.cycle() as u32
 }
 } else {
 // NOOP instruction.
 1
 };
 let mut t_cycles = m_cycles * 4;

 // Memory cycle.
 if m_cycles > 0 {
 // Memory cycle.
 let (_, r) = self.memory.cycle(t_cycles);
 if !r {
 // Close down.
 self.running = false;
 return (0, 0, false);
 }
 } else {
 t_cycles = 0;
 m_cycles = 0;
 }

 // Update IME.
 self.ime_update();

 // Handle interrupts if necessary.
 let interrupt_m_cycles = self.interrupt_handling();

 // Return.
 if interrupt_m_cycles > 0 {
 (interrupt_m_cycles * 4, interrupt_m_cycles, true)
 } else {
 (t_cycles, m_cycles, true)
 }
 }

We’ll talk about the memory cycle when we reach the memory section. For now, let’s look at interrupts, as it was an interesting challenge. The Game Boy supports five interrupt types: V-Blank, LCD STAT, Timer, Serial, and Joypad. The CPU checks for pending interrupts after each instruction, and if interrupts are enabled and one is pending, it pushes the PC onto the stack and jumps to the corresponding interrupt vector. Implementing the IME flag and its one-instruction delay after the EI instruction was a subtle detail that broke several games until I got it right. This is handled in ime_update() and interrupt_handling():

 /// Updates the IME (Interrupt Master Enable) flag.
 /// This is necessary because the effect of the EI and DI instructions
 /// is delayed by one instruction.
 fn ime_update(&mut self) {
 self.di = match self.di {
 2 => 1,
 1 => {
 self.ime = false;
 0
 }
 _ => 0,
 };
 self.ei = match self.ei {
 2 => 1,
 1 => {
 self.ime = true;
 0
 }
 _ => 0,
 };
 }

 /// Interrupt handling. The IF bit corresponding to this interrupt, and the IME flag
 /// are reset by the CUP. IF acknowledges the interrupt, and IME prevents any other
 /// interrupts from being handled until re-enabled (with RETI).
 /// In this case, the corresponding interrupt handler is called by pushing the PC
 /// to the stack, and then setting it to the address of the interrupt handler.
 fn interrupt_handling(&mut self) -> u32 {
 let pending = self.memory.ie & self.memory.iff;

 // Wake from HALT if any interrupt is pending (even if IME is disabled).
 if pending != 0 {
 if self.halted {
 self.halted = false;
 }
 }

 // If no interrupts are pending OR IME is disabled, return.
 if !self.ime || pending == 0 {
 return 0;
 }

 // Reset IME.
 self.ime = false;

 // IE and IF have the following format:
 //
 // | 7 6 5 | 4 | 3 | 2 | 1 | 0 |
 // | 1 | Joypad | Serial | Timer | LCD | VBlank |
 //
 for i in 0..5 {
 let bit = 1 << i;
 if pending & bit != 0 {
 self.memory.iff &= !bit;
 let pc = self.registers.pc;
 self.push_stack(pc);
 self.registers.pc = match bit {
 0x01 => 0x0040,
 0x02 => 0x0048,
 0x04 => 0x0050,
 0x08 => 0x0058,
 0x10 => 0x0060,
 _ => panic!("Unknown interrupt bit {}", bit),
 };
 break;
 }
 }

 5 // 5 M-cycles = 20 T-cycles
 }

Memory

The Game Boy’s 16-bit address space is divided into distinct regions, each serving a specific purpose:

0x0000-0x3FFF: ROM bank 0 (16 KB, fixed)
0x4000-0x7FFF: ROM bank 1-N (16 KB, switchable via MBCs, more on that later)
0x8000-0x9FFF: Video RAM (8 KB)
0xA000-0xBFFF: External RAM (8 KB, cartridge)
0xC000-0xDFFF: Work RAM (8 KB)
0xE000-0xFDFF: Echo RAM (mirror of 0xC000-0xDDFF)
0xFE00-0xFE9F: OAM (sprite data)
0xFF00-0xFF7F: I/O registers
0xFF80-0xFFFE: High RAM (127 bytes)
0xFFFF: IE register

Implementing memory access required a dispatcher that routes reads and writes to the appropriate hardware component. Reading from VRAM accesses the PPU’s memory, writing to 0xFF00-0xFF7F triggers side effects in various subsystems, and accessing cartridge ROM or RAM goes through the mapper.

The memory struct contains the Work RAM, the High RAM, I/O registers, the IF and IE, a reference to the PPU, a reference to the APU, a reference to the cartridge, the timer, and the joypad.

/// # Memory
/// The Game Boy uses a 2-byte address space (0x0000 to 0xFFFF) to map the different
/// types of memory (RAM, VRAM, Cartridge memory, etc.)
pub struct Memory<'a> {
 /// Work RAM.
 pub wram: [u8; constants::WRAM_SIZE],
 // High RAM.
 pub hram: [u8; constants::HRAM_SIZE],
 // I/O registers.
 pub io: [u8; constants::IO_SIZE],
 // IF: interrupt flag.
 pub iff: u8,
 // IE flag: interrupt enable.
 pub ie: u8,
 // Cartridge reference.
 pub cart: &'a mut Cartridge,
 /// The PPU, Picture Processing Unit.
 pub ppu: PPU,
 /// The timer.
 pub timer: Timer,
 /// The joypad.
 pub joypad: Joypad,
 /// The APU, Audio Processing Unit.
 pub apu: APU,
}

The actual mapping is implemented in the read() and write() operations. The address is matched to the specific range in the address space and forwarded to the appropriate component. For instance, the read method:

 /// Read a byte of memory at the given `address`.
 pub fn read8(&self, address: u16) -> u8 {
 match address {
 0x0000..=0x3FFF => {
 // 16kB bank #0 (cartridge).
 self.cart.read(address)
 }
 0x4000..=0x7FFF => {
 // 16kB switchable ROM bank (cartridge).
 self.cart.read(address)
 }
 0x8000..=0x9FFF => {
 // VRAM.
 self.ppu.read(address)
 }
 0xA000..=0xBFFF => {
 // 8kB switchable RAM bank (cartridge).
 self.cart.read_ram(address)
 }
 [...]
 }
 }

Memory-mapped I/O registers control everything from LCD modes to audio channels. Some registers are read-only, some write-only, and some have bits that behave differently on read versus write. The DMA register at 0xFF46 is particularly interesting. Writing to it triggers a transfer of 160 bytes from ROM or RAM to OAM, during which the CPU should only access High RAM. Getting DMA timing correct was essential for sprite-heavy games.

The PPU

The Picture Processing Unit is what puts pixels on the screen. It generates a 160×144 display running at around 60 Hz. Getting it working meant understanding scanlines, timing, and how backgrounds, windows, and sprites all come together.

The PPU works one scanline at a time. Each scanline takes 456 T-cycles (dots), and a full frame has 154 scanlines: 144 visible ones and 10 during V-Blank. While rendering a scanline, the PPU goes through different modes:

Mode 2 (OAM Search): Scanning which sprites will be visible on this scanline (80 dots)
Mode 3 (Pixel Transfer): Actually drawing the pixels (variable, ~172 dots)
Mode 0 (H-Blank): Horizontal blank, waiting for the scanline to finish (~204 dots)
Mode 1 (V-Blank): Vertical blank, screen done rendering (10 scanlines × 456 dots)

During V-Blank, games usually update their state and prepare for the next frame. My implementation transitions between modes based on the current scanline and dot count:

let new_mode = if self.ly >= 144 {
 1 // VBlank.
} else {
 match self.fdot {
 0..=80 => 2, // OAM Scan.
 81..=252 => 3, // Drawing.
 _ => 0, // HBlank.
 }
};

The background is made up of 8x8 pixel tiles stored in VRAM. Each tile uses 2 bits per pixel, giving you four shades of gray (or colors if you apply a palette). There are two tile data areas (at 0x8000 and 0x8800) and two tile maps that tell you which tiles go where. The LCDC register controls which areas to use, and you can think of it as configuring how the PPU interprets the data in VRAM.

Fetching tile data means reading two bytes per tile row and combining them to get the color IDs:

let tile_addr = tile_addr_base + (line * 2);
let low_byte = self.read(tile_addr);
let high_byte = self.read(tile_addr + 1);

let mut pixels = [0u8; 8];
for i in 0..8 {
 let color_id = ((high_byte >> (7 - i)) & 0x1) << 1 | ((low_byte >> (7 - i)) & 0x1);
 pixels[i] = color_id;
}

Windows work similarly to backgrounds but ignore scrolling. They’re positioned absolutely on screen using WX and WY registers. My implementation tracks an internal window line counter (WLY) that only increments when the window actually renders on a scanline, which is important for games that turn the window on and off mid-frame.

Sprites (or objects) are 8x8 or 8x16 pixels that sit on top of the background. The OAM holds up to 40 sprite definitions, each with position, tile index, and attributes like palette selection, flip flags, and priority. The hardware can only show 10 sprites per scanline, and when more than 10 overlap, priority goes to sprites with lower X coordinates first, then lower OAM index:

// Game Boy priority: X asc, then OAM index asc.
sprites.sort_by(|a, b| {
 if a.x != b.x {
 a.x.cmp(&b.x)
 } else {
 a.oam_index.cmp(&b.oam_index)
 }
});

One of the trickier parts was getting sprite priority right. The priority bit in the sprite attributes controls whether a sprite appears above all background pixels or only above background color 0. I store the background color IDs in a separate buffer as I render the background, then check against those values when drawing sprites:

let bg_color_id = self.priorities[self.ly as usize * constants::DISPLAY_WIDTH + x_pos];

let obj_behind_bg = (sprite.attributes & 0x80) != 0;
let sprite_has_priority = if obj_behind_bg {
 // Priority bit set: only draw sprite over BG color 0.
 bg_color_id == 0
} else {
 // Priority bit clear: always draw sprite (over any BG color).
 true
};

This lets me respect the priority rules properly without having to re-read background data.

I cache tile data while rendering each scanline to avoid repeatedly decoding the same tiles. For backgrounds and windows, I cache up to 32 tiles (one row). For sprites, I use a HashMap keyed by tile ID, line number, and flip flags.

The PPU’s timing had to be cycle-accurate. Each frame needs exactly 70224 T-cycles (154 scanlines × 456 dots). The main loop advances the PPU by the number of cycles the CPU just executed, and the PPU updates its mode and LY register (current scanline) accordingly:

pub fn cycle(&mut self, t_cycles: u32) {
 if !self.is_ppu_enabled() {
 return;
 }

 self.fdot += t_cycles;

 // Update mode if it changed.
 if new_mode != self.mode {
 self.update_mode(new_mode);
 }

 // Handle transition to next scanline.
 if self.fdot >= 456 {
 self.fdot -= 456;
 self.ly = (self.ly + 1) % 154;
 }
}

Getting the timing right was kind of important because some games rely on precise V-Blank and H-Blank timing. Otherwise, you get visual glitches, and good luck debugging them.

In order to test my PPU implementation, I turned to the DMG Acid2 test. This is a rather exhaustive PPU test in the form of a ROM. If you see a happy face with HELLO WORLD! above, and dmg-acid2 by Matt Currie below, you’re good. In my case, the right chin was missing. I spent an ginormous amount of time trying to get it to render properly. This is what I was getting:

Oh no, the right chin is missing!

I was so close. The README has an entry that looks exactly like it. The right chin is rendered using the window, and the window tile map, so I kept looking at the window code. It was a pain, and for the life of me, I could not find the issue even though I looked incessantly. In the end, I fed my code to an LLM and it found the error right away: In my read and write VRAM operations, I had WX and WY backwards!

 // This is NOT correct!
 // WY.
 0xFF4A => self.wx,
 // WX.
 0xFF4B => self.wy,

This was frustrating, but it really goes to show what random side effects such a stupid mistake can have.

After countless hours, the stupid bug was mended and Acid2 was good!

Now that my PPU was more or less feature-complete, I tested it with other games. Seeing the first sprite render correctly, or watching the background scroll smoothly in Mario for the first time made it worth it. The PPU is obviously the most (only?) visible part of the emulator, so getting it to a state where it was able to run all the games I wanted to run felt good.

At the very end, when the graphics were already in an excellent shape, I implemented multiple color palettes. The original Game Boy was monochrome, but you can map those four gray shades to any colors you want. IIRC, the Super Game Boy did something similar, and it allowed you to create your own too! I included a few classic palettes (DMG Classic, Pocket, etc.) and some more creative ones. You can cycle through them while playing with p, which makes the games look fresh.

Some of the palettes in Play Kid.

At this point, I could play lots of games, but I had no audio. This had to be mended.

The APU

I consider the Audio Processing Unit (APU) to be one of the most boring parts of this project. It synthesizes sound through four channels: two pulse wave channels (square waves), one wave channel (custom waveform), and one noise channel (pseudo-random). Each channel has its own registers for frequency, volume, envelope, and duty cycle.

The APU is fully synchronized with the CPU and the PPU, and tus runs at 4.19 MHz. This is the master clock that we’ve seen previously. However, the APU doesn’t do everything at 4.19 MHz. Internally, it uses a frame sequencer that divides this master clock down to 512 Hz to manage high-level audio events:

Length Counters: Clocked at 256 Hz (every 2nd step).
Frequency Sweep: Clocked at 128 Hz (every 4th step).
Volume Envelopes: Clocked at 64 Hz (every 8th step).

It steps through eight different phases, some updating length counters, some update the sweep function, and some updating volume envelopes. My implementation tracks this with a frame timer that increments with each T-cycle and runs at 512 Hz (4194304 Hz / 8192 = 512 Hz):

self.frame_timer += t_cycles;
if self.frame_timer >= 8192 {
 // 512Hz is 4.19 Mhz / 8192.
 self.frame_timer -= 8192;

 // Phases from 0 to 7.
 match self.frame_sequencer {
 0 | 2 | 4 | 6 => {
 self.step_length();
 if self.frame_sequencer == 2 || self.frame_sequencer == 6 {
 self.step_sweep();
 }
 }
 7 => {
 self.step_envelope_ch1();
 self.step_envelope_ch2();
 self.step_envelope_ch4();
 }
 _ => {}
 }
 self.frame_sequencer = (self.frame_sequencer + 1) % 8;
}

Each pulse channel (1 and 2) generates square waves with four different duty cycles. The duty cycle controls the waveform shape and is stored as patterns:

let duty_idx = (self.read(0xFF11) >> 6) as usize;
let patterns = [
 [0, 0, 0, 0, 0, 0, 0, 1], // 12.5%
 [1, 0, 0, 0, 0, 0, 0, 1], // 25%
 [1, 0, 0, 0, 0, 1, 1, 1], // 50%
 [0, 1, 1, 1, 1, 1, 1, 0], // 75%
];

let signal = patterns[duty_idx][self.ch1_duty_step];
let volume_multiplier = (self.ch1_volume as f32 / 15.0) * 0.05;

The wave channel (channel 3) is different. It plays back samples from a 32-nibble waveform stored in Wave RAM. Games can write custom waveforms here, which makes this channel super flexible:

// Get 4-bit sample from Wave RAM (32 samples total, 2 per byte).
let byte = self.wave_ram[self.ch3_sample_idx / 2];
let mut sample = if self.ch3_sample_idx % 2 == 0 {
 byte >> 4 // High nibble.
} else {
 byte & 0x0F // Low nibble.
};

// Volume shift based on NR32 register.
sample >>= volume_shift;

The noise channel (channel 4) uses a LFSR to generate pseudo-random noise. The LFSR is a 15-bit shift register that XORs bits 0 and 1, then shifts the result back in:

let bit0 = self.ch4_lfsr & 0x01;
let bit1 = (self.ch4_lfsr >> 1) & 0x01;
let result = bit0 ^ bit1;

self.ch4_lfsr = (self.ch4_lfsr >> 1) | (result << 14);
if (nr43 & 0x08) != 0 {
 // Short mode (7-bit)
 self.ch4_lfsr = (self.ch4_lfsr & !(1 << 6)) | (result << 6);
}

This creates the characteristic noise used for drums, explosions, and other effects.

The sweep function on channel 1 was particularly tricky. It modulates the frequency over time by shifting the current frequency right and either adding or subtracting the result. The overflow check happens both when sweep is triggered and when calculating the new frequency:

fn calculate_sweep_freq(&mut self) -> u16 {
 let nr10 = self.read(0xFF10);
 let sweep_step = nr10 & 0x07;
 let descending = (nr10 & 0x08) != 0;

 let delta = self.ch1_sweep_shadow_freq >> sweep_step;
 let new_freq = if descending {
 self.ch1_sweep_shadow_freq.saturating_sub(delta)
 } else {
 self.ch1_sweep_shadow_freq + delta
 };

 // Overflow check.
 if new_freq > 2047 {
 self.ch1_enabled = false;
 }

 new_freq
}

Getting the timing and overflow behavior right required reading Pan Docs extensively, and I also looked at RBoy’s implementation.

Sound samples are generated at 44100 Hz and mixed into stereo output based on the NR51 register, which controls left/right panning for each channel. SDL2’s audio queue system fit naturally with the APU design. I generate samples as the emulator runs and queue them for SDL to play back:

let (l_sample, r_sample) = self.generate_sample();
self.buffer.push(l_sample);
self.buffer.push(r_sample);

if self.buffer.len() >= 2048 {
 // Throttle if queue is getting too full
 while self.device.size() > 8192 {
 std::thread::sleep(std::time::Duration::from_millis(1));
 }
 self.device.queue_audio(&self.buffer).unwrap();
 self.buffer.clear();
}

Keeping audio and video synchronized without drift required careful management of cycle counts and buffer sizes. The throttling mechanism prevents the emulator from running too fast when the audio queue backs up.

I implemented channel 1 first, and then ran Super Mario Land. I could hear some sounds (jumps, parts of the music), but the experience was underwhelming. Since all channels follow a similar structure, I had channel 2 working in no time. Now I could hear the iconic melody of the first stage, all the jumps, coins, and enemies. I implemented the other channels soon after.

Interestingly, in Super Mario Land, the music is mostly in channels 2 and 3. Channel 1 is for things like jumps, coins, and some high-pitched parts of melodies. Channel 4 is used in the random noise that some enemies produce when they die (like the bomb turtles), and some percussion beats in the background.

I still have no idea whether my APU contains any bugs, or if it sounds exactly as it’s supposed to. I have no way to tell. To my ears, it sounds amazing.

Cartridges

Most Game Boy cartridges aren’t just ROM chips. They contain MBCs that expand the addressable memory beyond the CPU’s 64 KB limit. Different games use different MBCs, each with its own banking scheme.

Every cartridge has a header header that contains essential information about the game, stored in the first few bytes of the cartridge’s ROM. This information helps the system identify and load the game properly. The header includes:

Logo – the first 48 bytes (0x00-0x2F) contain the Nintendo logo, ensuring the authenticity of the cartridge.
Title – the next 16 bytes (0x34-0x43) store the 8-character game title, which is displayed when the game is loaded.
Game Boy/GBC Flag – the byte at 0x143 indicates whether the game is for the original Game Boy (0x00), or a Game Boy Color-compatible game (0x80).
Cartridge Type – the byte at 0x147 specifies the cartridge type, such as ROM-only or with memory banks. Here is a non-exhaustive list of common values:
- 0x00 – ROM only
- 0x01 – MBC1
- 0x02 – MBC1+RAM
- 0x03 – MBC1+RAM+BATTERY
- 0x05 – MBC2
- 0x06 – MBC2+BATTERY
- 0x11 – MBC3
- 0x12 – MBC3+RAM
- 0x13 – MBC3+RAM+BATTERY
ROM Size – the byte at 0x148 specifies the ROM size in powers of 2. For example, 0x00 is 32KB, 0x01 is 64KB, 0x02 is 128KB, etc.
RAM Size – the byte at 0x149 specifies the RAM size in kilobytes. For example, 0x00 means no RAM, 0x03 means 32KB, 0x04 means 128KB, etc.
Destination Code – located at 0x14A, this byte indicates the region the game is intended for. 0x00 is for Japan, 0x01 for Overseas.
Licensee Code – the byte at 0x14B contains a code identifying the licensed company that produced the game.
Mask ROM Version – the byte at 0x14C is the version number of the game’s mask ROM (usually set to 0x00).
Complement Checksum – the byte at 0x14D is used for the checksum calculation to verify the integrity of the ROM data.
Global Checksum – the last 2 bytes (0x14E-0x14F) store a checksum value that ensures the data integrity of the entire ROM.

This header allows the Game Boy to load and display the game properly by checking for compatibility and loading the correct resources for the specified hardware.

When it comes to MBCs, I implemented ROM-only and MBC1/2/3 into Play Kid.

ROM-only

ROM-only cartridges are the simplest, with no banking at all. These are limited to 32 KB and include simple games like Tetris. No mapper logic needed, just direct reads from the ROM data.

Tetris and Dr. Mario use ROM-only cartridges.

MBC1

MBC1 is the most common mapper, supporting up to 2 MB of ROM and 32 KB of RAM. It uses banking to switch between ROM and RAM banks by writing to specific address ranges. Understanding that writes to ROM addresses don’t actually write to ROM but instead configure the mapper took some mental adjustment.

MBC1 has two banking modes controlled by a mode register. In ROM banking mode (mode 0), you get access to more ROM banks but only one RAM bank. In RAM banking mode (mode 1), you can switch between multiple RAM banks but lose access to the upper ROM banks:

pub fn write(&mut self, address: u16, value: u8) {
 match address {
 // RAM enable.
 0x0000..=0x1FFF => {
 self.ram_enabled = (value & 0x0F) == 0x0A;
 }
 // ROM bank number (lower 5 bits).
 0x2000..=0x3FFF => {
 let mut bank = value as usize & 0x1F;
 if bank == 0 {
 bank = 1; // Bank 0 becomes bank 1.
 }
 self.rom_bank = (self.rom_bank & 0x60) | bank;
 }
 // RAM bank or upper ROM bank bits.
 0x4000..=0x5FFF => {
 let bits = value as usize & 0x03;
 if self.banking_mode == 0 {
 self.rom_bank = (bits << 5) | (self.rom_bank & 0x1F);
 } else {
 self.ram_bank = bits;
 }
 }
 // Banking mode select.
 0x6000..=0x7FFF => {
 self.banking_mode = value & 0x01;
 }
 _ => {}
 }
}

Super Mario Land (1 and 2), Wario Land, Kirby’s Dream Land, and The Amazing Spider-Man all use MBC1.

MBC2

MBC2 has a quirky design. It includes 512×4 bits of built-in RAM, and bit 8 of the write address determines whether you’re configuring ROM banking or RAM enable. Both operations share the same address range (0x0000-0x1FFF):

match addr {
 0x0000..=0x1FFF => {
 if (addr >> 8) & 0x01 == 0 {
 // Bit 8 is 0: RAM enable.
 self.ram_enabled = (val & 0x0F) == 0x0A;
 } else {
 // Bit 8 is 1: ROM bank select.
 self.rom_bank = val & 0x0F;
 if self.rom_bank == 0 {
 self.rom_bank = 1;
 }
 }
 }
 // ...
}

Since MBC2’s RAM is only 4 bits per byte, reads return the value OR’d with 0xF0, and writes mask to the lower 4 bits.

I think I have not tested any game that uses MBC2.

MBC3

MBC3 adds a real-time clock (RTC) alongside standard ROM/RAM banking. The RTC includes seconds, minutes, hours, and days counters that can be accessed by setting the RAM bank register to values 0x08-0x0C. I understand that some games use the RTC to track time-based events, but I have not tested this:

pub fn read_ram(&self, address: u16) -> u8 {
 if !self.ram_enabled {
 return 0xFF;
 }

 match self.ram_bank {
 0x00..=0x03 => {
 // Standard RAM banks.
 let offset = (self.ram_bank * 0x2000) + (address as usize - 0xA000);
 self.ram[offset % self.ram.len()]
 }
 0x08 => self.rtc_seconds,
 0x09 => self.rtc_minutes,
 0x0A => self.rtc_hours,
 0x0B => self.rtc_days_low,
 0x0C => self.rtc_days_high,
 _ => 0xFF,
 }
}

The RTC is latched by writing 0 then 1 to address range 0x6000-0x7FFF. This copies the current time into the registers so the game can read a consistent value. My implementation updates the RTC with system time:

fn update_rtc(&mut self) {
 let now = SystemTime::now()
 .duration_since(UNIX_EPOCH)
 .unwrap()
 .as_secs();

 self.rtc_seconds = (now % 60) as u8;
 self.rtc_minutes = ((now / 60) % 60) as u8;
 self.rtc_hours = ((now / 3600) % 24) as u8;

 let days = (now / 86400) as u16;
 self.rtc_days_low = (days & 0xFF) as u8;

 // Day MSB and carry flag.
 let day_msb = ((days >> 8) & 0x01) as u8;
 self.rtc_days_high = (self.rtc_days_high & 0xFE) | day_msb;

 if days > 511 {
 self.rtc_days_high |= 0x80; // Set overflow flag.
 }
}

Pokémon and Wario Land II use MBC3.

Finally, I implemented battery-backed SRAM. It meant saving the contents of cartridge RAM to .sav files when the emulator exits and loading them on startup, if they exist. You can also trigger the save operation at any time with w. I get the RAM from the cartridge, and then persist it (if not empty):

 /// Save SRAM of current cartridge to `.sav` file.
 pub fn save_sram(&self, rom_path: &str) {
 let save_path = Path::new(rom_path).with_extension("sav");

 // Only save if the mapper actually has RAM.
 let ram_data = match &self.cart_type {
 CartridgeType::MBC1(mbc) => &mbc.get_ram(),
 CartridgeType::MBC2(mbc) => &mbc.get_ram(),
 CartridgeType::MBC3(mbc) => &mbc.get_ram(),
 _ => return,
 };

 if !ram_data.is_empty() || matches!(self.cart_type, CartridgeType::MBC2(_)) {
 if let Ok(mut file) = File::create(&save_path) {
 let _ = file.write_all(ram_data);
 println!(
 "{}: Save data written to disk: {}",
 "WR".magenta(),
 save_path.display()
 );
 }
 }
 }

The load operation checks for the file, loads it if it exists, and passes it to each MBC’s set_ram() method.

This feature is essential for saving progress in many games. Currently, the .sav files are written to the same directory where the ROM file is, with the same name and sav extension.

UI and debug mode

In order to help me debug the emulator as development progressed, I implemented a terminal-based debug interface that let me look under the hood and inspect the internal state of the machine. This is implemented in the src/debugcli.rs file in the repository, but it is now unused.

When the emulator was already working quite well, I implemented a GUI debug mode with a custom immediate-mode UI library built on top of SDL2 rendering. You can enable it at any time with d.

The UI library supports horizontal and vertical layouts, labels, buttons, and text fields. Enough to build a useful debug interface without pulling in a fat dependency. It’s minimal but effective. Writing it was not trivial. I knew that UI libraries are complex and take a lot of work to get right, and this little spin-off sub-project only confirmed this knowledge.

Super Mario Land 2 running in Play Kid with the debug UI.

The debug mode features:

Current instruction, operand, and PC
Opcode
CPU status (either RUNNING or HALTED)
M- and T-cycles
CPU registers and flags
PPU state (LCDC, STAT, LYC, LY, LX)
Joypad state for all inputs (U, D, L, R, A, B, START, SELECT)
Breakpoints

You can step through execution one instruction (F6) or one scanline (F7) at a time, pause and continue (F9), and set breakpoints at specific addresses. You can also reset the CPU with r. I also included an FPS counter (f). You can enable and disable the debug mode at any time with d.

One quality-of-life feature I’m proud of is automatic DPI scaling. On high-DPI displays, the UI scales appropriately so everything remains readable without manual configuration. This was a pain to handle manually. The emulator also respects the Game Boy’s 160:144 aspect ratio by letterboxing the display. It’s a small detail, but it ensures games look correct rather than stretched or squashed.

Future work

Here are some items I would like to do at some point:

Move away from SDL2 in favor of Rust-native libraries like winit+pixels+rodio for easy cross-compilation, and web. I actually have a branch where I implemented the APU using rodio
Implement more MBCs if I stumble upon games that need them.
Add Game Boy Color support. As I understand it, it is not too complicated.

Conclusions

Building Play Kid taught me more about computer architecture, bit manipulation, and systems programming than any tutorial could. Every component of the Game Boy (CPU, memory, graphics, audio) works in concert, and understanding their interactions required patience, testing, and a lot of debugging.

Rust was an excellent choice for this project. My CHIP-8 emulator was already written in Rust, so this was a safe choice for me. I think the language has been praised enough at this time, so I’ll spare this part. SDL2 provided a simple, cross-platform foundation for rendering, input, and audio without unnecessary complexity.

If you’re interested in emulation, I think the Game Boy is a solid choice as first project if you have previous programming experience. The documentation is excellent and the hardware is approachable.

Play Kid’s source code is on Codeberg and GitHub.