Category: AI

In this post I document a little experiment I have been playing around with for a while that explores an absolutely minimal language model. I’m specifically interested in how small a language model can be made while still doing something. I got the idea while I was reading a post about GPT-2. A pre-COVID model, GPT-2 now seems minuscule in comparison to current frontier models. Its largest incarnation was trained on about 40 GB of text, had 1.5B parameters and used an embedding dimension of 1600. Nowadays, a GPT-2-like model can easily be trained in a consumer GPU with ~16-24 GB of VRAM.

If the GPT-2 family of models pales in comparison to current model dimensions, the model I present here today pales in comparison to GPT-2. Since it is extremely minimal, it can be trained on a single CPU core in a few seconds and needs very little RAM. It can have single-digit embedding dimensions, does not follow the transformer architecture, has no attention mechanism, and may not even use any hidden layers! It is a purely academic model, in the sense that it is absolutely simple but does nothing useful.

Building on the previous post about how good current open TTS models are, this post focuses on local LLMs, and their explosion in quality in the recent years months. I have been using OpenRouter with, for the most part, DeepSeek v4 Flash. It is a very capable model, dirt-cheap (seems to be heavily subsidized, at $0.14/M tokens input and $0.28/M tokens output), and it works very well when it is free to browse the internet. I have also been tinkering with local models like the Qwen3.5 family, Gemma 3/4, Mistral, and more. They are good, I haven’t been able to use the versions that can run reasonably on consumer hardware for anything too productive. That is until now. The brand new Qwen3.6 models, particularly the 27B dense, and the 35B A3B MoE, have really impressed the community. They are trivial to run on common setups (e.g. 16GB RTX card, ~32-64GB RAM), and when boosted with agentic capabilities, they perform very well.

Around 2007 I spent half a year in the University of Aberdeen working on my final year project involving NLP. The project consisted of a series of modules to build an graphical adventure-type game controlled by language input in the form of text. It also featured speech output generated by an early TTS system. To achieve this, we managed to partner with a group at La Salle University that were working on a TTS system for Catalan. It was a closed system that was accessible via a web API, but that option was far too slow for real time use. I ended up preprocessing the audio of all dialogs in the project into WAV files just to be able to play them in sync with mouth movements. At that time, I was amazed that a computer could so easily convert text to an understandable audio file. The voice was very plain and robotic, with no emotion whatsoever. The results were hit or miss, but it worked.

Fast forward to today, TTS systems are everywhere. Several groups have released low-parameter TTS models that run very well on consumer hardware. I have been using the lightweight Kitten TTS for a while with fantastic results.

A little over a year ago I set up a local pipeline to use different LLMs to respond to Gaia Sky questions using RAG. In that post, I built a dynamic scrapper that parsed the Gaia Sky website and documentation and ingested the content it into a vector database. Then, I built a minimal terminal chatbot interface that received the user prompt, queried the database for semantically similar data, and built up the context for each LLM call. The results were promising, and I found that they (obviously) strongly depended on the model used.

Fast forward a few months, and the Qwen 3.5 models were released by Alibaba. The general consensus is that they are quite good for their size. I’ve been testing them for local inference with a similar impression. I thought that it would be interesting to repeat the exercise of creating a Gaia Sky AI assistant, but using a radically different approach: Instead of RAG, I would fine-tune the model itself. In this post, I describe this fine-tuning project, from the creation and engineering of the training dataset to the fine-tuning and production of the final GGUF models.

I like running my own LLMs locally. Open models are becoming more and more powerful, with exciting releases like the latest Qwen 3.5 family scoring highly in benchmarks even in their smaller variants. This makes managing and running your own models more viable, as it becomes increasingly easy to repurpose old hardware for local inference with progressively better results. For local users and modest purposes, the GGUF format introduced by llama.cpp is the de-facto default.

Since local inference is typically heavily restricted by the available hardware, several optimization techniques have been implemented to make the models leaner and faster. Perhaps the most important of these is quantization, which trims down the bit count per parameter to achieve lower memory usage and (sometimes) faster inference¹. The challenge is that there are many different formats and strategies for quantization. In this post, I summarize them, providing a bird’s-eye view on the available techniques, their strengths, and their weaknesses.

The release of LM Studio 0.4.5 has introduced a much needed feature in this local LLM suite that has it much more attractive with respect to other similar projects. LM Link allows you to connect multiple LM Studio instances across your network to share models and perform inference seamlessly.