I started working on xkey project with a simple goal: make my keyboard act like a gamepad on Linux. I knew I could read key events from /dev/input/event*, and I had heard about uinput for creating virtual input devices, but I did not really understand what happened between my user-space program and the kernel.

To make xkey work, I had to answer questions like:

• How does ioctl(fd, EVIOCGRAB, 1) steal the keyboard from the rest of the system?

• How does writing to /dev/uinput make a new device appear in /dev/input?

This post is the story of what I learned while digging through the Linux kernel source. It traces ioctl from user space, through the VFS dispatch layer, and into driver-specific handlers, using the input subsystem (/dev/input/event*) and uinput (/dev/uinput) as the running examples from xkey.

Everything Is a File: Including Your Keyboard

I was already familiar with the Unix mantra: everything is a file. In Linux, hardware devices are exposed through the virtual filesystem as special files. This is not just a cute abstraction, it is the reason the same open(), read(), write(), and close() calls work on regular files, keyboards, mice, and virtual controllers.

When I list the devices on my laptop using evtest, I see entries like this:

Available input devices:
Event      Name                 Path
------------------------------------------------------------
17         HD-Audio Generic Headphone /dev/input/event17
16         HD-Audio Generic Mic /dev/input/event16
15         HDA NVidia HDMI/DP,pcm=9 /dev/input/event15
14         HDA NVidia HDMI/DP,pcm=8 /dev/input/event14
13         HDA NVidia HDMI/DP,pcm=7 /dev/input/event13
12         HDA NVidia HDMI/DP,pcm=3 /dev/input/event12
11         Ideapad extra buttons /dev/input/event11
10         ELAN06FA:00 04F3:327E Touchpad /dev/input/event10
9          ELAN06FA:00 04F3:327E Mouse /dev/input/event9
8          Video Bus            /dev/input/event8
7          ITE Tech. Inc. ITE Device(8176) Wireless Radio Control /dev/input/event7
6          Video Bus            /dev/input/event6
5          ITE Tech. Inc. ITE Device(8176) Keyboard /dev/input/event5
2          AT Translated Set 2 keyboard /dev/input/event2
1          Lid Switch           /dev/input/event1
0          Power Button         /dev/input/event0

Each row represents a real piece of hardware (or a virtual device) that the kernel has registered as a file. The kernel's device model creates these nodes so that user-space programs do not need to know the physical bus address or driver name they just need a path.

What makes a device file different from a regular file is the file operations table attached to it. When I call open("/dev/input/event5", O_RDONLY), the kernel resolves the path to an inode, then to a struct file *, and finally to a struct file_operations * that tells the kernel what functions to call for read, write, ioctl, and so on.

This is the key insight: a device driver registers a table of callbacks, and the kernel uses that table to treat hardware as a file.

The User-Space Call

In xkey, the first ioctl I use is EVIOCGRAB:

#include <sys/ioctl.h>

int keyboard_fd = open("/dev/input/event5", O_RDONLY);
ioctl(keyboard_fd, EVIOCGRAB, 1);  // Grab exclusive access to the keyboard

At this point, the C library (glibc) wraps the call into a single system call. On x86_64, that becomes something like:

mov eax, 16         # __NR_ioctl
mov edi, edi        # fd
mov esi, 0x40044518 # cmd (EVIOCGRAB = _IOW('E', 0x90, int))
mov edx, 1          # arg (integer 1, not a pointer here)
syscall

The argument arg is passed as an unsigned long. On 64-bit systems it usually holds a user-space pointer, but for simple commands like EVIOCGRAB it can be a literal integer. The driver decides how to interpret it.

Entering Kernel Space: fs/ioctl.c

All three arguments (fd, cmd, arg) arrive in the kernel's syscall entry code, which eventually lands in the C wrapper defined in fs/ioctl.c.

The Syscall Wrapper

fs/ioctl.c:583:

SYSCALL_DEFINE3(ioctl, unsigned int, fd, unsigned int, cmd, unsigned long, arg)
{
	CLASS(fd, f)(fd);
	int error;

	if (fd_empty(f))
		return -EBADF;

	error = security_file_ioctl(fd_file(f), cmd, arg);
	if (error)
		return error;

	error = do_vfs_ioctl(fd_file(f), fd, cmd, arg);
	if (error == -ENOIOCTLCMD)
		error = vfs_ioctl(fd_file(f), cmd, arg);

	return error;
}

This is just a thin wrapper. Its job is:

1. Resolve the fd → get a struct fd (file descriptor + reference count).

2. Run LSM/security checks via security_file_ioctl().

3. Try common VFS ioctls first via do_vfs_ioctl().

4. Fall back to the driver's handler via vfs_ioctl() if the command is not a generic one.

The Driver Dispatch: vfs_ioctl()

fs/ioctl.c:33:

static int vfs_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
{
	int error = -ENOTTY;

	if (!filp->f_op->unlocked_ioctl)
		goto out;

	error = filp->f_op->unlocked_ioctl(filp, cmd, arg);
	if (error == -ENOIOCTLCMD)
		error = -ENOTTY;
 out:
	return error;
}

This is when things started clicking to me. The kernel does not know what ioctl does. It looks up the file_operations table associated with the open file and jumps to whatever function the driver registered under unlocked_ioctl.

The ioctl syscall is a generic dispatcher. The actual meaning of EVIOCGRAB, UI_SET_EVBIT, or UI_DEV_CREATE lives entirely in the driver.

The file_operations Table

Every open file in Linux has a struct file *. That structure holds a pointer to the file's operations table:

include/linux/fs.h:1926:

struct file_operations {
	struct module *owner;
	fop_flags_t fop_flags;
	loff_t (*llseek) (struct file *, loff_t, int);
	ssize_t (*read) (struct file *, char __user *, size_t, loff_t *);
	ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *);
	ssize_t (*read_iter) (struct kiocb *, struct iov_iter *);
	ssize_t (*write_iter) (struct kiocb *, struct iov_iter *);
	int (*iopoll)(struct kiocb *kiocb, struct io_comp_batch *,
			unsigned int flags);
	///
}

The Input Subsystem's Table

For /dev/input/event* files, the table is defined in drivers/input/evdev.c:1290:

static const struct file_operations evdev_fops = {
	.owner		= THIS_MODULE,
	.read		= evdev_read,
	.write		= evdev_write,
	.poll		= evdev_poll,
	.open		= evdev_open,
	.release	= evdev_release,
	.unlocked_ioctl	= evdev_ioctl,
#ifdef CONFIG_COMPAT
	.compat_ioctl	= evdev_ioctl_compat,
#endif
	.fasync		= evdev_fasync,
};

When ioctl(fd, EVIOCGRAB, 1) is called on an event node, f_op->unlocked_ioctl points to evdev_ioctl(). This leads to a chain of function calls and ends at evdev_do_ioctl() defined at drivers/input/evdev.c:1027 that has the function calls defined for different values of cmd, this is where the EVIOCGRAB call is finally handled.

static long evdev_do_ioctl(struct file *file, unsigned int cmd, svoid __user *p, int compat_mode)
{
	struct evdev_client *client = file->private_data;
	struct evdev *evdev = client->evdev;
	struct input_dev *dev = evdev->handle.dev;
	struct input_absinfo abs;
	struct input_mask mask;
	struct ff_effect effect;
	int __user *ip = (int __user *)p;
	unsigned int i, t, u, v;
	unsigned int size;
	int error;

	/* First we check for fixed-length commands */
	switch (cmd) {
	/// ...
	case EVIOCGRAB:
		if (p)
			return evdev_grab(evdev, client);
		else
			return evdev_ungrab(evdev, client);
	/// ...
	}
}

Key insight: dev->grab stores a pointer to the client that owns the device exclusively. This is how the kernel knows which process owns the device.

drivers/input/evdev.c:28:

struct evdev {
	int open;
	struct input_handle handle;
	struct evdev_client __rcu *grab; // pointer to the client that owns the device
	struct list_head client_list;
	spinlock_t client_lock; /* protects client_list */
	struct mutex mutex;
	struct device dev;
	struct cdev cdev;
	bool exist;
};

The uinput Table

For /dev/uinput, the table is in drivers/input/misc/uinput.c:1146:

static const struct file_operations uinput_fops = {
	.owner		= THIS_MODULE,
	.open		= uinput_open,
	.release	= uinput_release,
	.read		= uinput_read,
	.write		= uinput_write,
	.poll		= uinput_poll,
	.unlocked_ioctl	= uinput_ioctl,
#ifdef CONFIG_COMPAT
	.compat_ioctl	= uinput_compat_ioctl,
#endif
};

Here, the handler is uinput_ioctl() which calls into uniput_ioctl_handler() drivers/input/misc/uinput.c.

Each driver defines its own semantics. The generic fs/ioctl.c is completely agnostic about what the command means.

Encoding the cmd Parameter

The cmd (request) number is not an arbitrary integer. It is a carefully packed 32-bit value that encodes:

The Macros in include/linux/ioctl.h

include/uapi/asm-generic/ioctl.h:

#define _IOC_NONE  0U
#define _IOC_WRITE 1U
#define _IOC_READ  2U

#define _IOC(dir,type,nr,size) \
    (((dir)  << _IOC_DIRSHIFT) | \
     ((type) << _IOC_TYPESHIFT) | \
     ((nr)   << _IOC_NRSHIFT) | \
     ((size) << _IOC_SIZESHIFT))

/*
 * Used to create numbers.
 *
 * NOTE: _IOW means userland is writing and kernel is reading. _IOR
 * means userland is reading and kernel is writing.
 */
#define _IO(type,nr)          _IOC(_IOC_NONE,(type),(nr),0)
#define _IOR(type,nr,size)    _IOC(_IOC_READ,(type),(nr),sizeof(size))
#define _IOW(type,nr,size)    _IOC(_IOC_WRITE,(type),(nr),sizeof(size))
#define _IOWR(type,nr,size)   _IOC(_IOC_READ|_IOC_WRITE,(type),(nr),sizeof(size))

/* used to decode ioctl numbers.. */
#define _IOC_DIR(nr)		(((nr) >> _IOC_DIRSHIFT) & _IOC_DIRMASK)
#define _IOC_TYPE(nr)		(((nr) >> _IOC_TYPESHIFT) & _IOC_TYPEMASK)
#define _IOC_NR(nr)		(((nr) >> _IOC_NRSHIFT) & _IOC_NRMASK)
#define _IOC_SIZE(nr)		(((nr) >> _IOC_SIZESHIFT) & _IOC_SIZEMASK)

So EVIOCGRAB expands to:

#define EVIOCGRAB		_IOW('E', 0x90, int)			/* Grab/Release device */
// = _IOC(_IOC_WRITE, 'E', 0x18, sizeof(int))
// = 0x40044518  (on little-endian)

Why Size Matters

The size field is used to validate user data. When the driver calls copy_from_user() or copy_to_user(), the encoded size tells the caller how much memory is involved.

uinput: Creating a Virtual Device via ioctl

The second half of xkey creates a virtual Xbox 360 controller.

ioctl(xkey_fd, UI_SET_EVBIT, EV_KEY);
ioctl(xkey_fd, UI_SET_EVBIT, EV_ABS);
ioctl(xkey_fd, UI_DEV_SETUP, &usetup);
ioctl(xkey_fd, UI_DEV_CREATE);

input_ioctl_handler()

drivers/input/misc/uinput.c:

static long uinput_ioctl_handler(struct file *file, unsigned int cmd, unsigned long arg, void __user *p)
{
	int retval;
	struct uinput_device *udev = file->private_data;
	struct uinput_ff_upload ff_up;
	struct uinput_ff_erase  ff_erase;
	struct uinput_request *req;
	char *phys;
	const char *name;
	unsigned int size;

	// ...

	switch (cmd) {
	/// ...
	case UI_SET_EVBIT:
		retval = uinput_set_bit(arg, evbit, EV_MAX);
		goto out;

	case UI_DEV_CREATE:
		retval = uinput_create_device(udev);
		goto out;

	case UI_DEV_SETUP:
		retval = uinput_dev_setup(udev, p);
		goto out;
	/// ...
	}
}

UI_DEV_CREATE: Creating the Device Node

drivers/input/misc/uinput.c:237:

static int uinput_create_device(struct uinput_device *udev)
{
    struct input_dev *dev = ud->dev;
    int error, nslot;

	/// ...

    dev->event = uinput_dev_event;

	input_set_drvdata(udev->dev, udev);

	error = input_register_device(udev->dev);
	if (error)
		goto fail2;

	// ...
    return 0;
}

The input_register_device() function does a lot of heavy lifting from here:

1. Assigns a minor number from the input subsystem's dynamic range.

2. Creates the device node (/dev/input/eventN).

3. Adds the device to the global input device list.

4. Makes the device visible to user-space tools like libinput and evtest.

So when xkey calls ioctl(xkey_fd, UI_DEV_CREATE), a new file appears in /dev/input/, and it behaves exactly like a real controller.

Where Does /dev/uinput Itself Come From?

One more question bothered me while writing xkey: if UI_DEV_CREATE creates the virtual controller node, what creates the /dev/uinput node that xkey opens in the first place? The answer is in the driver's own registration code at the bottom of drivers/input/misc/uinput.c:

static const struct file_operations uinput_fops = {
	.owner		= THIS_MODULE,
	.open		= uinput_open,
	.release	= uinput_release,
	.read		= uinput_read,
	.write		= uinput_write,
	.poll		= uinput_poll,
	.unlocked_ioctl	= uinput_ioctl,
#ifdef CONFIG_COMPAT
	.compat_ioctl	= uinput_compat_ioctl,
#endif
};

static struct miscdevice uinput_misc = {
	.fops		= &uinput_fops,
	.minor		= UINPUT_MINOR,
	.name		= UINPUT_NAME,
};
module_misc_device(uinput_misc);

This is the piece that made the "everything is a file" idea concrete for me:

1. uinput_fops is the file operations table for /dev/uinput. It points unlocked_ioctl to uinput_ioctl(), which is how our ioctl calls from xkey reach the kernel.

2. uinput_misc is a miscdevice structure. The misc subsystem is a simple way for drivers to register a single character device node without needing a dedicated major number.

3. module_misc_device(uinput_misc) registers the device. When the uinput module loads, the kernel creates /dev/uinput with the registered minor number (UINPUT_MINOR = 223).

So the full chain is:

module_misc_device(uinput_misc)
        ↓
   /dev/uinput appears
        ↓
   xkey: open("/dev/uinput", O_WRONLY | O_NONBLOCK)
        ↓
   ioctl(xkey_fd, UI_DEV_CREATE)
        ↓
   input_register_device()
        ↓
   /dev/input/eventN (Microsoft X-Box 360 pad) appears

The same pattern is repeated across the kernel: a driver registers a file_operations table, the kernel exposes it as a file, and user space talks to hardware or in this case, creates new hardware through that file.

xkey in Action: Program Logs

Here is what xkey looks like when it runs on my laptop. First it lists input devices, I select the keyboard, and then it prints raw key events.

Available input devices:
Event      Name                 Path
------------------------------------------------------------
17         HD-Audio Generic Headphone /dev/input/event17
16         HD-Audio Generic Mic /dev/input/event16
15         HDA NVidia HDMI/DP,pcm=9 /dev/input/event15
14         HDA NVidia HDMI/DP,pcm=8 /dev/input/event14
13         HDA NVidia HDMI/DP,pcm=7 /dev/input/event13
12         HDA NVidia HDMI/DP,pcm=3 /dev/input/event12
11         Ideapad extra buttons /dev/input/event11
10         ELAN06FA:00 04F3:327E Touchpad /dev/input/event10
9          ELAN06FA:00 04F3:327E Mouse /dev/input/event9
8          Video Bus            /dev/input/event8
7          ITE Tech. Inc. ITE Device(8176) Wireless Radio Control /dev/input/event7
6          Video Bus            /dev/input/event6
5          ITE Tech. Inc. ITE Device(8176) Keyboard /dev/input/event5
2          AT Translated Set 2 keyboard /dev/input/event2
1          Lid Switch           /dev/input/event1
0          Power Button         /dev/input/event0

Enter the event number for your keyboard: 5
RELEASE  code=0x001c (28)
--------SYN REPORT--------
PRESS  code=0x001e (30)
--------SYN REPORT--------
aRELEASE  code=0x001e (30)
--------SYN REPORT--------
PRESS  code=0x001f (31)
--------SYN REPORT--------
sPRESS  code=0x0020 (32)
--------SYN REPORT--------
dRELEASE  code=0x001f (31)
--------SYN REPORT--------
RELEASE  code=0x0020 (32)
--------SYN REPORT--------

I pressed Enter (code 28), then A (code 30), S (code 31), and D (code 32). Each key press and release is reported as an EV_KEY event, followed by an EV_SYN/SYN_REPORT event that marks the end of a batch.

When I press Esc, xkey toggles controller mode, grabs the keyboard, and starts writing translated events to /dev/uinput.

Verification with evtest

To prove that the virtual Xbox 360 controller actually exists and produces events, I run evtest on the new device node. After starting xkey and toggling controller mode, a new /dev/input/eventN node appears with the name Microsoft X-Box 360 pad.

When I press mapped keys, evtest shows the translated controller events. For example, pressing W and A together produces:

Event: time 1234567890.123456, type 3 (EV_ABS), code 0 (ABS_X), value -23170
Event: time 1234567890.123456, type 3 (EV_ABS), code 1 (ABS_Y), value -23170
Event: time 1234567890.123456, type 0 (EV_SYN), code 0 (SYN_REPORT), value 0

This confirms that the virtual device is registered correctly, the axis ranges match the setup values, and events flow from the physical keyboard through xkey into the kernel's input subsystem.

Conclusion

The xkey project is small, but it touches the full stack: device nodes, the VFS, the input subsystem, and user/kernel data copying. Reading the kernel source for fs/ioctl.c, drivers/input/evdev.c, and drivers/input/misc/uinput.c turned a confusing API into a clear mental model. That is the most valuable thing I got out of it. And yes, it can run Doom.

Hi guys, I've been working with SLMs recently and I wanted to share some thoughts and experiences from a rabbit hole I accidentally fell into.

For context, I'm really into systems programming, and my daily driver is a fairly modest machine with 24 GB RAM, an NVIDIA RTX 4050 with 6 GB of VRAM, and an AMD Ryzen 7 processor.

From Ollama to llama.cpp

Like most developers getting into local AI, my first choice was Ollama. It's incredibly easy to get started with and does a great job of making local models accessible. But after using it for a while, I found myself wanting more observablity and control. I wanted to understand where my memory was going, how layers were being offloaded, what was sitting in VRAM versus system RAM, and why performance changed so dramatically between different configurations. Ollama abstracts most of that away, which is great for usability, but I wanted to go beyond the abstraction.

That curiosity eventually led me to llama.cpp. What started as a search for a more customizable inference engine quickly turned into an exercise in systems programming. Being able to compile custom binaries, inspect memory allocations, benchmark different runtime configurations, and observe exactly what the model was doing made a huge difference. Instead of treating the model as a black box, I could see every tradeoff being made underneath.

The Challenge: Qwen 3.6 35B-A3B MoE on 6 GB VRAM

At some point I decided to see if I could run a Q4_K_M quantized version of Qwen 3.6 35B-A3B MoE locally. On paper, that sounds ridiculous for a laptop with only 6 GB of VRAM. While that would generally be true for traditional dense models, MoE architectures change the equation. The model contains 35 billion parameters, yet only around 3 billion are active for a given token. Once I started understanding how the architecture worked, the challenge stopped being about model size and became a question of data movement and memory management.

The naive approach fails fast

Initially, I offloaded most layers to the GPU and left the rest on the CPU as it made the most sense to do so. The results were terrible, with generation speeds hovering around 3 tokens per second. At first I assumed the GPU simply wasn't powerful enough, but the utilization numbers didn't support that conclusion. After digging deeper, I realized I was bottlenecked by PCIe transfers. Because Qwen 3.6 is a MoE model, the active experts and shared layers were spread across different devices, creating significant CPU to GPU communication overhead. The model was spending more time moving data than generating tokens.

The fixes that moved the needle

After a lot of experimentation, benchmarking, and failed configurations, I found a setup that dramatically improved performance. By keeping the MoE experts pinned in system RAM while letting the GPU handle the shared layers, generation speed increased from roughly 3 tokens per second to around 15 tokens per second. The improvement came entirely from understanding how data was flowing through the system and reducing unnecessary transfers.

I also enabled Flash Attention (--flash-attn on) and experimented with TurboQuant KV cache compression (--cache-type-k turbo4, --cache-type-v turbo3 — 4-bit keys, 3-bit values) by using a fork of llama.cpp maintained by TheTom. The latter ended up being one of the most impressive optimizations I came across. It made context lengths that would otherwise be impossible fit comfortably within the hardware constraints. It's worth thinking about how different "fitting" and "performing well" really are.

The Final Configuration

After all the experimentation, here is the llama-server launch command I settled on, running the Q4_K_M quant of bartowski/Qwen_Qwen3.6-35B-A3B-GGUF:

llama-server \
    --model Qwen_Qwen3.6-35B-A3B-Q4_K_M.gguf \
    --port 8080 \
    --host 0.0.0.0 \
    --ctx-size 262144 \
    --cache-type-k turbo4 \
    --cache-type-v turbo3 \
    --n-gpu-layers 999 \
    --n-cpu-moe 41 \
    --reasoning on \
    --reasoning-format deepseek \
    --reasoning-budget -1 \
    --parallel 1 \
    --flash-attn on

A quick walkthrough of what each flag is doing:

• --ctx-size 262144: Allocates the full 262K context window. TurboQuant is what makes this fit in 6 GB VRAM; without it, you'd be hard-capped at a fraction of this.

• --cache-type-k turbo4 / --cache-type-v turbo3: 4-bit keys and 3-bit values on the KV cache. Near-lossless compression from Google DeepMind's technique, the single biggest reason 262K context is reachable at all.

• --n-gpu-layers 999: Pulls everything possible onto the GPU. Combined with the next flag, this means shared/base layers live in VRAM while the experts stay in system RAM.

• --n-cpu-moe 41: Pins the expert weights of the first 41 layers to system RAM instead of VRAM. This is the floor on a 6 GB card, lowering it to push more experts onto the GPU OOMs once the KV cache needs room. This was the change that took generation from ~3 tok/s to ~15 tok/s.

• --reasoning on / --reasoning-format deepseek / --reasoning-budget -1: Enables chain-of-thought output, rendered into message.reasoning_content using the DeepSeek-style format. The -1 budget lets the model decide how much reasoning to emit per request.

• --parallel 1: Keeps a single request slot on the server. This prevents context collisions when the frontend fires retry or prefill requests in parallel.

• --flash-attn on: Reduces the memory and compute cost of attention layers — a real speedup and VRAM saving that compounds as the context window grows.

Benchmarks

To quantify this, I benchmarked the final configuration using llama-bench with five runs per test, --n-cpu-moe 41, and no --no-mmap.

At a context length of 512 tokens, the model generated at approximately 31.2 tokens per second. At 2K context it was still around 31.8 tokens per second, and at 8K context it remained a very usable 28.1 tokens per second. Even at 32K context the model was producing roughly 18.9 tokens per second, which feels perfectly comfortable for interactive work.

The interesting part comes when you start pushing truly massive contexts. At 131K tokens, throughput drops to around 7.5 tokens per second. At the full 262K context window, generation falls to approximately 3.5 tokens per second. The model can technically handle the context, and TurboQuant makes it fit into memory, but the computational cost doesn't disappear. Large context windows are possible, but they are definitely not free.

The --no-mmap Tradeoff

The rabbit hole got even deeper when I started investigating memory mapping. A single runtime flag ended up teaching me more about operating system behavior than I expected. With memory mapping enabled, RAM usage stayed lower because the operating system could page model weights in as needed. The downside was that as context lengths grew larger, page faults became increasingly visible in latency. Disabling memory mapping forced the model weights into RAM and significantly improved responsiveness, but now the model was consuming over twenty gigabytes of memory. The machine became noticeably less pleasant to use for anything else.

CLI Tools: SLMs, Prompts, and Agent Design

To actually use the model with a harness I went ahead with opencode, but it didn't work nearly as well as I expected. Probably because it has large system prompts bundled into the agent which consumes a significant portion of the model's reasoning budget, which smaller models struggle to handle. The responses felt inconsistent and the models often lost track of the task.

To counter this I switched to a much more minimal agent setup using pi. The difference was surprisingly noticeable. Simpler prompts, less orchestration overhead, and the model stayed focused for much longer. It was a good reminder that agent design matters just as much as model size.

Web Chat Interface: Open WebUI

Running the server with a single request slot also eliminated a number of strange context issues caused by overlapping requests and retries from the frontend. At this point my setup is fairly simple: one terminal running llama-server, another running Open WebUI, and a browser tab pointed at localhost. It's become a surprisingly comfortable daily-driver workflow.

Who Is This Actually Useful For?

I can see this being used for low-risk, high-impact tasks like document summarization, scripting, working with small codebases, debugging configs, and generating initial project scaffolding. Anyone who knows what they're doing can get a lot out of smaller models like these.

Closing Thoughts

The whole process was really fun as I started out trying to run a large language model on a laptop and ended up applying concepts from operating systems, memory management, hardware bottlenecks, and performance engineering instead. It's really interesting how AI is not just a machine learning problem anymore but also a distributed systems problem, a compiler optimization problem, a hardware problem, an enterprise software problem and a product problem at the same time.