The Bajoran Engineer

Ornith-1.5-9B: strong benchmarks, stale Python, and why that's fixable

Ornith-1.5-9B posts big agentic coding numbers and fits on a 16 GB laptop. It also barely knows Python 3.15. Here's what it is, how to run it, and how a community fine-tune closes the gap.

Title card: Ornith-1.5-9B: strong benchmarks, stale Python

Ornith-1.5-9B is the base model under Satyrn AI's first Python 3.15 fine-tune, Ornith-1.5-9B-py3.15. It's a good case study in a lesson every local-AI user needs: a strong benchmark score doesn't mean the model knows the Python you write today.

#What Ornith-1.5 is

Ornith-1.5 came out in August 2026 from the Ornith team, in three sizes: a 397B mixture-of-experts model, a 35B mixture-of-experts model with 3B active parameters, and a 9B dense model.

The 9B is the one that matters for laptops:

  • License: MIT, per the model card, so anyone can fine-tune and redistribute it.
  • Lineage: Ornith-1.0 was built on Qwen3.5 and Gemma 4, and 1.5 extends it.
  • Context: up to 262,144 tokens.
  • Training idea: a self-improvement loop. The model proposes harder tasks, writes a scaffold for each one, and attempts solutions, and the rewards from those attempts train all three steps with reinforcement learning.

#The numbers it advertises

These are Ornith's own figures for the 9B, averaged over five runs:

text
SWE-bench Verified           70.6
GPQA Diamond                 86.4
BrowseComp                   56.4
Terminal-Bench 2.1           46.2
HLE (no tools)               20.2

They're impressive for a model this size. They're also vendor-reported, and they measure general coding and agentic skill. None of them asks whether the model knows a Python release that came out after its training.

#Run it on a 16 GB laptop

The GGUF build comes in Q4_K_M (5.78 GB), Q5_K_M (6.64 GB), Q6_K (7.56 GB), Q8_0 (9.79 GB) and BF16 (18.4 GB). Q4_K_M fits comfortably on a 16 GB machine:

bash
brew install llama.cpp
curl -L -o ornith-9b.gguf https://huggingface.co/ornith-ai/Ornith-1.5-9B-GGUF/resolve/main/Ornith-1.5-9B-Q4_K_M.gguf
llama-server -m ornith-9b.gguf --jinja -c 4096 -ngl 99 --port 8080

That gives you an OpenAI-compatible endpoint on port 8080. On an Apple M1 with 16 GB, it generated about 4.8 tokens per second.

#What it doesn't know: Python 3.15

I asked the base 9B to write code for five features that are new in Python 3.15, eight times each, then ran every answer. It got credit only if the code used the feature, ran, and printed the right result.

It scored 1 out of 40.

The failures were confident. Asked for the new frozendict built-in, it wrote from types import FrozenDict, which doesn't exist. Asked to flatten a list with * in a comprehension, it wrote print([*[[1, 2], [3], [4, 5]]]) every time. That code runs, and it prints the list back unflattened.

None of this is a knock on Ornith. Python 3.15 shipped on October 1, after the model was trained. Every model has a knowledge cutoff, and a strong agent that doesn't know the language version you use will confidently write the old version.

#Closing the gap with a fine-tune

Because Ornith is MIT-licensed, the Satyrn AI community could fine-tune it on Python 3.15 as Ornith-1.5-9B-py3.15. The satyrn-models pipeline continues pretraining on CPython's docs and source, then fine-tunes on prompt-and-response pairs generated from real code. A reinforcement learning stage is still in progress.

Same laptop, same quantization, same settings, same 40 prompts:

text
                     Base Ornith 9B   Satyrn fine-tune
Correct              1/40             12/40
Used the feature     1/40             17/40
Tokens per second    4.8              4.6

It isn't fluent yet. Lazy imports and the new sampling profiler didn't move at all. But it went from almost nothing to useful on day one, at the same speed.

#What this means for developer velocity

Developer velocity isn't tokens per second. On this laptop both models generate at about the same speed, and neither is fast. Velocity is how quickly you get from asking to working, merged code. That depends far more on how often the model is right than on how fast it types.

  • Wrong answers cost more than slow ones. An invented from types import FrozenDict fails loudly, but you still stop, read the traceback and look up the real API. The unflattened list is worse: it runs, so you may not notice until a test or a user does. Each one turns a few seconds of generation into minutes of debugging.
  • Near misses are fast to fix. The fine-tune's Movie.__closed is one underscore pair from right, and Python 3.15's error message names the fix. Going from 1 to 12 correct out of 40, and from 1 to 17 answers that at least reached for the right feature, means more answers you can keep or fix in seconds.
  • Local removes the waiting you don't see. There's no API key to request, no rate limit, no outage and no per-token bill that makes you ration questions. You can run the same check on the train.
  • Current knowledge keeps you in flow. A model that knows the Python you're writing lets you stay in your editor, instead of switching to the docs to check whether its suggestion even exists.

The fastest model is the one you don't have to second-guess.

#Takeaways

  1. Check the knowledge cutoff, not just the benchmarks. Test any model on the newest features of the tools you use.
  2. Grade by running the code. Plausible-looking output is the failure you can't see.
  3. Pick permissively licensed bases. MIT is what made a community fine-tune possible.
  4. Fine-tune for currency, not size. A 9B that knows today's Python is worth more on your laptop than a bigger model that doesn't.
  5. Measure velocity by fixes, not speed. Count how many answers you can use as-is or fix in seconds, not how fast tokens arrive.

Rerun my test against Ornith or any other model with py315-check, and share your numbers on the Satyrn Discord.

If this post helped, you can support my open source work on Ko-fi.