The Bajoran Engineer

Does your local model actually know new Python? A tactical guide

Small models can do real work on a 16 GB laptop, but they're convincing long before they're correct. Here's how to check what a local model really knows, with code you can rerun.

By Dawn Wages 3 min read

Title card: Does your local model actually know new Python?

Small models now do real work on ordinary laptops. The catch is that they're convincing long before they're correct. Here's a practical way to find out what a local model actually knows, based on testing Satyrn AI's Python 3.15 fine-tune against its base model.

Code: github.com/dawnwages/py315-check

Slides with speaker notes (PDF): ai-our-way-slides-and-notes.pdf

#1. Test what the model can't have memorized

Ask about features newer than most training data. Python 3.15 is ideal: lazy imports, * in comprehensions, the frozendict built-in, closed TypedDicts and the new sampling profiler. A model trained on last year's Python can't fake those.

#2. Grade by running, not reading

A model only gets credit if its answer does all three:

  1. Uses the feature, checked with a regex per task.
  2. Runs on Python 3.15.
  3. Prints the right answer.

The first check matters most. Asked to flatten a list with * in a comprehension, a base 9B model wrote this every time:

python 
print([*[[1, 2], [3], [4, 5]]])
# [[1, 2], [3], [4, 5]]

It runs, it uses a star, and it's wrong. Without the feature check, an answer like this would pass.

#3. Change one thing at a time

Compare a base model with its fine-tune on the same laptop, quantization, server and settings. On an M1 with 16 GB:

text 
Python 3.15 feature        Base 9B   Satyrn 9B
frozendict built-in        0/8       4/8
closed TypedDict           0/8       4/8
* in comprehensions        0/8       3/8
lazy import                1/8       1/8
sampling profiler          0/8       0/8
total                      1/40      12/40

Generation speed barely moved: 4.8 vs 4.6 tokens per second.

#4. Read the failures, not just the score

How a model is wrong tells you what data to add next.

  • Confidently invented: the base model wrote from types import FrozenDict, which doesn't exist. That gives you nothing to fix.
  • Nearly right: the fine-tune wrote Movie.__closed, and Python 3.15's error message replied "Did you mean __closed__?" That's one underscore pair from a pass.
  • Didn't move at all: lazy imports and the profiler scored the same for both models. That's the next training-data gap to fill.

#5. Know when you need a bigger test

Forty regex-graded answers is a flashlight, not a lighthouse. For real claims, use stricter rules like Satyrn Evals: save every transcript before grading, let an oracle test decide, report real denominators, and pre-register the comparison. With that setup, the Satyrn Engine delivered passing builds on 16 of 24 medium tasks, against 2 of 24 for the Pi agent on its own. The bare agent often built a working change and then never handed it over.

#6. Keep routine work local, on purpose

16 GB is the common developer laptop. Routine Python work (refactoring, tests, debugging) can stay on it, and you send the big problems to a hosted model when you choose to. Your code stays on hardware you control, and when something breaks you can read the code that did it.

#Try it, then compare notes

  • Run the check against any OpenAI-compatible server (Ollama, LM Studio, llama.cpp): uv run check.py --label mymodel --url http://127.0.0.1:11434
  • Help with training data for lazy imports and the profiler in satyrn-models.
  • Post your results on the Satyrn Discord.

Satyrn AI is community-owned, the way Python and Jupyter are. No single vendor owns the roadmap. AI our way: from us, for us.

If this post helped, you can support my open source work on Ko-fi.