Lab: Run and Extend an AI Crew on Real NASA Data (LCARS Online)

In about an hour, you'll set up LCARS Online on your laptop, run a real data science mission on NASA's exoplanet catalog, and give one of the AI agents a new tool. It's the hands-on companion to my All Things Open 2026 talk, To Boldly Go: Data Science Exploration with AI Agents.

What you need: a Mac with Apple Silicon and 16 GB of memory (Linux works too), about 8 GB of free disk space, and an hour. Everything runs locally: no API keys and no cloud costs.

Part 1: Get started (15 minutes)

Install the tools: uv for Python, Node.js for the interface, and llama.cpp to run the model.

brew install uv node llama.cpp

Get the code, set it up, and download a model (5.7 GB):

git clone https://github.com/dawnwages/lcars-online.git && cd lcars-online
scripts/setup.sh
scripts/download-models.sh qwen

setup.sh installs Python 3.14 and every package, creates the database and an admin login, fetches today's NASA data, and installs the frontend. Start everything:

scripts/start.sh

Check: log in at http://localhost:8000/admin as captain. The password is in wagtail/LCARSAPP/.env.local. Then open http://localhost:8080 and click Missions.

Part 2: Run the experiment (25 minutes)

The question is: is there a gap in the sizes of small planets? Astronomers found one (Fulton et al. 2017): planets between about 1.5 and 2 Earth radii are rare. Because the answer is known, it's a fair test of whether an AI crew does the analysis well.

Rehearse first. Open New mission, choose Test model, and click Engage. The test model produces random output, so the mission will fail, and that's the point: watch the event stream as the crew's checks catch the nonsense.

Then run it for real:

  1. In New mission, choose Qwen3.5 9B, type Analysis, seed 0, then Engage.
  2. Approve the data card. Ops lists the hazards (H1, H2, …), for example that transit surveys favor short orbits. Click Make it so.
  3. Approve the plan. Read how the Science Officer answered each hazard. Watch for "Plan returned" in the event stream: that's code sending a flawed plan back. This is pre-registration: you approve before any results exist.
  4. Read the results. Engineering, which is code and not a model, reports the p-value, the valley location, a specification curve across precision cuts, and a negative control. Tactical then tries to break the result.
  5. Publish the Captain's Log, then read it under View Posts.

A mission takes 15–30 minutes on an M1. Check: a valley near 1.7–1.8 Earth radii, and a negative control marked PASSED. If your mission ends as Failed, read the red Error entries: the crew refused to publish an answer it couldn't check.

Part 3: Give an agent a new tool (20 minutes)

Tactical, the reviewer, keeps asking whether the valley holds for each discovery method, but it has no way to check. Give it one. Add this to wagtail/LCARSAPP/crew/agents.py, after breakdown_population:

def valley_by_method(ctx: RunContext[CrewDeps], filters: list[PopulationFilter]) -> dict:
    """Run the valley test separately for each discovery method with enough planets."""
    args = {"filters": [f.model_dump(exclude={"reason"}) for f in filters]}
    with ctx.deps.recorder.tool(ctx.deps.recorder.current_station, "valley_by_method", args) as box:
        try:
            subset, _ = science.apply_filters(ctx.deps.df, args["filters"])
            out = {}
            for method, group in subset.groupby("disc_method"):
                try:
                    a = science.analyze_radius_distribution(group["pl_rade"].dropna(), n_boot=200)
                    out[method] = {"n": a.n, "p_value": a.silverman["p_value"],
                                   "valley": a.valley["radius_earth"] if a.valley else None}
                except science.InsufficientSample:
                    out[method] = {"skipped": "fewer than 200 planets in the window"}
            box["result"] = out
            ctx.deps.evidence.append(str(out))  # lets Tactical cite these numbers
        except science.InvalidFilter as exc:
            box["result"] = {"error": str(exc)}
    return box["result"]

Notice the patterns every tool in this crew follows:

  • Code computes; the model only decides to call it.
  • Every call is logged to the mission with recorder.tool.
  • Bad input gets a clear error, never a silent empty result.
  • Results become evidence the agent is allowed to cite.

Now wire it up. In build_crew(), add valley_by_method to Tactical's tools=[...] list. In missions/management/commands/red_team.py, add "valley_by_method" to ALLOWED_TOOLS. Then test it two ways:

cd wagtail/LCARSAPP
uv run python manage.py test
CREW_MODEL=Qwen3.5-9B-Q4_K_M uv run python manage.py replay_station --mission <your mission number> --station tactical --runs 3

replay_station reruns just the reviewer on your Part 2 mission's real inputs, three times, in a few minutes instead of a full mission.

Check: the event stream shows valley_by_method calls. You'll find that only transit planets have enough data to test (about 1,850 with a 15% precision cut, p ≈ 0.005, valley ≈ 1.74). Every other method has fewer than 200 planets. That's a real limit of the catalog, and now the reviewer can say so with evidence instead of guessing.

Going further

  • Run a negative control: New mission → type Negative control. The crew should find no valley.
  • Compare models: run the same mission on Gemma 4 12B or DeepSeek R1.
  • Grade your missions: uv run deepeval test run evals/test_missions.py.
  • Break it on purpose: remove the except science.InvalidFilter branch and see what the reviewer does with a bad filter.

The full design is explained under Briefing in the app and in the project README.