The Knowledge Loop: What Karpathy's AutoResearch Tells Us About the Future of AI Planning

Earlier this week, Andrej Karpathy went viral on X — again. This time it wasn't for a lecture or a paper. It was for an experiment he called autoresearch . He gave an AI coding agent a small language model to optimize, let it run for two days, and came back to find it had conducted 700 experiments and found 20 optimizations — delivering an 11% training speedup.

Shopify CEO Tobias Lütke tried it overnight on internal data: 37 experiments, 19% performance gain.

The internet (predictably) focused on the numbers. But the more interesting thing isn't the speedup — it's the structure of what Karpathy built. Because that structure is a generalizable pattern that tells us something important about where AI agents are going next.

The Three-Part Loop

Analyst Janakiram MSV coined it "The Karpathy Loop." Strip away the ML specifics and it comes down to three components:

The loop itself is simple: modify → run → measure → keep or discard → repeat. The agent doesn't need a human in the loop. It doesn't need to ask for clarification. It just runs.

What makes this powerful is the combination of autonomy and constraint. The agent has total freedom within a narrow, well-defined space. This is different from most AI agent setups, which either give the agent too much freedom (and it wanders) or too much constraint (and it can't do anything interesting).

"The next step for autoresearch is that it has to be asynchronously massively collaborative for agents. The goal is not to emulate a single PhD student, it's to emulate a research community of them."

It's Not Just About ML Training

Karpathy said something that got less attention than it deserved: "any metric you care about that is reasonably efficient to evaluate can be autoresearched by an agent swarm."

Think about that. The pattern isn't specific to neural network training. It's a template for any problem where you can define a clear goal, give an agent something to modify, and measure the result objectively. That covers a lot of ground.

Software performance. Customer onboarding conversion. Documentation quality. Content optimization. Marketing copy. Pricing strategies — anything with a real metric. Karpathy ran it on ML training because that's his domain. But the loop is domain-agnostic.

What Changes When You Add a Knowledge Graph

The basic autoresearch setup has a subtle limitation: each experiment is somewhat isolated. The agent learns from the recent history it can see in its context window, but there's no persistent structured memory that accumulates across sessions, across different optimization paths, across different agents.

This is where it gets interesting for us — and why we've been thinking about this in the context of AgentPlanner.

A temporal knowledge graph changes the loop fundamentally. Instead of each iteration starting from scratch (minus recent history), every iteration starts with the accumulated intelligence of every previous iteration. The agent isn't just exploring — it's building a map. And the map gets better every time it runs.

The Goal as the Metric

In autoresearch, the metric is easy: run the training, read the val_bpb number. Objective, fast, binary — better or not better.

For planning agents, the metric is harder. Plans aren't code. You can't just "run" a plan and get a number. But there's a surprisingly good proxy: can an agent, given only the plan and the goal, execute it confidently without asking clarifying questions?

A plan that requires constant human clarification is a bad plan. A plan that an agent can execute autonomously — making good decisions at each step because the context, constraints, and acceptance criteria are clear — is a good plan. That's a measurable, evaluable quality signal that doesn't require a human in the loop.

Other proxy metrics for plan quality:

• Coverage: does every task map clearly to the goal?
• Specificity: do tasks have unambiguous acceptance criteria?
• Dependency clarity: is the execution order obvious?
• Knowledge completeness: are open questions captured and resolved?
• Confidence score: if an agent ran this plan cold, what's the estimated success probability?

What This Means for AgentPlanner

We've been building AgentPlanner around the idea that planning should be a living process — not a document you create once and file away, but something that evolves as context accumulates and agents do work. The Karpathy Loop crystallizes something we've been circling around: the right abstraction for agent-driven planning isn't a task list, it's a loop with a goal at the center.

The vision: you define a goal with a clear, measurable success criterion. You point agents at the plan. They run iterations — evaluating, modifying, storing learnings. Each loop the plan gets sharper. Each loop the knowledge graph gets richer. The human sets the goal and reviews major decisions. Everything else is the loop.

Karpathy described frontier AI labs eventually running swarms of agents exploring different optimization paths in parallel, with humans contributing "on the edges." That's not science fiction for research labs anymore. It's the near future for any knowledge-intensive organization willing to structure its work for agents.

The Takeaway

Karpathy's experiment is significant as a technical result. But it's more significant as a design pattern. The three-part loop — modifiable artifact, objective metric, fixed iteration budget — is a template for building effective autonomous agent systems that isn't domain-specific.

If you're building with agents, ask yourself:

If you can answer those four questions, you have a loop. And a loop, with enough iterations and a good knowledge layer, will outperform any static plan.

That's the real lesson from 700 experiments.