AI Agent Architect

John Yang Portfolio

PhD student at Stanford

PhD student at Stanford (work began at Princeton); created SWE-bench and SWE-agent, which established "agent-computer interfaces" — a structured set of tools (file viewer, search, edit commands) purpose-built for how a language-model agent, not a human, interacts with a codebase.

Agent-Computer InterfacesCoding AgentsTool UseBenchmarking

Location

Somewhere on planet Earth, most likely near a laptop.

Companies

Stanford UniversityMeta

Credentials

  • Princeton University
  • Stanford University

Skills

Agent-Computer InterfacesCoding AgentsTool UseBenchmarking

Notable

  • SWE-bench: foundational benchmark for code agents
  • SWE-agent: agent-computer interface design

What makes it work

A breakdown of the choices that make this portfolio stand out.

1

A named interface layer designed specifically for agents, not humans

SWE-agent's "agent-computer interface" concept treats the tools an agent uses (not just its reasoning) as a first-class design problem, recognizing that interfaces built for human developers are often the wrong shape for a language model to use reliably.

2

A benchmark built to resist gaming, then hardened further

SWE-bench Verified and the multimodal/multilingual variants show iterating on a benchmark's rigor over time as agents got better at exploiting the original version's weaknesses, rather than treating the first release as final.

3

A tight suite of complementary tools around one research question

InterCode, SWE-bench, SWE-agent, and SWE-smith each address a different piece of the same problem — how coding agents should be evaluated and interfaced with — rather than being unrelated one-off projects.

What AI Agent Architects can take from this

Specific, actionable tips to apply to your own portfolio — no generic advice.

  • Treat the interface between your agent and its environment as its own design problem, separate from the agent's reasoning — the wrong tool shape can break an otherwise good agent.

  • If you publish a benchmark, expect it to get gamed, and plan to release harder or corrected versions rather than treating v1 as permanent.

  • Build a small suite of complementary tools around one specific research question instead of one-off unrelated projects — it makes your body of work read as a coherent research program.

Projects

Almost everything here is about measuring and training coding agents: a benchmark from real GitHub issues, an agent built for it, and a pipeline that produces training data for the next agent. Each is released as data, code and a paper together.

ResearchadvancedarXiv Oct 2023; ICLR 2024 (Oral)

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

PythonGitHub issues and pull requestsDocker test harnessSWE-Llama

An evaluation set of 2,294 real software-engineering problems taken from GitHub issues and their merged pull requests across 12 popular Python repositories. The model gets a codebase and an issue and must edit the code so the repository's tests pass, which often means coordinated changes across several files.

It set the standard way to measure coding agents: real repositories, real tests, no multiple choice. Keeping a benchmark useful as models improved shows lasting ownership.

  • 2,294 task instances from 12 Python repos

  • Pass/fail graded by the repository's own tests

  • Best model at launch: 1.96% resolved

Open SourceadvancedarXiv May 2024; NeurIPS 2024

SWE-agent

PythonAgent-computer interfaceYAML configDockerGPT-4o / Claude

An agent that takes a GitHub issue and tries to fix it on its own. Its key idea is the agent-computer interface (ACI): file viewing, search, editing and test commands designed for how a language model works rather than a human, and the paper shows that interface design alone changes agent performance.

It showed that the tool layer the model sees matters as much as the model, and then the team replaced its own system with a smaller one when the evidence supported it.

  • Agent-computer interface: tools designed for LMs, not humans

  • 12.5% on SWE-bench and 87.7% on HumanEvalFix at publication

  • Superseded by mini-swe-agent: 65% on Verified in ~100 lines

ResearchadvancedReleased Apr 30, 2025

SWE-smith: Scaling Data for Software Engineering Agents

PythonSynthetic bug generationDockerFine-tuningSWE-agent

A pipeline that generates hundreds to thousands of software-engineering task instances for any GitHub repository by creating bugs that can be checked against tests. The team used it to build a training set and fine-tune their own model for SWE-agent.

Having built the benchmark, he then built the data supply to train agents for it, a path from evaluation to training that few researchers complete.

  • Generates task instances for any GitHub repo, with a tutorial claiming 100 in 10 minutes

  • 50k+ instances across 128 repos

  • Trained SWE-agent-LM-32B: 40% on SWE-bench Verified

Ready to build your portfolio?

Follow the AI Agent Architect roadmap — skills, projects, and timeline to get hired.

AI Agent Architect Roadmap