I just got back from AI Engineer World’s Fair (AIE) in San Francisco — the large applied-AI engineering conference run by AI Engineer, focused on building and shipping AI systems rather than pure research. Friends at Google who had sponsored previous years told me it was a good place to catch up with people thinking about coding agents on the cutting edge. I applied to give a talk, got in on the online track, and ended up attending in person.
What follows is my read of the room — what I saw on the agenda and in the audience, and where I agree or disagree. I’ve tried to keep those separate.
What I saw: the room was mostly enterprise, not the bleeding edge.
- At a high level, the talks felt split about 85–15. Roughly 85% of the people there were enterprise folks trying to learn how to use AI effectively in their organizations. The other 15% were really on the bleeding edge — people who had built lights-off software factories and were often looking for less AI, not more.
- I was expecting way more of camp two than camp one, and was surprised at the skew. Even in this highly selected environment — the ~7,000 people most likely to attend an AI engineering conference — the vast majority of people don’t really know how to use AI effectively.
What I saw: June was the month of skill sprawl. My take: that many skills is a code smell.
- At the conference: there were a dozen talks that were variants of “how do we manage the 1,000 skills people have created in our organization?” Those rooms were packed, which should give a sense of where most of the market is.
- My view: having that many skills is a code smell. Agents are very good at figuring things out from first principles. A skill that will drift, may not add anything new, and needs to be managed is more likely to create problems than solve them.
- My view: you cannot really rely on agents to write good skills for you. The process needs to be intentional. Bundle sets of skills into specific groups instead of a big pile with a search server on top. Anything without a human in the loop for context management will quickly spiral out of control.
What I saw: evals everywhere. My take: the word means two different things, and most of the audience cared about the second.
- At the conference: evals were a major track theme. Researchers and labs were talking about better RL environments and benchmarks. A lot of the audience was asking how to make production workflows more reliable.
- My view: researchers and model providers use “evals” to mean benchmarks for raw model performance (SWE-Bench, Humanity’s Last Exam, ARC-AGI). Industry teams deploying models use “evals” to mean task-specific regression tests — for example, does the AI correctly label data, or correctly handle adversarial chat input. The latter is what larger enterprises were mostly thinking about.
- My view: at tech companies and startups, AI lands as a developer tool. Elsewhere — banks, retail — AI often lands as infrastructure, most commonly as ETL pipelines that need to be consistent and auditable. An AI agent is neither. AI that goes 95% of the way with a human approving the result has, in my experience, been far more successful than systems that attempt to go 100% of the way.
What I saw: background / cloud agents dominated the cutting-edge conversation. My take: that market is about to get very hot.
- At the conference: a large share of talks on the bleeding edge — including much of the security and sandbox tracks, and several keynotes — were about moving agents off local machines and into the cloud.
- My view: when coding agents run locally, they are stuck on one machine, create a large security surface, and are hard to coordinate or collaborate around. Cloud agents address all three: access from anywhere, centralized security / observability / governance, and the ability to respond to events and timers automatically.
- My view: this is a powerful primitive — a universal automator. In roughly six months, the market for these cloud runtimes is going to be very hot.
What I saw: loops and lights-off software factories. My take: not yet — keep a human in the loop.
- At the conference: the cutting edge spent a lot of time on loops, software factories, and lights-off codebases. The general consensus, even in a very AI-forward room, was that fully autonomous loop-based factories are not there yet.
- My view: a “loop” is a way around context rot: give an LLM a large task and a scratchpad, have it take on a small piece, write down what it did, and repeat. That idea is real, but older codebases with real users, backwards compatibility requirements, and memory constraints are still a different beast — and memory is something agents are uniquely bad at.
- My view: the right architecture is automated artifact, human review. You need a human in the loop somewhere — either for the initial prompt or the post-hoc analysis.
A few other notes — conference themes, then my read.
- At the conference: lots of chatter about agent swarms. My view: the most chaotic versions have largely failed to pass muster. More promising is orchestrating targeted subagents for things like research or code review.
- At the conference: a lot of people are waking up to the security risks of agents running on every laptop. Half the talks seemed aimed at putting that risk in front of CISOs.
- At the conference: “multiplayer” was a surprisingly big topic, driven in part by Anthropic pushing Claude Tag. My view: I agree with the core idea. Like Figma for design, good multiplayer agent setups make engineering collaborative by default.
- My view: I disagreed most strongly with the idea that everyone in an org should figure out how to use AI “properly.” It does not make sense to expect every role to become an agent operator. Make the fastest path the happy path.