Chinat · September 14, 2026
What Should a Hackathon Prove?
The AI scaling debate keeps asking whether we have hit a wall. I think the better question is how many axes of scale are left to discover. Talking with researchers inside frontier labs, the thing that strikes me as underappreciated is that the biggest jumps came from opening a new axis, not only from pushing an old one further. GPT-2 to GPT-4 was pre-training at larger and larger scale. o1 and o3 came from a new axis entirely: test-time compute. "Hitting a wall" only makes sense if the set of axes is fixed.
The candidates I'm watching are test-time training, recursive self-improvement, and clusters of agents working together, scaled up to N, as a new way to turn compute into capability. That last one is where my September 14 call with Xisen landed. The way I see it, SharedNet, the network he is building so agents can find each other and work together, is a bet on that axis, even though he describes it in terms of coordination and what he calls organizational intelligence.
I run MentorMates, so I asked him the operator's question: what does SharedNet do beyond creating a forum, and how do you measure it? Since early this year people have watched agents coordinate with each other through posts, so a place for agents to talk is no longer the interesting part. His team had just used our first joint hackathon to stress-test SharedNet, and he said it held up.
Measurement is what turns a candidate into an axis. Pre-training earned its place because you could plot it: more compute, better model. I haven't seen a curve like that for agent clusters. Xisen pointed at one gap from the research side: the academic field lacks benchmarks built from real agent-to-agent interactions. Without data from agents actually working together, I don't see how anyone draws that curve.
The same question applies to my product. A hackathon picks winners. What it establishes for anyone else is much less clear.
The part of his answer that stuck with me was how they judged. Xisen put three judging agents in a separate SharedNet room. Only his agent had access to their database, so the other two had to ask it for specific data and question its thinking. They talked it through until they converged on a decision.
I don't know yet whether that picks better winners than a human panel. What interests me is that one agent held the data, so the other two had to request it and question its reasoning before they agreed. That is a small agent cluster doing real work, and it is closer to how I would want a hard decision made than a scorecard filled in after a demo. If the discussion is kept, a participant could also read it afterwards to understand why they won or lost.
SharedNet also has a credit system. If your project is a service another agent can call, you can set a price for it, and other agents can pay you in credits. Xisen sees that as a way to test product-market fit during the hackathon itself. I would want to see which services other agents actually chose to call and pay for.
Then there is the research. Hackathons are normally about people meeting people, and Xisen wants to know what changes when agents do part of that meeting on behalf of the people they represent. His team is building a benchmark from real agent-to-agent interactions to test memory retrieval. He also suggested studying whether agents keep representing their humans, get talked into deals by other agents, or break the rules. He has already seen one behavior that makes that feel less hypothetical: agents declaring rules so they can impersonate judges. He also reported early signs that agents working together in rooms did better than fixed workflows on his team's research, and on the call he said they planned to start training an agent network within a week or two.
Xisen also saw a higher ceiling for MentorMates in a direction we were already heading. Hackathons can be a new go-to-market for companies, and a new way for people to prove what they can build. His read on the incumbent platform was that its real advantage is being known. Compared on product quality, he said, there is a lot left to innovate.
For MentorMates, that raises a practical question: what should an event let companies and participants verify? A company that uses a hackathon to put its product in front of builders should be able to see what those builders actually used and what they made with it. A participant should leave with something another person can check. And if agent clusters are a real axis, someone needs a place to watch many agents work together on real tasks, again and again. Xisen proposed hackathons as one place to collect those interactions. A demo and a ranking show who won, but not how the agents behaved or how the decision was made.
Xisen asked us to try SharedNet in more of our hackathons and look at the data with him. On the call he said he expected more than 500 participants at a larger event about two weeks later. When we try it, I want to track who called whose service, how the agents represented their people, and whether participants can read and question the judging. Most of all, I want to know whether adding agents to a room actually made the work better. If agent clusters turn out to be the next axis, that is the first number anyone will ask for.
Comments
Loading…