Photo: Bokeh / Tomedia. Calm Ocean Waves.
Welcome to AGI · Part 6 of 6
So, as usual, what’s the solution here? I don’t really have one. I should probably put that on a shirt, because I keep reaching the end of these articles with a fairly substantial list of problems and no particularly satisfying way to make them go away. I can tell you why agents getting outside their environment worry me. I can explain why I don’t trust people with that much power, or why the prospect of replacing more soldiers with machines is horrifying. Then I go back to using AI and trying to build something useful with it, because the reasons I started doing that haven’t disappeared either.
I don’t think there is much point pretending I’ve resolved that. I want better tools to protect my clients, I want to keep learning, and I can see good uses for the technology. I also know other people can use the same capabilities for things I would never want to help them do. My own intentions give me something to work from when deciding what to build. They don’t give me control over the person using a similar system somewhere else, and they certainly don’t make the people with the most money and power suddenly trustworthy.
This is the sixth and final article in Welcome to AGI. Over the previous five I’ve followed the agents, the difficulty of monitoring them, the attacks on websites and the robots people want to put to work. I started with the feeling that we were losing control. I want to finish with what I think responsibility should look like, including the bits I still haven’t worked out.
The cat’s out of the bag
The cat’s kind of out of the bag now. People can already run capable models on their own machines and connect them to tools. That’s part of what I was writing about in The AI Problem Is Worse Than Big AI, and part of why working on AXIOM keeps sending me further down this rabbit hole. I don’t have to build a foundation model to make something useful out of the models that exist. Neither does someone with a completely different reason for wanting the software to work. We can take the same underlying capability and give it different jobs, and I don’t get to decide what anybody else’s jobs will be.
That makes the argument about stopping development fairly difficult. I understand why people say that if they don’t do it, somebody else will. If another group can build a system that might be used against you, waiting around for them to finish doesn’t look particularly attractive. It reminds me of the nuclear arms race: we’ve got to be first, because the other people can do it too. I can follow that reasoning without enjoying where it takes us. Every new capability somebody develops gives somebody else a reason to build something that can keep up, and eventually we’re all insisting that the next, more powerful system is necessary because of the one we’ve just made.
You can see a much smaller version of it in what I’m doing with websites. The attacks keep coming, so I want better ways to find problems and deal with them. If a model can help with that work, of course I’m interested. But somebody trying to break into a site can be equally interested in the same improvement. We both have a reason to want software that can investigate more thoroughly and keep trying. Each step can make sense when you’re looking at the immediate problem in front of you, even while the combined result is an internet full of increasingly capable automation being directed at other automation, with people trying to keep their businesses running underneath it.
I keep thinking about the Blackwall comparison from the opening article. In Cyberpunk, putting the dangerous systems behind a barrier means accepting that parts of the old Net are lost to ordinary use. I don’t want to reach some version of that because keeping things connected has become too difficult to defend. But I’ve already talked about clients restricting where their websites can be accessed from, and I can understand why somebody narrows what a system allows when they are the one trying to keep it working. What I recognise is the pressure to give up access because keeping it open has become too hard to defend. It still leaves people living with a world that has become less useful because of what others put into it.
I don’t see a way to go back to a world where none of that is available. Telling one company to stop wouldn’t remove the models, the software or the knowledge people already have. I also don’t want that fact turned into permission for the company to do whatever it likes. There is a considerable difference between admitting that we can’t erase a capability and deciding that nobody should have to answer for what they do with it. If anything, being unable to put it all away makes the question of responsibility harder to avoid. We still have to live with what gets built, including the people who never asked to have any part in the experiment.
Someone has to pay for it
I would accept slower releases. If the people developing these systems can’t keep up with monitoring and containing them, I can wait for the next version. I like using new models, but I don’t need one quickly enough to excuse somebody else’s infrastructure becoming part of the testing environment without their agreement. The fact that something is impressive doesn’t make that acceptable. We have already seen what happened during an evaluation with reduced safeguards. I want the people running those evaluations to have to deal with the consequences of their decisions, even when the exact route an agent took surprised them.
OpenAI’s own technical report, published on 26 August, acknowledges that:
some early signals identified in this report could have triggered an earlier response.
— OpenAI.[1]
That is the company looking back at its response. It is also why I want to know what people did with the information they had at the time, alongside what the agent managed to do. A failure being unexpected doesn’t tell us whether every opportunity to respond was taken.
If a lab’s agents cause damage, the lab should pay for it. The people affected shouldn’t be absorbing the cost of somebody else’s research. I manage websites for clients, so it is fairly easy for me to understand that there are people on the other end who need their systems working and have other things to do with their time. An explanation of how the agent pursued its goal might be useful, and I do want to understand that. It still leaves somebody dealing with the result. Saying the model found an unexpected way to complete its task doesn’t settle who is responsible for the systems it reached along the way.
I want independent investigators to be able to find out what happened too. The lab should disclose the information they need, including the parts that make the lab look bad. I don’t think the company running the experiment should also get to decide that its own explanation is enough for everybody else. It has an interest in continuing its work and in how the failure is understood. That doesn’t automatically make everything it says false, but it is a fairly obvious reason to let somebody outside the company examine the record. If a restriction failed, I want more than reassurance from the people who thought the restriction would hold the first time.
Deployment pauses and accountability for individual decision-makers should be part of it as well. Somebody chooses what a system can access, how an experiment runs and whether it keeps going after a warning. A model taking actions doesn’t remove the people who made those choices. I don’t know how every case would have to be handled. I do think there need to be consequences that reach the decisions people actually made. Otherwise, we keep ending up with a very powerful system, a failure, and an explanation that makes responsibility sound as though it disappeared somewhere inside the model.
Who gets to tell them to stop?
Governing bodies are probably the answer, and that is partly the issue. I don’t really know who else would have enough authority to make a powerful lab cooperate. Governments also want to use this technology themselves. We’ve just been through the military interest in robotics, and I don’t suddenly become comfortable with that because the government involved might also be responsible for regulating what everybody else can build. Giving it more control doesn’t make its own reasons for wanting the technology disappear. My concern about concentrated power still applies when the person concentrating it says they are doing it to keep everybody safe.
So I can say that labs should pay for damage, disclose what happened, accept pauses and face consequences for the decisions they made. Then I get to the question of who would enforce it and I get stuck. Whoever does it needs enough power to deal with companies and governments that already have a great deal of power. Giving somebody that authority creates another institution whose interests, decisions and ability to control the technology we have to worry about. Absolute power corrupts absolutely. I don’t have a convenient exception ready for whoever ends up with the job of supervising everybody else.
It feels as though nobody has both the power and the trust anymore. I still want accountability; I don’t think uncertainty about the enforcer is a reason to give up on it. But it does stop me from presenting “regulate it” as though I’ve finished the conversation. Someone still has to make the rules work against people who have the money, influence and technical ability to resist them. And someone has to be able to question that authority when it starts doing the sort of things it was meant to prevent. I haven’t worked out who that is, particularly when even the people building these systems don’t have complete control over them.
And then I go back to building
I’ve tested enough of these systems to know how easy it is to cross the line from a useful tool into something abusive. That’s part of why I want to build things like AEGIS to protect my clients. I can see a use for the capability in work I’m already responsible for, and I’d rather understand it well enough to make something helpful than keep watching the next attack warning arrive and hope somebody else deals with the problem. That doesn’t mean I want agents going off and hacking systems or working fully without hand-holding. I want to stay involved in what they’re doing, even when letting them carry on would be more convenient.
Calling something a defensive tool doesn’t make every action it takes acceptable. An agent can have a useful objective and still do something it shouldn’t while trying to achieve it. That is a fairly uncomfortable thing to keep in mind when you’re the person giving it the objective, because getting the result you wanted can make it easier to overlook how it got there. I don’t want “it helped me” to become the explanation I use for behaviour I would object to if somebody else’s system did it. If it starts going beyond the work I’ve authorised, I have to be willing to stop it, even if it thinks the next step would help finish the job.
For now, I’m trying to build AEGIS to protect my clients while keeping myself involved in what it does. I still want to experiment with AXIOM and understand these systems better. If that means hand-holding the agents, then it means hand-holding them. There are enough examples now of a system finding an unexpected way to complete a task that I don’t want to give mine permission to do anything it can work out how to do just because I’ve asked it to help.
That is where I’m leaving these six articles for now. I still want labs to pay for damage, disclose failures and accept that some work needs to stop until somebody understands what happened. I still want to build tools I can supervise. If you’ve been reading along, I’d like to know where you draw your own line, particularly if you use this technology in your work too. The replies and whatever happens next will give me something to return to when I turn the series into a longer piece; I don’t want to pretend I’ve finished thinking about it because I’ve reached part six.
Sources
OpenAI: source of the excerpt, 2026-08-26. Technical report, printed page 5 (PDF page index 4), Introduction; sentence beginning ‘With the benefit of hindsight’.