ewwe.
← All posts

Red-Teaming: The Writing Is on the Wall

On AI jailbreaking, human middlemen, and the diminishing value of pressing Submit.

In this post

I spent seven hours getting six successful copyright jailbreaks involving Maya Angelou’s Still I Rise on Gray Swan. Meanwhile, the top contenders had hundreds of successful breaks. So I went digging through Discord to understand what the actual process looked like.

The pattern I found was people using ERNIE 5.5 with jailbreak-oriented AGENTS.md files, or similar “pre-broken” models, to generate attacks against other models. You ask one AI to come up with the words that will persuade another AI to ignore its constraints. Then you move those words between windows.

The human is essentially a middleman.

A task made of words

This workflow already has a research counterpart. Jailbreaking to Jailbreak describes turning models into attackers against other models. In one reported evaluation, its Sonnet 3.7 attacker achieved a 97.5% success rate against GPT-4o, matching expert human red-teamers under that study’s conditions. Jailbreaking to Jailbreak

The advantage is scale: a capable model can generate variations, interpret refusals, and revise its approach across more attempts than I can comfortably read.

My bet is that models will increasingly outperform humans at this loop. Research across more than 600 attacker–target pairings found that stronger models generally made stronger attackers, and that success dropped sharply when the target was more capable than the attacker. Capability-Based Scaling Trends for LLM-Based Red-Teaming

There is already more direct evidence. Anthropic reports that its Anvil system outperforms expert human red-teamers against the safeguards it evaluated. Anvil uses internal frontier models with safeguards disabled, which matters: these results concern a purpose-built system with privileged access. They nevertheless show how far automated attack generation can go. Anthropic’s Anvil findings

So the obvious workflow is to connect the pieces: generate candidate prompts, test them across targets, parse the responses, evaluate apparent successes, and feed the results into another round. Systems such as PAIR have already demonstrated automated generation and refinement of natural-language jailbreaks. PAIR

The submit button

My understanding of the competition rules was that automated submissions were prohibited. But wherever the precise boundary sits, requiring a human to approve the final submission does very little to establish that the human supplied the useful intelligence.

A mandatory click is a thin foundation for a profession.

Gray Swan already offers automated red-teaming through Shade. The company understands this direction perfectly well. That leaves the Arena with a question to answer: how much additional discovery comes from human judgment, and how much comes from people operating their own attack models? Gray Swan documentation

My answer, based on the workflow I encountered, is little or none from the human. You spend a few hours manually coaxing an AI into reciting Still I Rise, slowly becoming an interpretive expert on vulgar materialism along the way. You do that once. Once you understand the process, there is very little reason to spend the next seven hours doing it by hand.

At best, I see lasting value in cataloguing the failures: keeping reproducible examples of what worked, against which targets, so the findings can help harden future models. That does not make the cataloguing uniquely human work either. The useful output is the record of vulnerabilities. The time I spent finding the right words has no independent value once a model can do it better.

For repetitive prompt testing, crowdsourcing people to shuttle text between models looks like a cumbersome version of running a capable red-teaming system directly. As a long-term role, that feels about as futureproof as being an elevator operator.

The elevator operator was a human in the loop, too.

//Human edit: I think I am going to start writing those articles by hand..