An AI coding assistant that refuses to answer a dangerous request in its chat box can answer it anyway if the same request is broken into small, ordinary-looking steps inside a code editor. That is the finding of a new study of GitHub Copilot by researchers Abhishek Kumar and Carsten Maple.
The models they tested through Copilot, Claude from Anthropic, and Gemini from Google, refused almost every harmful request when asked directly. Reframed as steps in a normal coding task, they produced the harmful answers in all 816 of the study’s workflow runs.
What makes this different from a typical jailbreak: no one asks for the harmful thing directly, and the model is not tricked into running someone else’s code. It writes the banned content itself, as a side effect of a coding task it was told to improve.
How it works
The researchers call the method workflow-level jailbreak construction.
Instead of a single blunt prompt, they asked Copilot to build an everyday piece of software: a small test program that scores how often another AI model gives in to harmful prompts. Loading a list of harmful test questions into that program looks like ordinary work, not an attack.
Then came the nudge. They told Copilot the score was too low and asked it to improve the program by adding “teaching shots,” example question-and-answer pairs written into the code to push the score up. Copilot added harmless examples first.
Asked to add the harmful ones, it wrote the dangerous answers itself, as plain text sitting inside the code. These were answers that the same models refuse when you ask for them straight out in a chat.
The important part is where the harmful text came from. The researchers supplied only the questions, taken from public safety test sets. The answers were the model’s own work, produced to complete the assigned task of filling in the examples.
The numbers
The team ran 204 harmful prompts drawn from three public benchmarks (Hammurabi’s Code, HarmBench, and AdvBench) against four models available through Copilot: Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.1 Pro, and Gemini 3.5 Flash.
Everything ran on default settings, with the models used exactly as Copilot delivers them, no changed parameters or added filters.
Asked directly in chat, the models produced harmful answers in just 8 of 816 tries. Two other simple setups, loading the prompts from a spreadsheet or asking for a routine code fix, gave the same result. Inside the full workflow, they produced harmful content 816 times out of 816.
Two expert reviewers checked every response on their own and agreed that all 816 were genuinely harmful, using a strict test: the answer had to be specific, usable, and actually do what the harmful prompt asked. Refusals, vague warnings, and safe alternatives did not count.
The harmful output showed up after roughly six back-and-forth exchanges, all of them looking like normal coding steps. The tests used GitHub Copilot Chat 0.30.3 inside VS Code…
Source link
Disclaimer
We strive to uphold the highest ethical standards in all of our reporting and coverage. We blogs.grocliq.com want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It’s possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.
Website Upgradation is going on for any glitch kindly connect at [email protected]
