- Found
- Violations were common while tasks often still appeared to complete. The File Read check (not inspecting required files, or reading beyond the required set) flagged up to 98.50% of runs on system-interaction tasks and 21.67% to 33.33% on code generation. Environment changes beyond the task reached 53.00% (OpenHands with Gemini-2.5-Flash, system interaction). Unrequested changes to the given code or docstrings reached up to 59.33% in code generation (the abstract says 50.5%), and unrequested test or draft files appeared in up to 79% of system-interaction runs. The authors tie the differences to the agent framework and its tools more than to the model. Adding rules to the prompt gave inconsistent results. Code-KIT cut the average misbehavior rate with Gemini-2.5-Flash from 12.77% to 6.11% (52%) for smolagents and from 18.19% to 12.31% (32%) for OpenHands; runs judged not to meet the task goal fell from 32.11% to 23.89% and from 40.06% to 24.78%.
- How
- A benchmark of 7.4K cases built from 115 real GitHub repositories (function-completion tasks from the DevEval benchmark), covering code generation, file management (delete, add, move) and system interaction (execute, test), with 11 automatic checks for boundary violations such as changing unrelated files, creating unrequested files, altering the environment or not meeting the task goal. Four agent frameworks (OpenHands, smolagents, Agno, Claude Code) were tested with GPT-5, Gemini-2.5-Flash and Qwen3-Coder-30B (Claude Code with Claude Sonnet 4.5) on a shared sample: 200 cases per code-generation level, 200 file-management and 200 system-interaction cases. Four checks are scored by an AI judge (Gemini-3-Flash) whose labels were compared with three human annotators on 200 sampled runs. The mitigation, Code-KIT, adds guidelines and five purpose-built tools chosen per task by an AI selector; it was tested on smolagents and OpenHands with Gemini-2.5-Flash (Qwen3-Coder-30B in the appendix).
- Limits
- The authors say the benchmark does not cover failures such as multi-turn vulnerability injection or unauthorized data exfiltration, that Code-KIT will need different tool designs for longer, open-ended work, and that agents can bypass its tools through native editors or shell commands. Judge agreement with humans on 200 sampled runs was 96.5% for File Read, 72.0% for Instruction Following, 92.5% for Environment Integrity and 93.5% for Resource Exhaustion, but chance-corrected agreement (Cohen's kappa) was 0.85, 0.37, -0.01 and -0.02; the authors attribute the near-zero values to rare positives in two of the three task families. By design, the File Read check also counts reading more than a set list of files, and all checks count departures from the task whether or not they cause harm. Code-KIT was tested on two of the four frameworks and mainly one model. Results come from sampled cases built from a single source benchmark.