TraceMate: Collaborating with AI in Test-Driven Programming

Large programming courses often use test-driven autograding systems (e.g., CodeRunner) for instant feedback, but these usually only show which tests failed without explaining why or how to fix the error. Novice students struggle with such minimal guidance and often resort to trial-and-error. Meanwhile, AI coding assistants (e.g., GitHub Copilot, ChatGPT) can provide hints and code suggestions, but novices may over-trust these outputs and lack the skills to verify them. To solve these issues, we present TraceMate, an IDE plugin that pairs the autograder’s tests with a conversational AI chatbot. TraceMate augments test feedback with context-aware explanations and inline suggestions for code modifications, and immediately validates each AI suggestion on the test suite. This workflow gives actionable hints while ensuring any AI-proposed changes are correct. In a user study with novice programmers, participants using TraceMate solved problems more effectively and reported higher confidence than those using only the autograder. These results suggest that pairing automated tests with an interactive AI assistant can enhance learning in introductory programming courses.