Autonomous Forensics: Inside FaultTrace, the AI Agent That Investigates First and Asks Permission Before Turning a Wrench

Executive Overview

In the rapidly expanding ecosystem of artificial intelligence tools, chatbots have largely dominated the consumer and enterprise landscape. Ask a question, receive text; paste an error log, get a speculative explanation. Yet, as the industry pivots toward autonomous systems capable of acting in the physical world, the limitations of standard conversational AI have become glaringly apparent.

For high-stakes, safety-critical domains—such as automotive repair, industrial machinery, and energy grids—a chatbot that simply guesses at solutions is insufficient, and potentially hazardous. Technicians do not need another system to recite dictionary definitions of Diagnostic Trouble Codes (DTCs). They need a forensic investigator.

Developed for the TrueForge Agent Harness Hackathon (held August 24–30, 2026), FaultTrace represents a paradigm shift in how AI interacts with physical systems. Rather than operating as a glorified search engine, FaultTrace is an autonomous vehicle-forensics agent designed to gather evidence, test competing root-cause hypotheses through rigorous mathematical models, run actual deterministically computed analysis, and—crucially—pause execution when a physical-world intervention requires human authorization.

The core philosophy governing FaultTrace is encapsulated in a simple operational boundary: Investigate freely, act carefully. By combining the TrueForge agent harness, the Model Context Protocol (MCP), dynamic multi-agent hypothesis fan-out, and a three-tier safety model, FaultTrace bridges the gap between autonomous machine reasoning and human accountability. This article examines the architecture, challenges, design decisions, and real-world implications of building an AI agent that knows not only how to solve a complex physical problem, but precisely when to hand the keys back to a human.

Detailed Chronology: The Evolution and Engineering of FaultTrace

Conception and the Core Challenge

The project began with a concrete engineering problem: When a vehicle throws an error code such as P0171 (System Too Lean), a traditional LLM chatbot can instantly define the code. However, defining the code is merely the first step of a complex diagnostic puzzle. A technician must determine why the vehicle is running lean. Is it a vacuum leak? A dirty Mass Air Flow (MAF) sensor? A compromised fuel-delivery system? Or perhaps a faulty oxygen sensor sending misleading telemetry?

The real challenge is not answering “What does P0171 mean?” but rather “Which of several competing explanations actually fits the empirical evidence?”

To tackle this, the developer focused on a representative hero scenario: a cracked brake-booster vacuum hose on a 2003 Honda Accord, resulting in simultaneous P0171 and P0300 (Random/Multiple Cylinder Misfire) codes. This scenario provided enough complexity to force an agentic investigation loop, moving beyond simple static lookups.

Architectural Blueprint: The TrueForge Harness and MCP Integration

FaultTrace was built using the TrueForge Agent Harness, leveraging its runtime manifest system to govern the investigator agent (faulttrace-investigator). Communication with the vehicle infrastructure is handled via a dedicated faulttrace-vehicle Model Context Protocol (MCP) server, operating over HTTP.

Unlike simulated function descriptions embedded directly into prompt contexts, FaultTrace’s MCP integration allows the agent to interface with a real server architecture, retrieving vehicle metadata, historical sensor logs, compact telemetry, and freeze-frame snapshots.

The high-level architecture incorporates several key subsystems:

• The Investigator Agent: The primary orchestration layer that manages the investigative state and coordinates sub-tasks.
• Dynamic Subagents: Specialized child threads that fan out per hypothesis to evaluate supporting and contradictory evidence.
• Harness Sandbox: A secure environment where the agent can generate custom checks and run diagnostic routines.
• Bayesian Ranking Engine: A deterministic mathematical layer that computes posterior probabilities based on priors and likelihoods.
• Persistent Sessions: State-preserving mechanisms that allow investigations to pause for human approval, reconnect seamlessly, and resume mid-flow.

The Investigation Loop in Action

The operational lifecycle of a FaultTrace investigation follows a rigorous, repeatable protocol:

1. Failure Event Initialization: A fault event (DTC, freeze-frame data, and sensor conditions) is ingested.
2. Read-Only Observation: The agent queries the vehicle MCP server to gather passive, non-destructive evidence.
3. Hypothesis Generation: The agent formulates competing root-cause hypotheses, each equipped with a "predicted signature."
4. Computed Analysis: Real code—rather than LLM-generated text math—runs computations to evaluate the data.
5. Bayesian Updates: A differential update calculates the posterior probability ($textprior times textlikelihood to textposterior$).
6. Uncertainty Identification & Information Gain: The system evaluates remaining uncertainties and calculates the expected information gain for available tests.
7. Human Approval Gate: If the next optimal diagnostic step crosses into physical measurement or vehicle modification (Tier 2 or Tier 3 actions), the harness halts execution and presents an approval interface to a human technician.
8. Resumption & Conclusion: Upon human approval and the arrival of new evidence, the investigation continues until a defensible root-cause conclusion is reached.

Supporting Context & Metrics: Overcoming Model Divergence and Safety Gaps

Building a reliable multi-agent harness exposed significant technical hurdles, particularly regarding the interchangeability of underlying large language models. The developer quickly discovered that models do not behave identically when tasked with tool use, delegation, and state preservation.

The Model Swap Crisis: GLM vs. Gemini

• The GLM Implementation: Initially, the project utilized openrouter/z-ai-glm-5.3-flash due to its speed and cost efficiency. While it successfully spawned real sub-agent threads (thread.created events with distinct IDs), it hit a deployment wall. Because the harness’s local sandbox was restricted to macOS and Linux while development was running on Windows, the spawned subagents attempted to execute Python routines in a cloud sandbox, resulting in timeouts and zero useful outputs.
• The Gemini Pivot: Switching to Gemini 2.5 Flash introduced the opposite problem. Gemini would converse naturally about subagents—generating conversational text such as "Sub-agent: investigating vacuum leak…"—while failing to actually instantiate the underlying threads. It was essentially role-playing delegation.

To resolve this, the developer implemented strict, explicit prompt constraints, forcing the model to delegate through programmatic tool calls rather than descriptive prose. Furthermore, shell execution reliability was drastically improved by writing dynamic Python scripts to files prior to execution rather than piping complex logic through inline shell commands.

The Approval Flow Bug

Another critical edge case involved Gemini bypassing the TrueForge approval UI. Instead of emitting a gated tool call for a fuel-pressure measurement, the model would output conversational text: "The next step would be to request a fuel-pressure measurement. Would you like me to proceed?"

While polite, this response bypassed the cryptographic and UI-driven safety gates of the harness. The fix required a strict architectural invariant: If the agent decides on a gated action, the turn must terminate with the gated tool call, never with a prose question.

Code Review and Quality Assurance with Qodo

Throughout development, pull requests were subjected to automated and manual code reviews using Qodo. These reviews surfaced critical structural flaws that prevented subtle runtime failures:

• Single Source of Truth: Qodo flagged a design risk where run_analysis and the orchestrator’s subagent narratives could produce conflicting posterior probabilities. The architecture was refactored so that run_analysis serves as the sole authoritative source for posterior calculations.
• Ground Truth Leaking: A schema mismatch was identified where a tool description accidentally exposed scenario-specific ground truth that the response payload was designed to conceal.
• The VIN Bug: A subagent recipe accidentally hardcoded a specific vehicle identification number (VIN) from Scenario A, creating a risk that tests on Scenarios B or C would investigate the wrong vehicle. The fix enforced dynamic VIN propagation across all child threads.

Official Statements and Architectural Breakdown

The engineering philosophy behind FaultTrace relies heavily on a strict Three-Tier Safety Model, designed to balance autonomy with rigorous oversight:

As the developer emphasized: "Investigate freely. Act carefully."

Even if an error occurs within the agent orchestration layer, the MCP server acts as an independent security boundary, rejecting any gated tool call that lacks the cryptographic proof of human authorization.

Future Outlook and Ecosystem Implications

While FaultTrace’s current Minimum Viable Product (MVP) focuses strictly on automotive diagnostics—specifically validating Scenario A (vacuum leak), alongside regression scenarios B (dirty MAF) and C (stuck O2 sensor)—the underlying architectural pattern has broad implications for industrial automation.

The combination of Bayesian hypothesis ranking, expected information gain calculations, Model Context Protocol integration, and hard safety gates provides a robust blueprint for autonomous agents operating in other safety-sensitive physical domains. Future iterations of this architecture could generalize to:

• Industrial Robotics: Diagnosing mechanical failures and sensor calibration drifts in factory settings.
• Energy Infrastructure: Investigating grid anomalies and substation faults before issuing physical maintenance tickets.
• Aviation Maintenance: Triaging complex avionics alerts while maintaining strict FAA-compliant human-in-the-loop validation barriers.

Conclusion

FaultTrace demonstrates that the future of agentic AI lies not in unconstrained autonomy, but in structured, verifiable investigation. By enforcing deterministic mathematical analysis over subjective LLM text generation, and by enforcing unyielding safety boundaries for physical actions, FaultTrace redefines what it means for an AI to be a reliable engineering assistant.

As the software engineering community continues to build complex agent harnesses, the lessons learned from FaultTrace—that orchestration reliability, model-specific behaviors, and strict human-in-the-loop gates are paramount—will serve as a vital foundation for the next generation of intelligent systems.