Beyond Conformance: Building a Typed Decision Layer for FHIR R4 Resources Using Large Language Models

Executive Overview

In modern healthcare IT, passing structural validation is often mistaken for semantic readiness. For years, interoperability pipelines have relied heavily on FHIR (Fast Healthcare Interoperability Resources) conformance tools to ensure that incoming data meets specification standards. Yet, a syntactically pristine FHIR resource can easily represent a clinical contradiction, an outdated record, or a critical public health hazard that demands immediate downstream action.

To bridge the gap between structural validity and operational necessity, engineers and clinical informaticians are increasingly exploring specialized decision layers. Recently, an experimental proof-of-concept project emerged exploring this intersection. By pairing parsed FHIR R4 resources with TypeSafe AI’s Jev model—a structured decision-making engine—this architecture transforms raw healthcare data into actionable outcomes: accept, route, flag, or human review.

Operating on a synthetic dataset of 204 medical records, initial iterations of the model stumbled, trailing behind basic hand-written rules in both routing accuracy and disease detection. However, deep-dive debugging revealed these shortcomings stemmed from developer implementation errors—specifically, a serialization oversight and suboptimally framed prompts. Once resolved, the AI decision layer successfully outperformed traditional rules across multiple test categories.

Despite these algorithmic triumphs, the architecture currently highlights persistent trade-offs in clinical data pipelines, notably failing latency benchmarks and raising critical questions about prompt design, evaluation integrity, and the enduring utility of deterministic rule baselines.

Detailed Chronology: From First Run to Architectural Refinement

The development of the Jev-FHIR decision layer offers a case study in iterative engineering, exposing the delicate interplay between data parsing, model prompting, and ground-truth validation.

Phase 1: The Initial Live Run and Early Failures

During the initial benchmark across 204 synthetic records, the Jev-driven pipeline underperformed expectations. In routing tasks, the model scored 0.814, lagging significantly behind a simple, hand-written rule baseline that scored 0.922. Simultaneously, the pipeline missed numerous notifiable diseases, flagging them incorrectly or dropping them into ambiguous paths.

Initial diagnostic efforts pointed toward systemic limitations in utilizing generative models for predictable healthcare routing. However, a granular, row-by-row inspection using an integrated debugging UI exposed structural ingestion bugs rather than model incompetence.

Phase 2: Uncovering Serialization and Prompting Blind Spots

The first critical breakthrough occurred during a review of text-only dengue fever records. The state panel generated for these cases displayed null values for code_value, code_display, and code_system, leaving the model to evaluate an effectively blank record. The legacy serializer had been configured to read strictly from code.coding, completely ignoring the code.text field where localized diagnoses (such as "Demam berdarah dengue (DBD)" or "TB paru") resided.

To mitigate token bloat and prevent unstructured data leakage, a strict 200-character limit was applied to the extracted text:

# src/jev_fhir/serializer/condition.py
CODE_TEXT_MAX_LENGTH = 200

code = resource.get("code")
code_text = code.get("text") if isinstance(code, dict) else None
onset = resource.get("onsetDateTime")

"code_display": first_coding_value(code, "display"),
"code_text": code_text[:CODE_TEXT_MAX_LENGTH] if isinstance(code_text, str) else None,
"clinical_status": first_coding_code(resource.get("clinicalStatus")),

Following this fix, re-evaluation revealed an immediate correction in confidence and classification for text-only notifiable diseases, driving notifiable detection accuracy from 0.839 to 0.935 and recall from 0.737 to 0.895.

Phase 3: Solving the Routing Deficit

The routing discrepancy required a dual approach: re-evaluating both the serialization state and the framing of the core question. Analysis of misrouted FHIR Bundles showed a direct correlation with multi-condition encounters. When a bundle contained a single Encounter paired with multiple Conditions, the system failed to correctly categorize its primary clinical intent.

The prompt was adjusted from a descriptive inquiry to a prescriptive workflow instruction:

• Previous Question: "What type of clinical data does this FHIR Bundle primarily contain?"
• Revised Question: "Which processing pipeline should handle this FHIR Bundle?"

Combined with an updated serializer state tracking encounter references explicitly, Jev’s routing accuracy jumped from 0.814 to 0.951, outperforming the rule baseline of 0.922.

Supporting Context & Metrics

To objectively evaluate the decision layer, performance was benchmarked against transparent rule-based baselines using a balanced mix of standard and adversarial fixtures.

Comparative Performance Matrix

The Conformance vs. Decision Boundary

A foundational realization of this project is the conceptual divergence between structural validation and workflow decision-making:

• Conformance Checking: Answers the question, "Does this resource meet the formal specification?" Its output is binary (Pass/Fail) governed by rigid profiles and invariants (e.g., checking that Indonesia’s national ID, the NIK, contains exactly 16 digits instead of 15).
• Decision Layer: Answers the question, "What operational action must be taken next?" Its output consists of a typed action and a quantified confidence score, governed by institutional risk tolerance. When uncertainty arises, the system defers to human oversight rather than throwing a hard validation error.

+------------------+     +-------------------+     +------------------+     +---------------+
|  FHIR JSON Input | --> | Serialization &   | --> | Jev Decision     | --> | Threshold &   |
|                  |     | State Flattening  |     | Engine (Primitives)|   | Action Engine |
+------------------+     +-------------------+     +------------------+     +---------------+

The system leverages Jev’s three core primitives: Choice (selecting a single option from a list), Score (returning a normalized numeric scale), and Noul (evaluating a statement’s truth probability). By decoupling raw data parsing from probabilistic judgment, the pipeline maintains strict schema compliance while introducing semantic intelligence.

Official Statements and Architectural Insights

Developers and data architects working on enterprise healthcare frameworks note that integrating AI decision layers into legacy health data pipelines requires unprecedented caution.

"A FHIR resource can pass validation and still be the wrong thing to act on," notes the project’s lead architect. "Conformance checking tells you if the grammar is correct. A decision layer tells you what the hospital workflow demands next. When you evaluate an AI model on clinical data, you are frequently measuring the quality of your serialization pipeline and your prompt framing long before you are measuring the limits of the model itself."

Furthermore, experts emphasize that probabilistic outputs cannot be mapped directly to automated actions without rigorous thresholding bands. For instance, the notifiable disease detector implements a tripartite decision band:

1. Probability < 0.50: Deemed not notifiable; standard processing.
2. Probability 0.50 – 0.80: Falls within the review band; routed to clinical personnel.
3. Probability ≥ 0.80: Confirmed notifiable; automatically generates an urgent FHIR Flag resource.

This design guarantees that edge cases and ambiguous clinical presentations are safely quarantined rather than silently mishandled by automated scripts.

Future Outlook and Challenges Ahead

While the proof of concept demonstrates the viability of a typed decision layer over FHIR resources, significant hurdles remain before production deployment can be considered.

1. Latency Bottlenecks

Performance testing revealed a glaring shortfall in execution speed. The target median (p50) latency for automated processing was set under 30 milliseconds. However, live client-side round trips clocked p50 latencies at approximately 235 ms for routing and notifiable detection, and up to 458 ms for quality scoring (where a Patient resource requires dual model evaluations). These figures highlight that real-time clinical integration—such as inline Electronic Health Record (EHR) screening—will require localized model deployment or aggressive caching strategies.

2. Economic Viability and Token Economics

The full experimental run consumed 147,401 tokens, accumulating an operational cost of approximately $0.006—translating to roughly $0.03 per 1,000 records. While this easily clears basic cost parameters compared to manual human audits, it exceeds the initial strict target of $0.01 per 1,000 records.

3. Evaluation Rigor and Unseen Datasets

Crucially, out of 204 test fixtures, 124 remain pending formal human clinical review, and the remaining 80 were programmatically generated synthetic assets. Because iterative bug fixes were performed in direct response to failing fixtures, the final high accuracy scores (0.935 and 0.951) reflect fitted performance rather than true held-out generalization.

Next Steps for the Architecture

• Held-Out Validation: Establishing a strictly isolated, expert-reviewed clinical dataset to test out-of-distribution performance.
• Latency Profiling: Separating network overhead from core model generation time to optimize API interaction bottlenecks.
• Serializer Expansion: Upgrading secondary serializers (such as ObservationSerializer) to comprehensively ingest descriptive text fields (code.text) rather than relying solely on structured coding arrays.

Ultimately, this project proves that while deterministic rules and structured schemas remain indispensable pillars of healthcare interoperability, they are insufficient for managing the semantic ambiguity of real-world medicine. By wrapping parsed FHIR resources into lightweight, inspectable states governed by transparent confidence thresholds, systems can successfully blend the safety of rigid specifications with the adaptability of modern AI.