Empirical production milestones

Boeing

Ellie and Mia are applied to fault trees for production systems still in service. Boeing safety SMEs set the human baseline: each page Ellie reconstructs and each latent fault Mia surfaces is checked against what an experienced analyst produces, and the discrepancies are traced to their cause — a drawing convention, a detector miss, or a genuine finding the review had not reached.

• Head-to-head benchmarks against SME baselines on real production trees
• Vector-PDF and scanned drawings spanning decades of documentation conventions
• Findings validated page by page, so accuracy is measured per node, not per document

Gonzaga University

Gonzaga contributes the academic rigour: evaluation design, annotation protocols and independent measurement of what Ellie and Mia actually achieve. The held-out test sets, the per-node accuracy metric and the review-time studies that show where analyst minutes are really spent come out of this collaboration.

• Independent evaluation design and held-out test sets never touched during development
• Annotation protocols that make ground truth reproducible across annotators
• Research into where review time goes — the metric that decides whether the tooling helps

Topological and object-detection accuracy

• Object detection: every gate, event, transfer and connector symbol on a page, by class and position
• Topology: every parent-child connection between them — the number that decides whether the tree is right
• Measured on held-out production pages that were never inspected during development
• Reported per node: 99.96% means fewer than one misplaced connection in two thousand
• Every residual error is traced to a drawing convention and fed back into Ellie's parser rules

1,000+ hours of training and fine-tuning

• Detector training and fine-tuning on NVIDIA AI clusters we operate on our own premises
• Synthetic drawing pages generated from the real drawing grammar to cover rare gate types
• Every candidate model is evaluated on the same held-out set before it replaces the previous one
• Mia's engine is exact, not learned: cut sets come from a proven fault-tree solver and need no training
• The language assistant is a local model constrained to the analysed trees — it never invents a fact