One of the biggest challenges was designing evaluation logic that remains accurate, explainable, and trustworthy despite the non-deterministic nature of LLM outputs.

Major hurdles and solutions:

• Early versions mixed quality, safety, and structure into a single score, which produced misleading results when improvements in one area caused regressions in another.
• This was solved by separating deterministic metrics from interpretive analysis and enforcing safety-first decision rules.
• Another challenge was avoiding hallucinated or fabricated metrics from evaluation models. We solved this by ensuring that every metric shown is directly derived from observable model outputs, not inferred assumptions.
• A second LLM pass was introduced only for human-readable explanation, not scoring, which improved clarity without compromising reliability.

These design decisions significantly improved the robustness and credibility of the platform.