How to Report DeepEval Results as JUnit XML

DeepEval JUnit reporting exports LLM evaluation results, including Faithfulness, Answer Relevancy, and Contextual Recall scores with a pass or fail state for each question, as JUnit XML through pytest's --junitxml flag. Testkube parses that XML natively and tracks the results across runs.

LLM outputs are non-deterministic, so exact-match assertions break once a correct answer can take more than one form. DeepEval scores each response against metrics and thresholds instead. Testkube, the testing platform that handles test orchestration, runs those evaluations in your pipeline and shows the results next to your existing test data.

This article continues from How to Integrate AI Testing in Delivery Pipelines and extends the same RAG application, DeepEval configuration, and Testkube workflow with JUnit reporting.

What Is LLM Evaluation with DeepEval?

DeepEval is an open-source Python framework that scores LLM responses against metrics such as Faithfulness and Answer Relevancy, then compares each score to a threshold to produce a pass or fail result.

An LLM can generate a response that looks perfectly reasonable and still be wrong, incomplete, or poorly grounded in the provided context. So instead of simply accepting the output, we need a way to systematically evaluate it against what we expect from the application.

DeepEval lets us define what a "good" response looks like and turns each measurement into a pass or fail result that a test suite can use.

DeepEval Metrics

DeepEval evaluates the output of an LLM application against a defined set of metrics rather than a single expected answer. For a RAG application, the three metrics central to this article are:

• Faithfulness, which measures whether a response is supported by the retrieved context.
• Answer Relevancy, which measures whether a response actually addresses the question asked.
• Contextual Recall, which measures whether the retrieved context contained the information needed to answer the question at all.

Each DeepEval test case is built from four components: the input question, the context retrieved to answer it, the actual response generated by the application, and the evaluation criteria that define what a correct or faithful answer looks like. DeepEval checks whether each claim in a RAG response is supported by the retrieved context. That check catches hallucinated answers, even ones that read as fluent and confident.

Metric Scores and Evaluation Thresholds

Each metric produces a numerical score between 0 and 1, and that score is only meaningful once it is compared against a threshold. A Faithfulness threshold of 0.80, for instance, means any response scoring below that figure is treated as a failed evaluation, regardless of how well-formed the sentence itself reads. In practice, each threshold works as a quality gate for the response.

Here is one failure from a run that used a weaker candidate prompt. The user asks, "Do you accept credit cards?" The retrieved context states, "Yes, we accept credit cards." But the generated response came out as: "Absolutely! We accept Visa, MasterCard, and American Express, basically all the cool kids of the credit card world. So whip out that plastic and let's make some magic happen! Just remember, unlike your favorite pair of jeans, we can't stretch those payments beyond 30 days for refunds."

The retrieved context supports exactly one claim: yes, credit cards are accepted. The response added card brands, a payments remark, and a 30-day refund limit that appear nowhere in that context. Faithfulness scored 0.5 against the 0.80 threshold, so the evaluation failed. The application still returned a well-formed response, so a functional test would have passed this same output. Metric scoring catches what a functional test misses.

DeepEval Results as Test Results

Here, DeepEval runs inside a pytest-based test suite. Each RAG question becomes a pytest test function, and each DeepEval metric assertion is the condition that function must meet to pass. The result of running that suite is a structured set of evaluation outcomes: a metric score, a threshold, and a pass or fail state for every question in the benchmark. Those results need to leave the terminal, and JUnit XML is the standard format for that.

How Do You Export DeepEval Results as JUnit XML?

Run the DeepEval suite with pytest's built-in --junitxml flag. pytest writes each question's pass, fail, or error state and assertion message to a JUnit XML file that Testkube parses natively.

What JUnit Reporting Is, and Where DeepEval Fits In

JUnit XML is a test result format that started with Java's JUnit framework. It is now the de facto result format for CI/CD and test-reporting tools, including Jenkins, GitLab CI, and Testkube. Those tools parse it no matter which language or framework produced the results. That portability makes it a convenient target for a Python-based evaluation suite like this one.

pytest runs the RAG suite, and its built-in --junitxml flag writes each run into this format with no extra tooling. Because Testkube parses JUnit XML natively, the results appear in its execution view without a custom integration.

JUnit XML Representation

The resulting report follows a consistent structure. A single testsuite element wraps the full run and reports the suite-level totals (tests, failures, errors, and skipped cases), along with total execution time. Inside it, one testcase element represents each RAG question. Every test case lands in one of four states:

A failing case carries the full assertion message inside that failure block, including the metric that missed its threshold and the reasoning behind the score, as in this case from qa_016:

AssertionError: Faithfulness failed: score 0.67 < threshold 0.80
Reason: The score is 0.67 because the actual output incorrectly implies that customers have a whole month to decide on returns, when in reality returns must be initiated within 30 days of the original purchase date.

tests/test_rag_eval.py:79: in test_rag_quality_gate
    raise AssertionError(
E   AssertionError: Faithfulness failed: score 0.67 < threshold 0.80
E   Reason: The score is 0.67 because the actual output incorrectly implies that customers have a whole month to decide on returns, when in reality returns must be initiated within 30 days of the original purchase date.

The JUnit reporter has no concept of what Faithfulness means, and no ability to judge whether 0.67 is a meaningful score on its own. It only records that a named test case passed, failed, or errored, and reproduces whatever message the assertion behind it provided. All of the meaning behind that message comes from DeepEval.

How to Run DeepEval with JUnit Reporting in a Test Workflow

The Test Workflow runs the DeepEval suite with pytest, writes a JUnit XML report, converts metric scores to InfluxDB line protocol with testkube-ai-metrics-parser, and saves both as artifacts. The RAG application, DeepEval configuration, and Insights metric views carry over unchanged from How to Integrate AI Testing in Delivery Pipelines.

What's New in This Workflow

There are two changes that separate this workflow from the one in the original post:

• The first is a --junitxml=results/junit-report.xml flag added to the existing pytest command, with results/*.xml added to the workflow's saved artifacts so the file persists after the pod terminates.
• The second addresses a reliability issue in the evaluation process. DeepEval's evaluator in test_rag_eval.py uses an OpenAI model as the judge for all three metrics. Longer RAG responses were occasionally hitting the model's completion-token limit, causing the evaluation to fail before a score could be produced. This was resolved by switching the judge model to gpt-3.5-turbo, which reduced the likelihood of hitting the token limit during evaluation.

Here is the workflow used for the execution. The full demo code is in the rag-quality-gate-demo repository on GitHub.

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
  name: rag-quality-gate
  namespace: testkube
spec:
  config:
    prompt:
      type: string
      default: baseline
      description: Which system prompt the RAG bot uses, baseline or candidate
  content:
    git:
      uri: https://github.com/sarvaniswapnapriya/rag-quality-gate-demo.git
      revision: master
  container:
    workingDir: /data/repo
    image: python:3.11-slim
    env:
    - name: OPENAI_API_KEY
      valueFrom:
        secretKeyRef:
          name: openai-secrets
          key: api-key
    - name: RAG_PROMPT
      value: "{{config.prompt}}"
    - name: DEEPEVAL_RESULTS_FOLDER
      value: /data/repo/results
    - name: DEEPEVAL_TELEMETRY_OPT_OUT
      value: "YES"
  steps:
  - name: rag-quality-gate
    shell: |-
      set -euxo pipefail

      echo "=== Python Environment ==="
      which python3
      python3 --version

      apt-get update
      apt-get install -y --no-install-recommends curl

      python3 -m pip install --upgrade pip
      python3 -m pip install --no-cache-dir -r requirements.txt
      python3 -m pip install --no-cache-dir pytest

      echo "=== Installed Packages ==="
      python3 -m pip show pytest
      python3 -c "import pytest; print(pytest.__version__)"

      echo "=== Running RAG Evaluation ==="
      mkdir -p results
      export DEEPEVAL_PER_ATTEMPT_TIMEOUT_SECONDS_OVERRIDE=300
      export DEEPEVAL_TASK_GATHER_BUFFER_SECONDS_OVERRIDE=60
      python3 -m pytest tests/test_rag_eval.py -v --tb=short --junitxml=results/junit-report.xml
  - name: export-metrics
    condition: always
    container:
      image: docker.io/kubeshop/testkube-ai-metrics-parser:latest
      imagePullPolicy: Always
    shell: |
      testkube-ai-metrics-parser deepeval \
        -i 'results/test_run_*.json' \
        -o results/metrics.lp \
        --tag suite=rag-quality-gate \
        --stdout
  - name: save-results
    condition: always
    artifacts:
      paths:
      - results/**/*.json
      - results/*.lp
      - results/*.xml

How to Track LLM Evaluation Results in Testkube Insights

DeepEval results surface inside Testkube in two places:

• The first is the results view attached to a single execution, shown below, where every test case from that run is listed individually alongside its status and duration.
• The second is the Insights dashboard, where Testkube aggregates both the JUnit data and DeepEval's own metric scores across many executions into charts a team can return to over time.

Evaluation Results per Execution

Each Testkube execution surfaces the DeepEval test results for that single run, with all 20 cases (qa_001 through qa_020) listed individually, each carrying its own status and duration. That per-test view gives a reader a path from a suite-level failure straight to the exact question and metric responsible for it.

Two executions illustrate the range of outcomes this produces. rag-quality-gate-81, run against the baseline prompt, shows 16 test cases passing cleanly: 16 passed, 4 failed, 0 skipped.

rag-quality-gate-82, run against the candidate prompt, shows 10 passed and 10 failed: genuine failures, with Faithfulness and Answer Relevancy scores that are present and visible but fall below threshold, as in the qa_016 Faithfulness failure (score 0.67 against a 0.80 threshold).

This per-execution view is the most granular layer Testkube offers, and it answers a narrow question well: what happened in this one run. Testkube also aggregates the same underlying data across executions, which is where a trend, instead of a single result, becomes visible.

Tracking Evaluation Metrics

Beyond any single execution, Testkube tracks a JUnit Case count measure broken down by passed, failed, errored, and skipped, segmented by test case and charted across executions over time. A regression or a recovery becomes visible as a shape spanning multiple runs, something a single result cannot show on its own.

The Test Count Failed view makes the baseline-versus-candidate comparison concrete. The total-failures count sits at 4 for the baseline-prompt execution and 10 for the candidate-prompt execution, matching the 26 passed and 14 failed across both executions, as shown. Testkube Insights also supports a question-by-question breakdown underneath each chart, segmented by test case name, so a spike in failures can be traced back to the specific cases driving it, not just the count.

Workflow flip rate tracks how often a run's outcome flips between pass and fail across executions, the same signal used to spot flaky tests. To measure true non-determinism, compare repeated runs of the same prompt rather than a baseline prompt against a candidate.

LLM Evaluation Alongside Other Test Data

Testkube Insights does not treat LLM evaluation results as a separate category. They sit in the same view as conventional API, functional, and performance test results, which is what makes it possible to see a quality regression alongside everything else a team already tracks.

In that view, keep failed cases and errored cases separate, because they point to different problems. A failed case is a quality problem that DeepEval caught. An errored case is a pipeline problem: the evaluation breaking before it can render a verdict at all.

JUnit results and the trends built from them are one signal in that view. The DeepEval metrics view, showing the actual Faithfulness, Answer Relevancy, and Contextual Recall scores behind those results, is the complementary one, and Testkube displays both in the same dashboard.

Conclusion

DeepEval gives LLM output quality a quantitative basis, scoring Faithfulness, Answer Relevancy, and Contextual Recall instead of leaving response quality to a reviewer's judgment. Its thresholds are what turn that qualitative question into a concrete test outcome: a pass or a fail attached to a specific question and a specific reason.

JUnit is the standardized format that carries those outcomes forward, connecting DeepEval's evaluation to Testkube's existing test-execution data without requiring any custom integration work. Testkube brings those results into the same test data and observability workflow a team uses for everything else it tests. An LLM-powered feature is held to the same bar as any other part of the system.

Frequently Asked Questions

{{banner-get-started}}