Testing and Observing AI Applications

AI features are harder to test than normal code: the same prompt can produce different wording each time, answers can be subtly wrong, and every call costs money. Yet shipping untested prompts is how chatbots end up promising refunds that do not exist. You need a layered strategy — fast deterministic tests for your own code, and evaluation tests that measure the model's answers — plus production observability to see latency, token usage and failures.

This lesson shows how to unit-test code that uses ChatClient without calling a model, run integration tests against a local Ollama model with Testcontainers, evaluate answer quality with Spring AI's evaluators, and monitor AI calls with Micrometer metrics and traces.

Layer 1: Isolate the Model Behind Your Own Service

Put AI calls behind small, well-named service methods (TriageService.triage(text)) and have the rest of your application depend on those. Controllers and business logic can then be tested with @MockitoBean TriageService, returning fixed results — fast, free and deterministic. This also keeps prompt details out of your controllers.

Layer 2: Integration Tests with a Real (Local) Model

To test prompts, structured output and tool wiring end to end, run a small model in a Testcontainers OllamaContainer connected with @ServiceConnection. Assert on structure and key facts, not exact wording — for example that the category enum is BILLING, not that the summary equals a specific sentence. Tag these tests so they run in CI but not on every local build.

Layer 3: Evaluation (LLM-as-a-Judge)

Spring AI provides Evaluator implementations that use a model to grade answers. RelevancyEvaluator checks whether an answer is relevant to the question given the retrieved context — ideal for RAG. FactCheckingEvaluator checks whether a claim is supported by provided documents, detecting hallucinations. Build a set of 20–100 representative questions with expected facts, run them after every prompt or model change, and track the pass rate over time.

Observability in Production

Spring AI is instrumented with Micrometer. With Actuator on the classpath you get metrics such as gen_ai.client.operation (latency per model call) and gen_ai.client.token.usage (input and output tokens by model), plus tracing spans for ChatClient calls, advisors, tool calls and vector store queries. Export them to Prometheus/Grafana or any OpenTelemetry backend to answer: which feature uses the most tokens, which calls are slow, and how often do tools fail?

Prompt and completion content are not logged by default because they may contain personal data. Enable content logging only in development, or with proper redaction.

Guardrails

Combine testing with runtime safeguards: SafeGuardAdvisor blocks requests containing configured sensitive words, input length limits prevent huge prompts, output validation checks structured results, and rate limits per user protect your budget.

Examples

Controller test with the AI service mocked

Java

@WebMvcTest(TicketController.class)
class TicketControllerTest {

    @Autowired MockMvcTester mvc;
    @MockitoBean TriageService triageService;

    @Test
    void returnsTriageResult() {
        when(triageService.triage(anyString()))
            .thenReturn(new TicketTriage(TicketCategory.BILLING, 4, "Double charge"));

        assertThat(mvc.post().uri("/api/tickets/triage")
                .contentType(MediaType.TEXT_PLAIN)
                .content("I was charged twice"))
            .hasStatusOk()
            .bodyJson()
            .extractingPath("$.category").isEqualTo("BILLING");
    }
}

Output

TicketControllerTest > returnsTriageResult() PASSED (0.4s, no model call, no cost)

Integration test against a local model with Testcontainers

Java

@SpringBootTest
@Testcontainers
@Tag("ai")
class TriageServiceIT {

    @Container
    @ServiceConnection
    static OllamaContainer ollama = new OllamaContainer("ollama/ollama:latest");

    @DynamicPropertySource
    static void props(DynamicPropertyRegistry registry) {
        registry.add("spring.ai.ollama.chat.model", () -> "llama3.2");
        registry.add("spring.ai.ollama.init.pull-model-strategy", () -> "when_missing");
    }

    @Autowired TriageService triageService;

    @Test
    void classifiesBillingTickets() {
        TicketTriage result = triageService.triage("I was charged twice for my course, please refund one payment");
        assertThat(result.category()).isEqualTo(TicketCategory.BILLING);
        assertThat(result.priority()).isBetween(1, 5);
    }
}

// Run only AI tests:  mvn verify -Dgroups=ai

Output

Pulling model llama3.2 ...
TriageServiceIT > classifiesBillingTickets() PASSED (38.2s)

Evaluating RAG answers for relevancy

Java

@SpringBootTest
@Tag("ai-eval")
class PolicyAnswerEvaluationTest {

    @Autowired ChatClient.Builder builder;
    @Autowired VectorStore vectorStore;

    @ParameterizedTest
    @CsvSource(delimiter = '|', value = {
        "Can I get a refund after 10 days?        | refund",
        "How long is a certificate valid?          | certificate",
        "Can I download videos for offline viewing?| offline"
    })
    void answersAreRelevantToRetrievedContext(String question, String topic) {
        RetrievalAugmentationAdvisor rag = RetrievalAugmentationAdvisor.builder()
            .documentRetriever(VectorStoreDocumentRetriever.builder().vectorStore(vectorStore).build())
            .build();

        ChatResponse response = builder.build().prompt()
            .advisors(rag)
            .user(question)
            .call()
            .chatResponse();

        List<Document> context = response.getMetadata().get(RetrievalAugmentationAdvisor.DOCUMENT_CONTEXT);
        String answer = response.getResult().getOutput().getText();

        RelevancyEvaluator evaluator = new RelevancyEvaluator(builder);
        EvaluationResponse eval = evaluator.evaluate(new EvaluationRequest(question, context, answer));

        assertThat(eval.isPass()).as("Irrelevant answer for '%s': %s", question, answer).isTrue();
    }
}

Output

PolicyAnswerEvaluationTest
  ✔ [1] Can I get a refund after 10 days?, refund
  ✔ [2] How long is a certificate valid?, certificate
  ✘ [3] Can I download videos for offline viewing?, offline
      Irrelevant answer for 'Can I download videos...': Yes, you can download all videos... (no supporting context)

Metrics and tracing configuration

Java

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

# application.yml
management:
  endpoints:
    web:
      exposure:
        include: health, metrics, prometheus
spring:
  ai:
    chat:
      observations:
        log-prompt: false        # keep personal data out of logs in production
        log-completion: false
    tools:
      observations:
        include-content: false

Output

GET /actuator/metrics/gen_ai.client.token.usage
{"name":"gen_ai.client.token.usage","measurements":[{"statistic":"COUNT","value":48213.0}],
 "availableTags":[{"tag":"gen_ai.token.type","values":["input","output","total"]},
                  {"tag":"gen_ai.request.model","values":["gpt-5-mini"]}]}

Common Mistakes

Asserting exact model wording in tests, which makes them fail randomly; assert structure, enums and key facts.
Calling a paid hosted model in every unit test run, slowing builds and costing money.
Changing a prompt or switching model versions without re-running an evaluation set.
Logging full prompts and completions in production, leaking personal or confidential data.
Not tracking token usage per feature, then being surprised by the monthly bill.

Key Points to Remember

Hide AI calls behind services and mock them for fast, deterministic unit and web tests.
Use Testcontainers OllamaContainer with @ServiceConnection for realistic, free integration tests.
RelevancyEvaluator and FactCheckingEvaluator measure answer quality and detect hallucinations.
Spring AI emits Micrometer metrics and traces for latency, token usage, tools and vector searches.
Keep prompt/completion content out of production logs and monitor cost per feature.

Practice the examples

Change an input, predict the result, then compare it with the output. Explain why the result changes.

Use your local JDK or project IDE for these examples. Codelab currently runs Python and HTML/CSS/JavaScript; framework examples may need project dependencies.