Course topics

By WebNest Studio Engineering Team

Spring Boot Tutorial

Testing and Observing AI Applications

AI features are harder to test than normal code: the same prompt can produce different wording each time, answers can be subtly wrong, and every call costs money. Yet shipping untested prompts is how chatbots end up promising refunds that do not exist. You need a layered strategy — fast deterministic tests for your own code, and evaluation tests that measure the model's answers — plus production observability to see latency, token usage and failures.

This lesson shows how to unit-test code that uses ChatClient without calling a model, run integration tests against a local Ollama model with Testcontainers, evaluate answer quality with Spring AI's evaluators, and monitor AI calls with Micrometer metrics and traces.

Layer 1: Isolate the Model Behind Your Own Service

Put AI calls behind small, well-named service methods (TriageService.triage(text)) and have the rest of your application depend on those. Controllers and business logic can then be tested with @MockitoBean TriageService, returning fixed results — fast, free and deterministic. This also keeps prompt details out of your controllers.

Layer 2: Integration Tests with a Real (Local) Model

To test prompts, structured output and tool wiring end to end, run a small model in a Testcontainers OllamaContainer connected with @ServiceConnection. Assert on structure and key facts, not exact wording — for example that the category enum is BILLING, not that the summary equals a specific sentence. Tag these tests so they run in CI but not on every local build.

Layer 3: Evaluation (LLM-as-a-Judge)

Spring AI provides Evaluator implementations that use a model to grade answers. RelevancyEvaluator checks whether an answer is relevant to the question given the retrieved context — ideal for RAG. FactCheckingEvaluator checks whether a claim is supported by provided documents, detecting hallucinations. Build a set of 20–100 representative questions with expected facts, run them after every prompt or model change, and track the pass rate over time.

Observability in Production

Spring AI is instrumented with Micrometer. With Actuator on the classpath you get metrics such as gen_ai.client.operation (latency per model call) and gen_ai.client.token.usage (input and output tokens by model), plus tracing spans for ChatClient calls, advisors, tool calls and vector store queries. Export them to Prometheus/Grafana or any OpenTelemetry backend to answer: which feature uses the most tokens, which calls are slow, and how often do tools fail?

Prompt and completion content are not logged by default because they may contain personal data. Enable content logging only in development, or with proper redaction.

Guardrails

Combine testing with runtime safeguards: SafeGuardAdvisor blocks requests containing configured sensitive words, input length limits prevent huge prompts, output validation checks structured results, and rate limits per user protect your budget.

Examples

Controller test with the AI service mocked

Java
@WebMvcTest(TicketController.class)
class TicketControllerTest {

    @Autowired MockMvcTester mvc;
    @MockitoBean TriageService triageService;

    @Test
    void returnsTriageResult() {
        when(triageService.triage(anyString()))
            .thenReturn(new TicketTriage(TicketCategory.BILLING, 4, "Double charge"));

        assertThat(mvc.post().uri("/api/tickets/triage")
                .contentType(MediaType.TEXT_PLAIN)
                .content("I was charged twice"))
            .hasStatusOk()
            .bodyJson()
            .extractingPath("$.category").isEqualTo("BILLING");
    }
}
Output
TicketControllerTest > returnsTriageResult() PASSED (0.4s, no model call, no cost)

Integration test against a local model with Testcontainers

Java
@SpringBootTest
@Testcontainers
@Tag("ai")
class TriageServiceIT {

    @Container
    @ServiceConnection
    static OllamaContainer ollama = new OllamaContainer("ollama/ollama:latest");

    @DynamicPropertySource
    static void props(DynamicPropertyRegistry registry) {
        registry.add("spring.ai.ollama.chat.model", () -> "llama3.2");
        registry.add("spring.ai.ollama.init.pull-model-strategy", () -> "when_missing");
    }

    @Autowired TriageService triageService;

    @Test
    void classifiesBillingTickets() {
        TicketTriage result = triageService.triage("I was charged twice for my course, please refund one payment");
        assertThat(result.category()).isEqualTo(TicketCategory.BILLING);
        assertThat(result.priority()).isBetween(1, 5);
    }
}

// Run only AI tests:  mvn verify -Dgroups=ai
Output
Pulling model llama3.2 ...
TriageServiceIT > classifiesBillingTickets() PASSED (38.2s)

Evaluating RAG answers for relevancy

Java
@SpringBootTest
@Tag("ai-eval")
class PolicyAnswerEvaluationTest {

    @Autowired ChatClient.Builder builder;
    @Autowired VectorStore vectorStore;

    @ParameterizedTest
    @CsvSource(delimiter = '|', value = {
        "Can I get a refund after 10 days?        | refund",
        "How long is a certificate valid?          | certificate",
        "Can I download videos for offline viewing?| offline"
    })
    void answersAreRelevantToRetrievedContext(String question, String topic) {
        RetrievalAugmentationAdvisor rag = RetrievalAugmentationAdvisor.builder()
            .documentRetriever(VectorStoreDocumentRetriever.builder().vectorStore(vectorStore).build())
            .build();

        ChatResponse response = builder.build().prompt()
            .advisors(rag)
            .user(question)
            .call()
            .chatResponse();

        List<Document> context = response.getMetadata().get(RetrievalAugmentationAdvisor.DOCUMENT_CONTEXT);
        String answer = response.getResult().getOutput().getText();

        RelevancyEvaluator evaluator = new RelevancyEvaluator(builder);
        EvaluationResponse eval = evaluator.evaluate(new EvaluationRequest(question, context, answer));

        assertThat(eval.isPass()).as("Irrelevant answer for '%s': %s", question, answer).isTrue();
    }
}
Output
PolicyAnswerEvaluationTest
  ✔ [1] Can I get a refund after 10 days?, refund
  ✔ [2] How long is a certificate valid?, certificate
  ✘ [3] Can I download videos for offline viewing?, offline
      Irrelevant answer for 'Can I download videos...': Yes, you can download all videos... (no supporting context)

Metrics and tracing configuration

Java
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

# application.yml
management:
  endpoints:
    web:
      exposure:
        include: health, metrics, prometheus
spring:
  ai:
    chat:
      observations:
        log-prompt: false        # keep personal data out of logs in production
        log-completion: false
    tools:
      observations:
        include-content: false
Output
GET /actuator/metrics/gen_ai.client.token.usage
{"name":"gen_ai.client.token.usage","measurements":[{"statistic":"COUNT","value":48213.0}],
 "availableTags":[{"tag":"gen_ai.token.type","values":["input","output","total"]},
                  {"tag":"gen_ai.request.model","values":["gpt-5-mini"]}]}

Common Mistakes

  • Asserting exact model wording in tests, which makes them fail randomly; assert structure, enums and key facts.
  • Calling a paid hosted model in every unit test run, slowing builds and costing money.
  • Changing a prompt or switching model versions without re-running an evaluation set.
  • Logging full prompts and completions in production, leaking personal or confidential data.
  • Not tracking token usage per feature, then being surprised by the monthly bill.

Exercise

Using the lesson's Controller test with the AI service mocked example, create a small implementation or design sketch that demonstrates Hide AI calls behind services and mock them for fast, deterministic unit and web tests.. Add a test case or scenario that would catch Asserting exact model wording in tests, which makes them fail randomly; assert structure, enums and key facts., and describe the expected result.

Show hints
  • Start with Layer 1: Isolate the Model Behind Your Own Service and compare your approach with Controller test with the AI service mocked. Identify the decision or behavior the lesson recommends.
  • Walk through your scenario step by step. Confirm that it follows Hide AI calls behind services and mock them for fast, deterministic unit and web tests. and that it would catch the failure described by Asserting exact model wording in tests, which makes them fail randomly; assert structure, enums and key facts..

Quick quiz

1. Which choice is a mistake to avoid in Testing and Observing AI Applications?

  • A. Hide AI calls behind services and mock them for fast, deterministic unit and web tests.
  • B. Asserting exact model wording in tests, which makes them fail randomly; assert structure, enums and key facts.
  • C. Calling a paid hosted model in every unit test run, slowing builds and costing money.
Show answer

B. Avoid this approach: Asserting exact model wording in tests, which makes them fail randomly; assert structure, enums and key facts. The lesson recommends: Hide AI calls behind services and mock them for fast, deterministic unit and web tests.

Interview questions

How would you apply the guidance in Layer 1: Isolate the Model Behind Your Own Service to a real task, and what failure would you guard against?

Show answer

Hide AI calls behind services and mock them for fast, deterministic unit and web tests. Apply this in Layer 1: Isolate the Model Behind Your Own Service; the example Controller test with the AI service mocked shows one way to do it. Avoid Asserting exact model wording in tests, which makes them fail randomly; assert structure, enums and key facts. and explain how your implementation or design prevents that failure.

Key Points to Remember

  • Hide AI calls behind services and mock them for fast, deterministic unit and web tests.
  • Use Testcontainers OllamaContainer with @ServiceConnection for realistic, free integration tests.
  • RelevancyEvaluator and FactCheckingEvaluator measure answer quality and detect hallucinations.
  • Spring AI emits Micrometer metrics and traces for latency, token usage, tools and vector searches.
  • Keep prompt/completion content out of production logs and monitor cost per feature.

Practice in the playground

Write and run your own code for this topic, or start from the exercise or an example. Runs plain Java 17. Frameworks such as Spring, and databases, are not available here. Your code for each topic is kept in this browser.

Run the examples

Change an input, predict the result, then compare it with the output. Explain why the result changes.

These examples need a JDK project with their libraries on your computer, so they can be copied but not run here. Codelab runs Python, FastAPI backends with SQLite, plain Java 17 and HTML/CSS/JavaScript.

WebNest Studio

Written by

WebNest Studio Engineering Team

Practical lessons from developers working with Java, Python, React, databases and production systems.

Led by Vansh Duggal, Founder, WebNest Studio · Full-stack & software engineering