By WebNest Studio Engineering Team
Spring Boot Tutorial
Testing and Observing AI Applications
AI features are harder to test than normal code: the same prompt can produce different wording each time, answers can be subtly wrong, and every call costs money. Yet shipping untested prompts is how chatbots end up promising refunds that do not exist. You need a layered strategy — fast deterministic tests for your own code, and evaluation tests that measure the model's answers — plus production observability to see latency, token usage and failures.
This lesson shows how to unit-test code that uses ChatClient without calling a model, run integration tests against a local Ollama model with Testcontainers, evaluate answer quality with Spring AI's evaluators, and monitor AI calls with Micrometer metrics and traces.
Layer 1: Isolate the Model Behind Your Own Service
Put AI calls behind small, well-named service methods (TriageService.triage(text)) and have the rest of your application depend on those. Controllers and business logic can then be tested with @MockitoBean TriageService, returning fixed results — fast, free and deterministic. This also keeps prompt details out of your controllers.
Layer 2: Integration Tests with a Real (Local) Model
To test prompts, structured output and tool wiring end to end, run a small model in a Testcontainers OllamaContainer connected with @ServiceConnection. Assert on structure and key facts, not exact wording — for example that the category enum is BILLING, not that the summary equals a specific sentence. Tag these tests so they run in CI but not on every local build.
Layer 3: Evaluation (LLM-as-a-Judge)
Spring AI provides Evaluator implementations that use a model to grade answers. RelevancyEvaluator checks whether an answer is relevant to the question given the retrieved context — ideal for RAG. FactCheckingEvaluator checks whether a claim is supported by provided documents, detecting hallucinations. Build a set of 20–100 representative questions with expected facts, run them after every prompt or model change, and track the pass rate over time.
Observability in Production
Spring AI is instrumented with Micrometer. With Actuator on the classpath you get metrics such as gen_ai.client.operation (latency per model call) and gen_ai.client.token.usage (input and output tokens by model), plus tracing spans for ChatClient calls, advisors, tool calls and vector store queries. Export them to Prometheus/Grafana or any OpenTelemetry backend to answer: which feature uses the most tokens, which calls are slow, and how often do tools fail?
Prompt and completion content are not logged by default because they may contain personal data. Enable content logging only in development, or with proper redaction.
Guardrails
Combine testing with runtime safeguards: SafeGuardAdvisor blocks requests containing configured sensitive words, input length limits prevent huge prompts, output validation checks structured results, and rate limits per user protect your budget.
Examples
Controller test with the AI service mocked
@WebMvcTest(TicketController.class)
class TicketControllerTest {
@Autowired MockMvcTester mvc;
@MockitoBean TriageService triageService;
@Test
void returnsTriageResult() {
when(triageService.triage(anyString()))
.thenReturn(new TicketTriage(TicketCategory.BILLING, 4, "Double charge"));
assertThat(mvc.post().uri("/api/tickets/triage")
.contentType(MediaType.TEXT_PLAIN)
.content("I was charged twice"))
.hasStatusOk()
.bodyJson()
.extractingPath("$.category").isEqualTo("BILLING");
}
}
TicketControllerTest > returnsTriageResult() PASSED (0.4s, no model call, no cost)
Integration test against a local model with Testcontainers
@SpringBootTest
@Testcontainers
@Tag("ai")
class TriageServiceIT {
@Container
@ServiceConnection
static OllamaContainer ollama = new OllamaContainer("ollama/ollama:latest");
@DynamicPropertySource
static void props(DynamicPropertyRegistry registry) {
registry.add("spring.ai.ollama.chat.model", () -> "llama3.2");
registry.add("spring.ai.ollama.init.pull-model-strategy", () -> "when_missing");
}
@Autowired TriageService triageService;
@Test
void classifiesBillingTickets() {
TicketTriage result = triageService.triage("I was charged twice for my course, please refund one payment");
assertThat(result.category()).isEqualTo(TicketCategory.BILLING);
assertThat(result.priority()).isBetween(1, 5);
}
}
// Run only AI tests: mvn verify -Dgroups=ai
Pulling model llama3.2 ...
TriageServiceIT > classifiesBillingTickets() PASSED (38.2s)
Evaluating RAG answers for relevancy
@SpringBootTest
@Tag("ai-eval")
class PolicyAnswerEvaluationTest {
@Autowired ChatClient.Builder builder;
@Autowired VectorStore vectorStore;
@ParameterizedTest
@CsvSource(delimiter = '|', value = {
"Can I get a refund after 10 days? | refund",
"How long is a certificate valid? | certificate",
"Can I download videos for offline viewing?| offline"
})
void answersAreRelevantToRetrievedContext(String question, String topic) {
RetrievalAugmentationAdvisor rag = RetrievalAugmentationAdvisor.builder()
.documentRetriever(VectorStoreDocumentRetriever.builder().vectorStore(vectorStore).build())
.build();
ChatResponse response = builder.build().prompt()
.advisors(rag)
.user(question)
.call()
.chatResponse();
List<Document> context = response.getMetadata().get(RetrievalAugmentationAdvisor.DOCUMENT_CONTEXT);
String answer = response.getResult().getOutput().getText();
RelevancyEvaluator evaluator = new RelevancyEvaluator(builder);
EvaluationResponse eval = evaluator.evaluate(new EvaluationRequest(question, context, answer));
assertThat(eval.isPass()).as("Irrelevant answer for '%s': %s", question, answer).isTrue();
}
}
PolicyAnswerEvaluationTest
✔ [1] Can I get a refund after 10 days?, refund
✔ [2] How long is a certificate valid?, certificate
✘ [3] Can I download videos for offline viewing?, offline
Irrelevant answer for 'Can I download videos...': Yes, you can download all videos... (no supporting context)
Metrics and tracing configuration
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
# application.yml
management:
endpoints:
web:
exposure:
include: health, metrics, prometheus
spring:
ai:
chat:
observations:
log-prompt: false # keep personal data out of logs in production
log-completion: false
tools:
observations:
include-content: false
GET /actuator/metrics/gen_ai.client.token.usage
{"name":"gen_ai.client.token.usage","measurements":[{"statistic":"COUNT","value":48213.0}],
"availableTags":[{"tag":"gen_ai.token.type","values":["input","output","total"]},
{"tag":"gen_ai.request.model","values":["gpt-5-mini"]}]}
Common Mistakes
- Asserting exact model wording in tests, which makes them fail randomly; assert structure, enums and key facts.
- Calling a paid hosted model in every unit test run, slowing builds and costing money.
- Changing a prompt or switching model versions without re-running an evaluation set.
- Logging full prompts and completions in production, leaking personal or confidential data.
- Not tracking token usage per feature, then being surprised by the monthly bill.
Exercise
Using the lesson's Controller test with the AI service mocked example, create a small implementation or design sketch that demonstrates Hide AI calls behind services and mock them for fast, deterministic unit and web tests.. Add a test case or scenario that would catch Asserting exact model wording in tests, which makes them fail randomly; assert structure, enums and key facts., and describe the expected result.
Show hints
- Start with Layer 1: Isolate the Model Behind Your Own Service and compare your approach with Controller test with the AI service mocked. Identify the decision or behavior the lesson recommends.
- Walk through your scenario step by step. Confirm that it follows Hide AI calls behind services and mock them for fast, deterministic unit and web tests. and that it would catch the failure described by Asserting exact model wording in tests, which makes them fail randomly; assert structure, enums and key facts..
Quick quiz
1. Which choice is a mistake to avoid in Testing and Observing AI Applications?
- A. Hide AI calls behind services and mock them for fast, deterministic unit and web tests.
- B. Asserting exact model wording in tests, which makes them fail randomly; assert structure, enums and key facts.
- C. Calling a paid hosted model in every unit test run, slowing builds and costing money.
Show answer
B. Avoid this approach: Asserting exact model wording in tests, which makes them fail randomly; assert structure, enums and key facts. The lesson recommends: Hide AI calls behind services and mock them for fast, deterministic unit and web tests.
Interview questions
How would you apply the guidance in Layer 1: Isolate the Model Behind Your Own Service to a real task, and what failure would you guard against?
Show answer
Hide AI calls behind services and mock them for fast, deterministic unit and web tests. Apply this in Layer 1: Isolate the Model Behind Your Own Service; the example Controller test with the AI service mocked shows one way to do it. Avoid Asserting exact model wording in tests, which makes them fail randomly; assert structure, enums and key facts. and explain how your implementation or design prevents that failure.
Key Points to Remember
- Hide AI calls behind services and mock them for fast, deterministic unit and web tests.
- Use Testcontainers OllamaContainer with @ServiceConnection for realistic, free integration tests.
- RelevancyEvaluator and FactCheckingEvaluator measure answer quality and detect hallucinations.
- Spring AI emits Micrometer metrics and traces for latency, token usage, tools and vector searches.
- Keep prompt/completion content out of production logs and monitor cost per feature.
Practice in the playground
Write and run your own code for this topic, or start from the exercise or an example. Runs plain Java 17. Frameworks such as Spring, and databases, are not available here. Your code for each topic is kept in this browser.
Run the examples
Change an input, predict the result, then compare it with the output. Explain why the result changes.
These examples need a JDK project with their libraries on your computer, so they can be copied but not run here. Codelab runs Python, FastAPI backends with SQLite, plain Java 17 and HTML/CSS/JavaScript.
Written by
WebNest Studio Engineering Team
Practical lessons from developers working with Java, Python, React, databases and production systems.
Led by Vansh Duggal, Founder, WebNest Studio · Full-stack & software engineering