Running Local Models with Ollama

Hosted models are powerful, but they are not always an option: some data may not leave your network, API costs add up during development, and you may want to work offline. Ollama runs open-weight models — Llama, Mistral, Gemma, Qwen, DeepSeek, Phi and many more — locally on your laptop or on your own servers, with a simple HTTP API.

Spring AI supports Ollama for both chat and embeddings, and because your code uses the portable ChatClient and EmbeddingModel APIs, you can develop against a local model and switch to a hosted one in production (or the reverse) with configuration alone. This lesson covers installing Ollama, connecting Spring Boot to it, running it with Docker Compose, and using several providers in one application.

Installing Ollama and Pulling Models

Download Ollama from ollama.com for Windows, macOS or Linux, or run the official ollama/ollama Docker image. Pull a model with ollama pull llama3.2 and an embedding model with ollama pull nomic-embed-text. Ollama serves its API on http://localhost:11434.

Choose model size to fit your hardware: 1–4B parameter models run on most laptops; 7–8B models want 8–16 GB of RAM or a GPU; larger models need serious GPUs. Smaller models are faster but weaker at reasoning, tool calling and structured output.

Configuring Spring AI for Ollama

Add spring-ai-starter-model-ollama, then set spring.ai.ollama.base-url, spring.ai.ollama.chat.model and spring.ai.ollama.embedding.model. Spring AI can pull missing models automatically at startup with spring.ai.ollama.init.pull-model-strategy=when_missing, which is convenient for development and CI.

Docker Compose and Testcontainers

Spring Boot's Docker Compose support detects an ollama/ollama service in compose.yaml, starts it with the application, and configures the base URL automatically through a service connection. Testcontainers offers an OllamaContainer for integration tests, also connected with @ServiceConnection.

Using Several Model Providers

With more than one model starter on the classpath, the auto-configured ChatClient.Builder becomes ambiguous. Disable it with spring.ai.chat.client.enabled=false and build a ChatClient for each ChatModel bean yourself — for example a local Ollama model for cheap classification and a hosted model for complex reasoning.

Examples

# After installing from ollama.com
ollama pull llama3.2
ollama pull nomic-embed-text
ollama list

# Quick test from the terminal
ollama run llama3.2 "Explain Spring Boot starters in one sentence."

NAME                       SIZE
llama3.2:latest            2.0 GB
nomic-embed-text:latest    274 MB

Spring Boot starters are dependency bundles that pull in everything needed for a feature, like web or JPA, with compatible versions.

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-ollama</artifactId>
</dependency>

# application.yml
spring:
  ai:
    ollama:
      base-url: http://localhost:11434
      init:
        pull-model-strategy: when_missing
      chat:
        model: llama3.2
        temperature: 0.2
      embedding:
        model: nomic-embed-text

INFO  Pulling model llama3.2 (strategy when_missing)... already available
INFO  Started WebnestAiApplication in 2.1 seconds

# compose.yaml in the project root
services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
volumes:
  ollama-data:

<!-- pom.xml: Spring Boot starts compose.yaml automatically in development -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-docker-compose</artifactId>
    <optional>true</optional>
</dependency>

INFO  Using Docker Compose file compose.yaml
INFO  Container project-ollama-1 Started
(spring.ai.ollama.base-url is set automatically from the running container)

# application.yml
spring:
  ai:
    chat:
      client:
        enabled: false   # we create ChatClients ourselves

@Configuration
public class MultiModelConfig {

    @Bean
    ChatClient localClient(OllamaChatModel ollama) {
        return ChatClient.builder(ollama)
            .defaultSystem("Classify text. Reply with a single word.")
            .build();
    }

    @Bean
    ChatClient cloudClient(OpenAiChatModel openAi) {
        return ChatClient.builder(openAi)
            .defaultSystem("You are an expert Spring Boot architect.")
            .build();
    }
}

@Service
public class QuestionRouter {

    private final ChatClient localClient;
    private final ChatClient cloudClient;

    public QuestionRouter(ChatClient localClient, ChatClient cloudClient) {
        this.localClient = localClient;
        this.cloudClient = cloudClient;
    }

    public String answer(String question) {
        String kind = localClient.prompt()
            .user("Is this question SIMPLE or COMPLEX? " + question)
            .call().content().trim();
        return kind.startsWith("COMPLEX")
            ? cloudClient.prompt().user(question).call().content()
            : localClient.prompt().system("Answer briefly.").user(question).call().content();
    }
}

answer("What port does Spring Boot use by default?")                     -> (local) 8080.
answer("Design a multi-tenant schema strategy for a SaaS on Spring Boot") -> (cloud) There are three common strategies...