Your Agent Doesn’t Need to Leave the JVM (Mostly)

When a Spring team is asked to “add AI”, the first design move is usually the same. Stand up a Python service next to the backend and build the agent there. Python earned that default. I build agents in Python against the Anthropic API myself, and I will keep doing so for part of the work.

The default still deserves a second look. Spring AI 2.1.0-RC1 shipped on 23 October, a day after Spring Boot 4.2.0-RC1 and Spring Framework 7.1.0-RC1. These are release candidates, not final releases, and I will say where that matters. Even so, they make a different layout easy to justify: the agent lives inside the Spring service that already owns your domain.

This post is a decision guide, not a claim that Java beats Python. It covers where an agent really spends its time, what Spring AI offers today, the same small agent built in both stacks, and a path that uses each language where it is strongest.

The short version

• The model call is the smallest part of an agent’s work. Most of the effort goes into tool calls that reach your own systems.
• Those tools need your transactions, your authorization rules, and your tests. All three already live in your Spring service.
• Prototype, evaluate, and explore in Python. Run the production agent inside the service once its behaviour is stable.
• You do not need a separate Python service just to call a model API.

1. Where an agent actually spends its time

An agent is a loop. The model reads the conversation and asks for a tool. Your code runs the tool. The result goes back to the model, and the cycle repeats until the model answers.

In a typical run, the model contributes a handful of calls. The tools contribute everything else, and they reach into your own systems: the order database, the billing service, the permission model, a payment provider. Time and risk both concentrate there.

The problems that show up are ordinary backend problems:

• A tool needs the caller’s identity and tenant.
• A write must be idempotent, because the model may repeat a call or retry after a timeout.
• A tool fails halfway through.
• An argument is wrong in a way that a validator would reject on any normal endpoint.

A useful way to see this is that every tool is an endpoint whose caller is an untrusted client. That client composes its own arguments from free text. You validate, authorize, and limit it as you would any other caller.

So the real question is not which language suits language models better. It is where the loop sits relative to the code it calls. It can run in the same process, under the same transaction manager and security rules. Or it can run across a network boundary, with a second authentication story to build and maintain.

2. What Spring AI 2.1 gives you

Some context on versions. Spring AI 2.0 is generally available: 2.0.0 shipped on 12 June, and 2.0.2 is the current stable release. The 2.1.0-RC1 candidate adds a few features I will point out. Spring Boot 4.2.0-RC1 and Spring Framework 7.1.0-RC1 are the platform candidates published the same week. This section stays with Spring AI, because that is where the agent-specific pieces live.

Tools as plain methods

You mark a method with @Tool and describe its parameters with @ToolParam. You pass the object to the ChatClient, and the framework generates the schema for the model.

Since 2.0, the loop itself is a component. The ChatClient registers a ToolCallingAdvisor that drives it: the model asks for tools, a ToolCallingManager runs them, and the loop continues until the model replies without a tool call. Memory, observability, and retries compose with the loop through advisor ordering.

Context that never reaches the model

Tools often need data the model must not see or choose, such as a tenant ID or a user ID. Spring AI’s ToolContext carries that data to the tool method. It is not sent to the model. This is the right place for the scoping values your authorization depends on.

Human approval for risky tools (new in 2.1)

A tool can declare requiresConfirmation = true. When the model calls it, the loop pauses and returns the pending calls to your application. A person decides, and your application resumes the conversation with those decisions. The framework keeps no state while it waits, so the pause survives a restart as long as your application stores the conversation.

Structured output

.entity(...) on the ChatClient maps the reply onto a type you define. The documentation is plain about its default: the plain call gives no guarantees. Treat the result like any other input and validate it.

Memory: be careful here

Conversation memory is where the API is moving. The ChatMemory API and its advisor are deprecated since 2.1.0 and will be removed in 3.0.0. Their replacement is Session Memory. It keeps a conversation as an append-only log of events. It compacts history on turn boundaries, so the model never sees a tool call without its result. A repository stores the log, with in-memory and JDBC implementations. That means conversation state can sit in your own database, next to your domain data.

The deprecation is also a reminder of the release-candidate caveat. Pin your versions and keep the framework behind an interface of your own.

Observability

Spring AI publishes Micrometer observations for ChatClient calls and for the underlying model calls, including a token-usage metric. If you already export traces, these spans appear in them. Logging of prompts and completions is off by default, which is the right default when prompts can contain personal data.

The provider is the official SDK

Since 2.0.0-M3, the Anthropic module in Spring AI is a thin adapter over Anthropic’s official Java SDK. If you do not want the framework, you can use that SDK directly from Kotlin or Java. Leaving Python does not require adopting Spring AI.

3. The same small agent, twice

To compare fairly, I picked a deliberately small agent: a support assistant that can look up an order and issue a refund. It has two tools and one system prompt. The code is illustrative, with the domain calls stubbed.

Python, with the Anthropic SDK

import anthropic
import httpx
from anthropic import beta_tool

client = anthropic.Anthropic()
orders = httpx.Client(base_url="https://orders.internal")

SYSTEM = (
    "You help support staff with refunds. "
    "Never refund more than the order total."
)


def forwarded_auth() -> dict[str, str]:
    # Placeholder: whose identity calls the order API?
    ...


@beta_tool
def get_order(order_id: str) -> str:
    """Look up an order by id.

    Args:
        order_id: The order id, for example "A-1042".
    """
    url = f"/orders/{order_id}"
    return orders.get(url, headers=forwarded_auth()).text


@beta_tool
def issue_refund(order_id: str, amount_cents: int) -> str:
    """Refund part or all of an order.

    Args:
        order_id: The order id.
        amount_cents: The amount to refund, in cents.
    """
    response = orders.post(
        f"/orders/{order_id}/refunds",
        json={"amountCents": amount_cents},
        headers=forwarded_auth(),
    )
    return response.text


runner = client.beta.messages.tool_runner(
    model="claude-sonnet-5-5",
    max_tokens=1024,
    system=SYSTEM,
    tools=[get_order, issue_refund],
    messages=[{
        "role": "user",
        "content": "Refund 12.50 on order A-1042. It arrived damaged.",
    }],
)
final = runner.until_done()
print(next(b.text for b in final.content if b.type == "text"))

This is short and readable, and it is exactly what I would write in a notebook. The tool helpers live in the SDK’s beta namespace today. The part to look at is forwarded_auth(). The agent is a separate service, so it needs credentials to call the order API. In practice that is often a service account with wide access, or a token that must be forwarded and checked at every hop.

Kotlin, with Spring AI

private const val SYSTEM =
    "You help support staff with refunds. " +
        "Never refund more than the order total."

@Component
class RefundTools(
    private val orders: OrderService,
    private val refunds: RefundService,
) {
    @Tool(description = "Look up an order, for example A-1042")
    fun getOrder(
        @ToolParam(description = "The order id") orderId: String,
    ): OrderView = orders.view(orderId)

    @Tool(description = "Refund part or all of an order")
    fun issueRefund(
        @ToolParam(description = "The order id") orderId: String,
        @ToolParam(description = "Amount in cents") amountCents: Long,
    ): RefundReceipt = refunds.issue(orderId, amountCents)
}

@Service
class RefundAssistant(
    builder: ChatClient.Builder,
    tools: RefundTools,
) {
    private val chat = builder
        .defaultSystem(SYSTEM)
        .defaultTools(tools)
        .build()

    fun handle(request: String): String? =
        chat.prompt().user(request).call().content()
}

The tools are beans that call the same OrderService and RefundService as your REST controllers. The refunds.issue method carries its own @Transactional boundary and its own business rules. Authorization is whatever your service layer already enforces for the signed-in user.

Two notes on this sample. First, I left out requiresConfirmation = true on issueRefund to keep the code short. For a real refund tool I would set it and handle the pause. Second, thread-bound context such as Spring Security’s holder is available on the synchronous call() path because the loop runs on your request thread. If you stream, or move the work to other threads, check where tools execute and propagate the context deliberately.

Both versions are about the same size. The difference is not the line count. It is what the tool function is allowed to touch, and under whose identity.

4. Where the JVM wins

Transactions and invariants

The best reason to keep tools in the service is that they reuse your domain code with its invariants: optimistic locking, idempotency keys, the outbox, validation. A refund tool that calls RefundService.issue gets the same checks as the refund endpoint.

One caution applies. Do not hold a database transaction open across model calls. They take seconds, and a network call inside a transaction is a reliability problem in any stack. Make each tool call its own short transaction. Because the model can repeat a call, make writes safe to repeat, as in publish reliably, consume idempotently.

Identity and authorization

In the service, the agent can act with the signed-in user’s own permissions. That keeps the blast radius of a bad tool call small. A separate agent service tends to end up with a broad service account, because that is the quickest way to make it work. You still need per-tool authorization and an allow-list of what each agent may call. Moving the loop in-process makes that job easier but does not remove it.

Tests you already have

Tools in the service are ordinary beans. You test them with the integration tests and Testcontainers setup you already run. For the loop, replace the model with a fake that returns scripted tool calls, and keep a few live tests behind a flag. In my experience many agent bugs live in the tools, so much of the testing is the testing you already know how to do.

Operations

One deployable, one pipeline, one on-call rotation, one secrets story. Tool calls appear as spans in the same trace as the HTTP request and the SQL that follows. That connects to observability as part of done. Fewer hops also means fewer places where personal data passes through. The model provider is still a data processor whichever language you use, so the contract and region questions remain.

5. Where Python still wins

Python is the right home for several kinds of work.

• Speed of iteration. A notebook or REPL lets you change a prompt, rerun, and look at the result in seconds. Hot reload on the JVM helps, but it is not the same loop.
• Evals. Datasets, graders, and analysis are Python-first. If you measure agent quality seriously, you will want these tools.
• The newest features. New API capabilities show up first in Anthropic’s own SDKs, and the Python SDK ships often. Framework adapters follow. Spring AI’s Anthropic module wraps the Java SDK, so there can be a gap.
• The ML ecosystem. Embeddings, rerankers, local models, and data wrangling are simply easier there.

6. A hybrid layout

You do not have to choose once for the whole organization. This sequence works well for a single agent.

1. Prototype in Python. Use stubbed tools or a staging API. Iterate on the prompt and on tool descriptions. Build a small eval set of realistic cases.
2. Freeze the contract. The tool names, descriptions, parameter schemas, system prompt, and eval set are the output of this phase.
3. Move the tools, then the loop. Implement the tools as methods over your existing domain services. The loop is thin by comparison.
4. Keep the evals in Python. Point the same harness at the service’s agent endpoint. A regression check then covers the production stack, and Python remains the lab.
5. Decide per agent. Some agents will stay in Python because they depend on its libraries. That is fine.

A quick way to place an agent:

Before you start

Three cautions. First, Spring AI 2.1 and Boot 4.2 are release candidates. The memory change shows that APIs are still moving, so pin versions, read the release notes at final, and put the framework behind an interface of your own. Second, keep tool classes light on framework types, so they port easily if you change course. Third, test the loop with real data protection in mind: keep prompt logging off, and know what each tool returns to the model.

If someone asks you to add AI, the first question is not which language to use. It is what the agent will touch, and as whom. Answer that, and the layout mostly follows.

Sources

• Spring AI 2.1.0-RC1 release and the Spring AI reference
• Spring Boot 4.2.0-RC1 release
• Spring Framework 7.1.0-RC1 release
• Anthropic Python SDK and Anthropic Java SDK