Building Better AI Applications with LLM Tracing using Opik

Andreas Nigg
Andreas Nigg · Co-Founder, Pondhouse Data
Published Updated 9 min read
Building Better AI Applications with LLM Tracing using Opik
On this page

Opik is an open-source debugging tool created by Comet for tracing, evaluating and monitoring LLM applications. It shows you how your LLM applications actually behave, both during development and in production. In this guide we self-host Opik with Docker Compose or Kubernetes, trace OpenAI, PydanticAI and LangChain calls (adding tracing to an OpenAI client takes one line), annotate traces, track costs, and set up automatic LLM-as-a-judge scoring for production traffic.

Update (November 2026): LangChain 1.0 moved legacy chains like the LLMChain used below into the separate langchain-classic package (see the LangChain v1 migration guide). PydanticAI also changed how you pass your own client: it now goes through OpenAIProvider(openai_client=...) (see the PydanticAI OpenAI docs).

Opik traces list for a text-to-SQL project with feedback scores

Common challenges when developing LLM applications

If you're building AI applications with Large Language Models, you've likely run into some common frustrations: unexpected outputs, high API costs, and trouble figuring out why your application isn't performing as expected. These issues get even more complex once your application moves towards agentic workflows, where the AI interacts with other AI systems or humans in a more dynamic, context-aware way. Our blog series about advanced agent building from scratch shows one such example. The typical ways these systems fail are listed in our guide to building high-quality AI agent systems.

When building LLM applications, developers typically face these issues:

  • Not knowing why certain prompts fail while others succeed
  • Difficulty tracking token usage and associated costs
  • Uncertainty whether retrieved context (in RAG systems) is actually relevant
  • No clear way to measure application performance over time
  • Limited visibility into production behavior

How Opik helps

Opik has three main features. Tracing records every interaction with your LLM, including prompts, responses and metadata. Evaluation lets you test your application systematically with automated checks. Monitoring shows your application's behavior in production in easy-to-read dashboards.

Opik works with common frameworks like LangChain, OpenAI and LlamaIndex and fits into your existing development workflow.

This guide walks through the concrete steps to use Opik for debugging and improving your LLM applications, with examples you can implement today.

What is LLM tracing?

"Tracing" is a heavily used term in the industry, so let's clarify what it means for LLM applications. LLM tracing is a way to record and analyze every interaction between your application and a language model. Think of it as logging for traditional applications, but designed for LLM-based systems.

When your AI application isn't working as expected, you need to know:

  • What prompt was sent to the model
  • What response came back
  • What context or history influenced the interaction
  • How long the request took
  • How many tokens were used (and the associated cost)

Without tracing, debugging these issues is guesswork. A trace, meaning a log of each interaction, might look like this:

1# Example trace data
2{
3 "timestamp": "2024-03-15T10:30:45Z",
4 "model": "gpt-3.5-turbo",
5 "prompt": "Summarize this article about...",
6 "completion": "The article discusses...",
7 "tokens_used": 147,
8 "latency_ms": 850,
9 "metadata": {
10 "temperature": 0.7,
11 "context_length": 2048
12 }
13}

In short, LLM tracing is fancy wording for good old application logging.

Setting up Opik for LLM tracing

There are two ways to use Opik:

  1. Use the fully managed Comet LLM evaluation platform. It's a full platform for all your LLM logging and observability needs, and it includes a managed version of Opik.

  2. Use the open source, self-hosted, Apache-2 licensed Opik project. This is the version we focus on in this guide.

Installing Opik

You can install the self-hosted Opik with either Docker Compose or Kubernetes.

Make sure you have docker installed. For Kubernetes, you'll need a running Kubernetes cluster as well as helm and kubectl.

Running Opik using docker compose

  1. Get the Opik docker compose file. The easiest way is to clone the repo:

    1git clone https://github.com/comet-ml/opik.git
  2. Change into the opik directory and start the services:

    1 cd opik/deployment/docker-compose
    2 docker-compose up -d

Opik should now be running on http://localhost:5173. Either use it on the same server as your application, or expose the port to the outside world with a web server like nginx or caddy.

Note: You might want to look at the docker-compose.yml file and adjust the configuration to your needs. In particular, check the volume mounts for clickhouse, mysql and redis. These are docker volumes; depending on your needs, you might want to change them to folder mounts.

Running Opik using kubernetes

  1. Get the Opik helm chart and install it as follows:

    1 helm repo add opik https://comet-ml.github.io/opik/ && helm repo update
    2
    3 VERSION=latest; helm upgrade --install opik -n opik --create-namespace opik/opik \
    4 --set component.backend.image.tag=$VERSION --set component.frontend.image.tag=$VERSION

The installation provides a service svc/opik-frontend on port 5173. To forward the service to your local machine, use the following command:

1kubectl port-forward svc/opik-frontend 5173:80 -n opik

Note: Whichever method you choose, Opik itself does not provide any authentication: the application is freely accessible. Put an authentication provider in front of it, or run it in a secured network.

If everything went well, you can now open the Opik frontend at http://localhost:5173 and see this screen:

Opik welcome screen

With the platform running, install the Opik Python SDK to start tracing your LLM applications. Install it in the environment of the application you want to trace.

1pip install opik

Using Opik in development

Now let's use Opik in the development workflow to trace an LLM application. Start by initializing the Opik SDK on your development machine:

1# Select option 'self-hosted'
2opik configure

Tracing your LLM application with Opik

The simplest use case for Opik is tracing your LLM application: seeing every interaction with an LLM, including the prompt, the response and any metadata associated with the interaction.

For this guide, we assume an OpenAI LLM. Adding Opik tracing is as simple as wrapping the OpenAI client:

1from opik.integrations.openai import track_openai
2from openai import OpenAI
3
4client = OpenAI()
5client = track_openai(client)

That's it. From now on, Opik traces every call to the OpenAI client. For example, if we run this code:

1response = client.chat.completions.create(
2 model="gpt-4o",
3 messages=[
4 {"role":"user", "content": "How are you?"}
5 ]
6)

We get this first trace in Opik:

Our first Opik trace with input, output, tokens and cost

We could almost end the article here, because this is and will remain the most important use of tracing. We see the question, the answer, the model used, the tokens used, the latency and some metadata.

Most frameworks also accept a custom OpenAI client as a parameter, so you can use Opik tracing with any framework that uses the OpenAI client.

For example, to use Opik tracing with the new PydanticAI agent framework, use their custom OpenAI agent feature and pass the wrapped client to the agent. PydanticAI expects the async AsyncOpenAI client.

1from pydantic_ai import Agent
2from pydantic_ai.models.openai import OpenAIModel
3from opik.integrations.openai import track_openai
4from openai import AsyncOpenAI
5
6client = AsyncOpenAI()
7client = track_openai(client)
8
9model = OpenAIModel('gpt-4o', openai_client=client)
10agent = Agent(model)

If you have multiple function calls (like tools) that interact with the LLM, you can use the opik.track decorator to trace these calls.

1import opik
2from pydantic_ai import Agent
3from pydantic_ai.models.openai import OpenAIModel
4from opik.integrations.openai import track_openai
5from openai import AsyncOpenAI
6
7client = AsyncOpenAI()
8client = track_openai(client)
9
10model = OpenAIModel('gpt-4o', openai_client=client)
11agent = Agent(model)
12
13@opik.track
14def search_wikipedia_tool(query):
15 # Some code to search wikipedia
16 return response
17
18@opik.track
19async def llm_call(message):
20 response = await client.chat.completions.create(
21 model="gpt-4o",
22 messages=[{"role": "user", "content": message}]
23 )
24 return response.choices[0].message.content

This logs the function name as well as model details and function timings.

Opik integrations for LangChain and other frameworks

Opik makes this even more comfortable with a long list of native integrations for popular tools and frameworks.

Here is how to integrate Opik with Langchain:

1from langchain.chains import LLMChain
2from langchain_openai import OpenAI
3from langchain.prompts import PromptTemplate
4from opik.integrations.langchain import OpikTracer
5
6# That's the only additional line we need.
7opik_tracer = OpikTracer()
8
9# Create the LLM Chain using LangChain
10llm = OpenAI(temperature=0)
11prompt_template = PromptTemplate(
12 input_variables=["name"],
13 template="Hi, I'm {name}. How are you today?"
14)
15llm_chain = LLMChain(llm=llm, prompt=prompt_template)
16response = llm_chain.run("Andreas", callbacks=[opik_tracer])

All we need is one additional line. We create an OpikTracer and pass it as a callback to the run method of the LLMChain. Opik then automatically traces the interaction with the LLM via Langchain.

For a full list of integrations, see their outstanding documentation.

Manual tracing if none of the integrations fit

Say you use a framework that Opik doesn't support and that doesn't let you pass a custom LLM client. With all the new agent frameworks popping up, that can happen. In this case, you can always trace the interactions manually.

Take a simple agent application, similar to the one in our previous blog post, built with an agent framework that doesn't accept a custom LLM client. Here we can add the opik.track function decorator to any method that interacts with the LLM.

1import opik
2
3@opik.track
4def search_wikipedia_tool(query):
5 # Some code to search wikipedia
6 return response
7
8@opik.track
9def query_database_tool(query):
10 # Some code to query a database
11 return response
12
13@opik.track
14def chat_with_llm_tool(message):
15 # Some code to chat with an LLM
16 return response

This won't log the exact prompts and responses or the LLM costs, but we at least get an idea of tool usage and latency. In this case, Opik traces the inputs, outputs and timings of the functions.

Annotating traces with feedback scores

Opik also lets you annotate LLM traces with feedback scores. In the UI, navigate to the trace you want to annotate and click the Annotate button in the top-right corner.

The first time you annotate a trace, you'll be asked to create a Feedback definition. Fill out the feedback definition form and click Create feedback definition.

Creating a feedback definition

Now you can annotate the trace with the feedback category you just created by selecting it from the dropdown.

Annotating a trace

Tracking LLM costs

For OpenAI and Gemini models, Opik automatically tracks the cost of each LLM call and shows a cost breakdown in the trace view as well as in the project metrics overview.

Estimated costs overview

This is especially useful when you work with a budget or need to optimize your LLM usage.

Cost tracking isn't available for all LLM models, but token usage and latency are always tracked.

Setting the Opik project

All examples so far used the default project, but you will most likely create multiple projects. You can set the project to trace into with a simple function parameter, both when wrapping the client and on the track decorator.

1client = OpenAI()
2client = track_openai(client, project_name="opiktest")
3
4@opik.track(project_name="opiktest")
5def some_function(input):
6 return input

To change the project for all traces, set the project name as an environment variable:

1os.environ["OPIK_PROJECT_NAME"] = "opiktest"

LLM monitoring and tracing in production with Opik

Once you have a well-performing LLM application, you need to make sure it also behaves in production.

Trace logging in production

Nothing new here. Opik is designed to be highly scalable, so use the same setup as in development.

Monitoring dashboards

All trace metrics (number of traces, tokens and feedback scores) are available in a dashboard that gives you a high-level overview of your application. You can spot whether your LLM costs are running away or your application isn't performing as expected.

Opik Dashboard (kindly taken from https://www.comet.com/docs/opik/production/production_monitoring)

Online evaluation with LLM as a judge

In production, feedback scores need to be created for every trace. You can't do that manually, because there are far too many traces. Opik can create feedback scores automatically, using an LLM as the scorer. (This is called "LLM as a judge".)

Say you built a RAG system using Azure AI Search. You'd want to know whether your LLM generates relevant answers and doesn't hallucinate. (For offline evaluation of RAG pipelines, see our guide on evaluating RAG performance with Ragas.)

Opik provides 3 predefined LLM-as-a-judge validation prompts:

  • Hallucination checks whether the LLM output contains hallucinated information.
  • Moderation checks whether the LLM output contains offensive content.
  • Answer Relevance checks whether the LLM output is relevant to the given context.

You can also create your own validation prompts.

  1. First, configure an LLM provider to act as the judge. Navigate to the AI Provider configuration and add a new AI provider.

  2. Then navigate to your project, select the Rules tab and create a new rule.

    Rules tab of an Opik project before the first rule is created

  3. Fill out the form: select the LLM judge model and the prompt you want to use, and, most importantly, define the variable mappings. Text in two {{ brackets }} is replaced by the actual values from the trace.

    Note: You can use any value in your trace as a variable. Try the metadata parameter of the track decorator to add extra information to your traces.

    Finally, add a score definition.

    Click Create rule and you're done. From now on, every trace is annotated with the feedback score.

Note: Automatic scoring is one of Opik's most powerful features. It validates your LLM output in production and helps you detect issues early.

To sum up: start by wrapping your LLM client with track_openai (or the matching integration) during development, keep the same setup in production, and add an LLM-as-a-judge rule such as Hallucination to score live traces. If you want to compare Opik with another open-source option, read our introduction to Langfuse.


Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms

Further reading

Andreas Nigg

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.