What Is Langfuse? Open-Source LLM Observability, Self-Hosted

Andreas Nigg
Andreas Nigg · Co-Founder, Pondhouse Data
Published Updated 13 min read
What Is Langfuse? Open-Source LLM Observability, Self-Hosted
On this page

Langfuse is an open-source LLM engineering platform that records what your LLM application actually does. It captures the full context of every execution, including prompts, completions, retrieval steps and tool usage, and adds evaluations, datasets and prompt management on top. With that, you can debug a hallucinating chatbot or a misbehaving agent from data instead of guesswork.

In this post we self-host Langfuse with Docker Compose and walk through its main open-source features: tracing OpenAI calls, grouping them into traces, sessions and users, the dashboard, manual evaluations, datasets and prompt management. We covered a similar tool, Opik, in the past, so this is a good chance to compare.

Update (November 2026): Langfuse's Python SDK v3 replaced the langfuse.decorators module used below with from langfuse import observe, get_client, and langfuse_context.update_current_trace(...) became get_client().update_current_trace(...). Langfuse has since released v4 as well. The v2 code in this post still works with langfuse<3; see the upgrade guide for the new API.

Why AI applications and agents are hard to debug

Building AI applications, especially ones with autonomous agents, brings problems that traditional software engineering tools weren't designed for.

The black box problem

In conventional software, the same input reliably produces the same output. LLMs are unpredictable. A minor change in phrasing can change model behavior dramatically, and even identical prompts can produce different responses depending on sampling parameters. Without specialized tooling, it is nearly impossible to understand what happens inside your application.

Untangling complex execution paths

Modern AI applications rarely consist of simple prompt-response pairs. Instead, they involve sequences of:

  • Multiple LLM calls with different contexts
  • Retrieval operations pulling data from various sources
  • Tool usage and API interactions
  • Conditional branching based on model outputs
  • Parallel processing and asynchronous operations

When something goes wrong, and it will, pinpointing where the failure occurred is hard. Did the retrieval system return irrelevant context? Did the agent choose the wrong tool? Did the core reasoning step contain a logical flaw? Without visibility into each step, debugging turns into frustrating guesswork.

Measuring the quality of LLM outputs

Traditional software relies on unit tests with binary pass/fail results. The quality of LLM output is often subjective, and "correctness" spans a spectrum. Is a response accurate but not helpful? Helpful but not accurate? Both, but poorly formatted? Defining success metrics is hard, and the lack of standardized evaluation frameworks makes it harder.

LLM applications degrade in production

LLM applications often degrade in production in ways that development environments never reveal:

  • Users ask questions developers never anticipated
  • External APIs and tools change their behavior
  • The underlying models themselves receive updates
  • Edge cases emerge that weren't covered in testing

Without continuous monitoring and evaluation, an application that worked perfectly during development can quickly become unreliable in the real world.

Debugging AI agents

For agent-based systems, these problems multiply. Agents decide on their own which actions to take, which creates a huge space of possible execution paths. When an agent gets stuck in a loop, picks the wrong tools or fails to reach its objective, reconstructing what happened and why is very difficult.

Traditional logging of inputs and outputs is not enough. You need tracing of the agent's decision-making process: its reasoning steps, tool selections, intermediate results and the full context behind each choice. (We cover more of this in building high-quality AI agent systems.)

The iteration bottleneck

Together, these problems slow down iteration. Without clear visibility into what's happening, improvements are based on speculation rather than data. Teams end up asking:

  • Which prompts need refinement?
  • Where should we focus our optimization efforts?
  • Are our model choices appropriate for each task?
  • How do we measure whether changes are actually improvements?

Without answers, the development cycle becomes slow and expensive. LLM applications need tooling built for these problems, with observability, evaluation and systematic improvement at every stage of development.

What is Langfuse?

Langfuse is an open-source LLM engineering platform with the tools to debug, analyze and systematically improve AI applications. At its core, it is an observability layer that captures the complete execution context of language model applications, from simple chatbots to complex autonomous agents.

Unlike general-purpose logging or monitoring tools, Langfuse is built for the LLM development problems described above. Traces are first-class citizens, so you can visualize the hierarchical, often non-linear execution paths of modern AI applications. Every LLM call, retrieval operation, tool usage and intermediate reasoning step can be captured with full context, which gives you a detailed record of what happened during execution.

The platform has four integrated components that cover the LLM development lifecycle:

  1. Observability: lightweight SDKs and framework integrations record prompts, completions, metadata and the relationships between execution steps.

  2. Prompt management: store, version and iterate on prompts centrally, decoupled from application code, so you can experiment without redeploying.

  3. Evaluations: assess output quality with LLM-as-a-judge, user feedback, manual labeling or custom evaluation pipelines.

  4. Datasets: build benchmarks from production data, test sets or synthetic examples to evaluate application performance across versions.

What sets Langfuse apart from other tools is how tightly these components are integrated. A problematic trace in production can be instantly added to a dataset, evaluated with an LLM-as-a-judge, addressed with prompt improvements, and verified before deployment, all within the same platform.

Langfuse supports Python, JavaScript/TypeScript and popular frameworks like LangChain, LlamaIndex, and the OpenAI SDK. It's fully open-source (MIT license) and can be self-hosted or used as a managed service through Langfuse Cloud, with both EU and US hosting options available for teams with specific compliance requirements.

How to run Langfuse

Langfuse documents several ways to run it yourself:

Note: If you want to know more about the Langfuse architecture, you can check the Langfuse architecture overview page.

For a test setup, we use the Docker Compose configuration in this post.

  1. Clone the Langfuse repository:
1git clone https://github.com/langfuse/langfuse.git
2cd langfuse
  1. Open the docker-compose.yml file and set all the environment variables marked with # Change me to your own values. These are secrets used to set up and connect the various services. You can choose any values on initial setup.

  2. Run the following command to start the services:

1docker compose up -d

That's it: we have a running Langfuse instance with the following services:

  • langfuse worker
  • langfuse web
  • clickhouse
  • minio
  • redis
  • postgres

Nothing out of the ordinary, but keep in mind that Langfuse needs quite a few services. For production, you need a proper environment with all of these services configured and running.

Langfuse features

With Langfuse running, let's look at the features it provides.

Note: We'll stick to the features available in the open-source version of Langfuse. To get an idea of all the features available, see the Langfuse pricing page.

Tracing

The main feature of Langfuse is, obviously, tracing: it captures the execution of your LLM application and visualizes it.

Langfuse makes this easy:

  1. Install the required packages (in our case, we use an OpenAI model):

    1pip install langfuse openai
  2. In your Langfuse instance, create a new project and get the secret key and public key.

    In your organization overview, click "New Project" and fill out the form. Creating a new project in the Langfuse setup wizard

    On the next screen, click "Create API Key". This reveals the public and secret key. Copy both. Creating API keys in the Langfuse setup wizard

  3. Create a .env file:

    1LANGFUSE_SECRET_KEY="sk-lf-..." # private key
    2LANGFUSE_PUBLIC_KEY="pk-lf-..." # public key
    3LANGFUSE_HOST="https://<your-langfuse-instance>.com" # your langfuse instance
  4. In your project, make sure the environment variables are loaded. Then import the Langfuse wrapper of your AI SDK instead of the SDK itself. For OpenAI, it looks like this:

    1from langfuse.openai import openai # OpenAI integration
    2completion = openai.chat.completions.create(
    3 name="my-completion",
    4 model="gpt-4o",
    5 messages=[
    6 {"role": "system", "content": "You are a helpful assistant."},
    7 {"role": "user", "content": "What is the capital of France?"}
    8 ],
    9)

    Note: That's all it takes to trace your LLM calls. Every call to OpenAI is now logged, including latencies and costs. However, each call is a separate trace. The next step shows how to nest them where you want them.

Our first trace in Langfuse

Combining LLM calls into a single trace

If your application makes several LLM calls per request, you want them grouped together. A chat agent, for example, likely needs one call to extract relevant information, another to decide which tools to use and a final one to write the answer. Grouping these calls gives you an overview of a single user interaction.

Langfuse offers two options for that:

  1. Add a custom trace ID to the OpenAI LLM calls.

    1import uuid
    2
    3my_trace_id = str(uuid.uuid4()) # create a custom trace id
    4
    5completion = openai.chat.completions.create(
    6 name="my-completion",
    7 model="gpt-4o",
    8 messages=[
    9 {"role": "system", "content": "You are a helpful assistant."},
    10 {"role": "user", "content": "What is the capital of France?"}
    11 ],
    12 trace_id=my_trace_id,
    13)

    Add the same trace ID to all LLM calls you want to group. In most cases, you create the custom trace ID at the beginning of the user interaction.

    This is the easiest way to get a good picture of how your application works, but adding the trace_id to every LLM call is cumbersome. You also don't see in which step or function an LLM call was made.

  2. The @observe decorator solves both issues. Langfuse provides this decorator to wrap the functions in which you run your LLM calls.

    Put the decorator on top of your functions, and all LLM calls inside them are grouped together.

    1 from langfuse.decorators import observe
    2 from langfuse.openai import openai # OpenAI integration
    3
    4 @observe()
    5 def call_llm():
    6 return openai.chat.completions.create(
    7 model="gpt-4o",
    8 messages=[
    9 {"role": "system", "content": "You are a helpful assistant."},
    10 {"role": "user", "content": "What is the capital of France?"}
    11 ],
    12 ).choices[0].message.content
    13
    14 @observe()
    15 def main():
    16 return call_llm()
    17
    18 main()

    The result is a single trace that shows which functions were called and when, with the LLM calls nested one level deeper.

    Note: Add @observe to every function you want traced, across the whole call hierarchy. If main() calls do_something() and do_something() calls call_llm(), you need to decorate all three functions to get the complete picture.

    If you only decorate main() and call_llm(), you get two separate traces, one for main() and one for call_llm() (with the LLM call as a child). So make sure @observe is added to all functions in your call hierarchy.

    A trace with multiple LLM calls

    This view shows the tokens used, the latency and the cost of each LLM call at a glance.

Trace timeline

The "Timeline" toggle in the traces view switches to a timing-based overview. It shows what took the most time, where the bottlenecks are, where you have room for parallelization and where you can optimize your code.

In streaming scenarios, you even see the time to first token, which is very useful for chat applications.

Trace timeline

Sessions

In a chat agent, users rarely send just one message. They ask several questions and follow-ups in the same chat context.

You could group all of them in one trace, but that rarely makes sense. Most of the time you want to monitor and analyze a single question, which by definition is a trace. You still want to see how the full conversation went, though.

For this, Langfuse offers Sessions. A Langfuse session groups these traces together and gives you a simple replay of the entire interaction.

Instrumenting sessions is simple. Use langfuse_context to update the current trace with the session ID. (Instead of chat-id below, use your actual chat identifier. Most chat SDKs provide a chat ID.)

1from langfuse.decorators import langfuse_context, observe
2from langfuse.openai import openai
3
4@observe()
5def call_llm():
6 session_id = "chat-id"
7 langfuse_context.update_current_trace(
8 session_id=session_id,
9 )
10
11 completion = openai.chat.completions.create(
12 name="my-completion",
13 model="gpt-4o",
14 messages=[
15 {"role": "system", "content": "You are a helpful assistant."},
16 {"role": "user", "content": "What is the capital of France?"}
17 ],
18 )
19
20call_llm()

If you don't use the @observe decorator, add the session ID directly to the call:

1 completion = openai.chat.completions.create(
2 name="my-completion",
3 model="gpt-4o",
4 messages=[
5 {"role": "system", "content": "You are a helpful assistant."},
6 {"role": "user", "content": "What is the capital of France?"}
7 ],
8 session_id="chat-id",
9 )

Users

With traces and sessions in place, the last grouping level is users.

Note: A "user" doesn't have to be a single identifiable person. It can also be an agent or an organization. In Langfuse, a user is just a collection of traces and sessions. In general, make sure to respect privacy and data protection laws when using Langfuse.

To add user identifiers to your trace, use the same pattern as for sessions, with the user_id parameter.

1from langfuse.decorators import langfuse_context, observe
2from langfuse.openai import openai
3@observe()
4def call_llm():
5 user_id = "user-id"
6 langfuse_context.update_current_trace(
7 user_id=user_id,
8 )
9
10 completion = openai.chat.completions.create(
11 name="my-completion",
12 model="gpt-4o",
13 messages=[
14 {"role": "system", "content": "You are a helpful assistant."},
15 {"role": "user", "content": "What is the capital of France?"}
16 ],
17 )

Langfuse then shows all users, what they consumed and how much they cost you.

User overview. Source: https://langfuse.com/docs/observability/features/users

More tracing features

  • Trace metadata: add custom metadata to traces for better organization and filtering.

    1 # Update trace metadata from anywhere inside call stack
    2 langfuse_context.update_current_trace(
    3 metadata={"key":"value"}
    4 )
    5
    6 # Update observation metadata for current observation
    7 langfuse_context.update_current_observation(
    8 metadata={"key": "value"}
    9 )
  • Trace tags: categorize traces with tags, which makes overviews and filtering much easier.

    1 langfuse_context.update_current_trace(
    2 tags=["tag-1", "tag-2"]
    3 )

    Filtering traces by tag in Langfuse

  • Multi-modal tracing: capture and visualize multi-modal data, such as images or audio, alongside text traces.

    Multi-Modal Trace. Source: https://langfuse.com/docs/observability/features/multi-modality

  • Sampling: if you have a lot of users (and we hope you do), you probably don't want to trace every single interaction. Set the environment variable LANGFUSE_SAMPLE_RATE to a value between 0 and 1 to sample a percentage of traces. For example, 0.1 will sample 10% of all traces.

    1os.environ["LANGFUSE_SAMPLE_RATE"] = '0.5'
  • Trace sharing: share traces with your team or external stakeholders for collaboration and debugging. Click the (slightly too small) "share" icon and hit "Share" to get a public link to the trace.

    Sharing a trace via public link in Langfuse

  • Native integrations: tracing LLM calls is often enough, but Langfuse also offers dozens of integrations with other tools, such as:

    These integrations trace a framework's inner workings better than the plain LLM call integration can.

    (For completeness: Langfuse is also fully OpenTelemetry compatible.)

Dashboard

The Langfuse dashboard gives you a central view of all traces, sessions and user interactions. It shows how your application and its performance develop, and helps you spot degradations over time. It includes trace counts, model costs per model, scores, and latency percentiles for traces, generations and spans.

A picture or two say more than a thousand words:

Langfuse dashboard with trace counts, model costs and model usage

Langfuse dashboard with trace, generation, span and model latencies

Evaluations

Langfuse evaluations let you assess the quality of your LLM outputs systematically. Below, we score traces by manual annotation. (For automated RAG metrics, see our post on evaluating RAG pipelines with Ragas.)

In your project settings, click Scores/Evaluations and create a new score config. A score config defines how you want to evaluate your traces (e.g. with numbers or labels).

Creating a new score config

Now in your trace view, click the Annotate button (the small chat bubble) and select the score config you want to use.

Annotate a trace: Select score config

Enter your score (or select the label, depending on your score config) and optionally add a comment. Click Save to save your score.

Annotate a trace: Add score

The Tracing → Scores menu has an overview of all your scores. With filtering and sorting, you can see whether your scores are degrading or improving over time.

Scores overview

Datasets

Langfuse Datasets are test sets and benchmarks for evaluating the performance of your LLM application.

Use cases (according to the Langfuse documentation):

  • Continuous improvement: Create datasets from production edge cases to improve your application
  • Pre-deployment testing: Benchmark new releases before deploying to production
  • Structured testing: Run experiments on collections of inputs and expected outputs
  • Flexible evaluation: Add custom evaluation metrics or use LLM-as-a-judge
  • Integrates well: Works with popular frameworks like LangChain and LlamaIndex

Datasets are a bit out of scope for this monitoring-focused post, so I'll just link the rather good Langfuse Dataset documentation.

Prompt management

The last feature I want to cover is prompt management, a lovely addition to the platform.

In the Prompts menu, you can create prompts and put them under version control. This makes it easy to iterate on your prompts and track changes over time.

Click Create prompt and fill out the form. It should be self-explanatory.

Creating a new prompt

To create a new version, open the prompt and click "New" in the "Versions" tab. You get a nice git-like version history.

Prompt version history

The Metrics tab shows the usage of the prompt over time and what it cost. If you use the Evaluations feature described above, you also see the scores of the prompt and its versions.

Versioning helps when you write prompts by hand. To have prompts optimized automatically against a metric and a few examples, see our DSPy tutorial.

To use these prompts in your code:

1chat_prompt = langfuse.get_prompt("rag-chat-prompt", type="chat")
2
3# If you need to add variables to the prompt
4compiled_chat_prompt = chat_prompt.compile(rag_question="What is...", rag_context="...")

To link the prompt to a trace, you guessed it, add the prompt to the @observe-decorated function:

1from langfuse.decorators import langfuse_context, observe
2
3@observe()
4def main():
5 prompt = langfuse.get_prompt("rag-chat-prompt", type="chat")
6
7 langfuse_context.update_current_observation(
8 prompt=prompt,
9 )
10
11main()

Note: Langfuse offers caching for the prompts, see here.

Summary

Langfuse gives you tracing, sessions, user tracking, a cost and latency dashboard, manual evaluations, datasets and prompt versioning in one open-source package. Getting started takes a docker compose up -d and a changed import (from langfuse.openai import openai). From there, add @observe to your call hierarchy and session and user IDs to your traces. Self-hosting means running six services, including ClickHouse, MinIO, Redis and Postgres, so plan for that before you take it to production.

Interested in building high-quality AI agent systems?

We prepared a comprehensive guide based on cutting-edge research for how to build robust, reliable AI agent systems that actually work in production. This guide covers:

  • Understanding the 14 systematic failure modes in multi-agent systems
  • Evidence-based best practices for agent design
  • Structured communication protocols and verification mechanisms

Further reading

Andreas Nigg

About the author

Andreas Nigg

Co-Founder, Pondhouse Data

Andreas has spent more than eight years building data applications. At Pondhouse Data he designs and ships AI systems and integrates them into complex, real-world environments.