RAG observability in production: how Indian and UK builders should pick between Langfuse, LangSmith and Arize Phoenix

The "why did it hallucinate" problem every RAG team hits

There is a moment every team building retrieval-augmented generation discovers, usually around the third week of production traffic. A user shows you a screenshot. The model has invented a clause. Or quoted a price from a discontinued product. Or apologised for an outage that never happened. You open the logs, and what you see is a single line — response_id=ab12cd, tokens=2,418, latency_ms=1,840. None of which tells you why.

The honest answer to the question "why did it hallucinate" almost always lives inside the trace — which chunks were retrieved, what embedding scores they had, what the rewritten query looked like, what the system prompt actually rendered as after templating, which tool returned what, and which model version answered. None of that is in your standard APM dashboard. None of it is in Sentry. And none of it is in the cloud provider's LLM gateway logs either, which usually capture only the outermost request.

That blind spot is what the agent-observability category has spent the past eighteen months filling. As of April 2026 the field has consolidated. Six platforms now do the job at production grade — and they have specialised enough that the right choice for an Indian fintech is not the right choice for a London consultancy with a Datadog estate.

What "observability" actually means for a RAG pipeline (vs an LLM call)

Tracing a single LLM call is a solved problem. You log the prompt, the response, the token counts, the latency. Done. A RAG pipeline is two or three orders of magnitude more interesting because the inference call is the end of a chain, not the whole story.

A production RAG request typically passes through query rewriting (sometimes a smaller model), embedding generation, vector search across one or more indexes, optional re-ranking, context assembly, prompt templating, the inference call itself, and frequently a guardrail or moderation pass on the way out. If any of those steps misbehaves — a stale chunk wins the cosine-similarity contest, the re-ranker swaps the order, the system prompt is templated with the wrong tenant ID — the final answer is wrong and the inference log alone tells you nothing.

Modern observability platforms therefore provide three capabilities that ordinary APM tooling does not:

• End-to-end trace visibility — a single trace that nests every span (retrieve, rerank, generate, guardrail) under one parent request, with the full payload of each step captured.
• Cost and latency analytics — token-level costs aggregated by model, by user, by tenant, by route; latency breakdowns that show whether your p95 is retrieval or generation; cost-per-conversation broken down by feature.
• Hallucination debugging via full execution trace — replay the exact retrieved chunks, the exact rendered prompt, the exact tool outputs that fed the answer. If the source-of-truth chunk was in the index but lost the vector match, you can see it.

That third capability is what changes the conversation. "Why did it hallucinate" stops being a research project and becomes a five-minute investigation.

The six-platform consolidation as of mid-2026

The category had a long tail of starter projects through 2024 and 2025 — Langtrace, OpenLLMetry, TruLens, plus a dozen smaller efforts. By April 2026 the production-grade tier has settled to six platforms with materially different sweet spots.

Platform	Self-host	OSS core	LangChain-native	Eval suite	Cost analytics	Pricing model
LangSmith	Enterprise only	No	Yes (zero config)	Strong	Strong	SaaS, per-trace
Langfuse	Yes (Docker, Helm)	Yes (MIT)	Adapter	Strong	Strong	OSS free; cloud per-event
Arize Phoenix	Yes (OSS)	Yes (Elastic 2.0)	Adapter	Best in class	Moderate	OSS free; SaaS metered
Helicone	Yes	Yes (Apache 2.0)	Adapter	Moderate	Best in class	OSS free; cloud per-request
Datadog LLM Obs	No	No	Adapter	Moderate	Strong (APM bundle)	Add-on to Datadog
Honeycomb LLM Obs	No	No	OpenTelemetry	Limited	Strong	Add-on to Honeycomb

W&B Weave sits adjacent to this list — it is best for experimentation and prompt research rather than production telemetry, so we have left it off the production six. Treat it as a complementary tool for your research notebooks.

LangSmith — when LangChain-native zero-config wins

LangSmith is built by the LangChain team and instruments every chain, agent and tool call automatically the moment you set two environment variables in your runtime. There is no SDK to wrap. No decorators to add. Spans appear in the UI within seconds of your first request.

The trade-off is the obvious one: you are paying for a managed SaaS and self-host is gated behind an enterprise tier. For teams already all-in on LangChain or LangGraph this is almost always the right starting point — see our agent frameworks consolidation piece for where LangGraph sits in the wider field. If your stack is heterogeneous (some LangChain, some raw OpenAI SDK calls, some Pydantic AI), LangSmith covers the LangChain portion beautifully and leaves the rest invisible.

# LangSmith zero-config — two environment variables and a chain
export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=ls__...

# Your existing LangChain code now emits traces automatically.
# Nothing in application code changes.
from langchain.chains import RetrievalQA
qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)
qa.invoke({"query": "Which compliance clause covers data residency?"})

The official capability page at langchain.com/langsmith/observability documents the full feature surface. Pricing is per-trace with a generous developer free tier; enterprise self-host begins to make sense once you cross several million traces per month or have a data-residency mandate.

Langfuse — when self-host and OSS matter (with 19,000+ GitHub stars)

Langfuse has become the default open-source answer. As of writing it has crossed 19,000 GitHub stars and ships as both a Docker compose stack and a Helm chart, so you can run the entire observability layer inside your own VPC with traces stored in your Postgres. For Indian fintech teams sitting inside DPDP scope, and UK teams adjacent to NHS or financial-services data, this is the deciding factor.

The platform is also more framework-agnostic than LangSmith by design. The official Langfuse review of RAG observability at langfuse.com/blog/2025-10-28-rag-observability-and-evals walks through the OpenTelemetry, decorator and explicit-SDK ingestion paths. You can wrap a raw OpenAI client, a LlamaIndex pipeline, a Pydantic AI agent or a LangGraph workflow with the same primitives.

# Langfuse Python decorator — works with any LLM SDK
from langfuse.decorators import observe, langfuse_context

@observe()
def answer_compliance_question(question: str) -> str:
    langfuse_context.update_current_observation(
        metadata={"tenant": "uk-fintech-01", "region": "eu-west-2"},
    )
    chunks = retriever.retrieve(question)
    prompt = template.render(question=question, chunks=chunks)
    return llm.complete(prompt)

Arize Phoenix — when eval rigour matters more than operations

Arize Phoenix was built around evaluation rather than retrofitting evaluation onto a tracing tool. Context relevance, groundedness, answer correctness, retrieval drift, and embedding-cluster visualisations are first-class objects. If your team has an ML lead who wants to look at the embedding manifold, see which queries cluster around which intents, and identify retrieval drift across releases, this is the tool that maps to that mental model.

The trade-off is operational — Phoenix is excellent at telling you what is wrong with your retrieval and modest at telling you which user paid for it. Cost analytics exist but are less granular than Langfuse or Helicone. A common pattern in 2026 is to run Phoenix in parallel with one of the operations-first platforms: Phoenix for the eval harness, Langfuse or Helicone for the per-tenant cost dashboard.

Phoenix integrates cleanly with our production RAG playbook recommendations — particularly the section on retrieval evaluation and the chunk-relevance scorecards.

Helicone, Datadog, Honeycomb — the operations-team alternatives

These three platforms cluster around a different question — not "is my retrieval quality good" but "what is this costing me, who is paying for it, and is anything anomalous in production".

Helicone is the lightest-weight option of the six. You change one line — replace your OpenAI base URL with the Helicone proxy URL — and you get cost analytics, per-user tracking, prompt caching and rate-limit visibility. The eval suite is modest, but the time-to-first-dashboard is measured in minutes, and the OSS self-host story is mature. Strong choice for solo founders and small teams who want cost visibility above all else.

Datadog LLM Observability is the right answer for any team already running Datadog APM. Spans appear in the same trace tree as your service traces, so a slow RAG request shows up next to the slow database query that delayed it. Prompt diffing, token cost analytics, model performance and drift all live in the same workspace your SREs already use. There is no self-host story and no OSS tier, but for enterprise customers the bundling argument usually wins.

Honeycomb LLM Observability takes the OpenTelemetry-first route. If your platform team has standardised on OTel and you are willing to instrument calls yourself, Honeycomb gives you the BubbleUp anomaly detection and high-cardinality query model that already serves your microservices, applied to LLM spans. Less prescriptive than the others; rewards teams who like to compose their own dashboards.

A picking framework: framework + team-shape + APM stack

The honest decision tree is three dimensions, not one. What framework do you build on, what shape is your team, and what APM stack do you already pay for.

Team shape	Primary choice	Why
LangChain / LangGraph shop, <20 engineers	LangSmith	Zero-config integration; pricing free tier covers most teams below 100k traces/day
Multi-framework (LlamaIndex + raw SDK + Pydantic AI)	Langfuse	Framework-agnostic SDK; decorators wrap anything
Regulated workload (DPDP / NHS / banking)	Langfuse self-hosted	Data residency; full OSS source; runs inside VPC
OSS-first, no vendor lock-in	Langfuse or Arize Phoenix	Both have permissive licences and active OSS releases
Enterprise on Datadog APM	Datadog LLM Obs	Spans co-located with service traces; finance bundle wins
Research-heavy retrieval team	Arize Phoenix	Eval-first; embedding cluster visualisations; drift detection

The dual-market pattern we see most often: Indian start-ups default to self-hosted Langfuse for cost and data-residency reasons; UK teams inside larger firms default to LangSmith or Datadog depending on what their employer already pays for. Neither pattern is wrong; both reflect rational responses to local cost structure.

Cost of getting it wrong — three real failure modes

The cost of skipping observability is rarely a single dramatic incident. It is the slow accumulation of three failure modes that compound.

1. The silent regression. A model provider quietly changes the underlying weights of a non-pinned model. Your retrieval scorecards do not move, but answer quality drifts by 4–6 percentage points over a fortnight. Without an eval harness running against a golden set on every release, you do not notice until a customer complains. Cost: roughly one churned enterprise customer worth more than the entire observability bill.
2. The cost runaway. A new feature ships that loops the agent unnecessarily — three tool calls where one would do. Per-request cost triples. Aggregate cost climbs steadily over a month before finance flags it. Cost analytics broken down by feature route catch this in days, not weeks. We have seen genuinely large IN and UK teams burn six-figure inference bills against a single mis-implemented retry loop.
3. The compliance gap. A regulator asks for the exact prompt, retrieved chunks and model response for a flagged customer interaction from three months ago. Without trace persistence and replay, the answer is "we cannot reproduce it". For DPDP-scope workloads and UK FCA-supervised use cases, that answer is not acceptable. For teams shipping human-in-the-loop agents with checkpointed state, trace persistence is doubly important — you need both the checkpoint and the observability record.

Two strong public comparisons in mid-2026 corroborate the platform picture above — the Maxim review at getmaxim.ai/articles/top-5-rag-observability-platforms-in-2026 and the framework-centric breakdown at myengineeringpath.dev/tools/langsmith-vs-langfuse. A useful operations-team perspective lives at digitalapplied.com/blog/agent-observability-platforms-langsmith-langfuse-arize-2026.

So which one should you start with

If you are reading this on the morning of your first RAG deployment and need a single recommendation: install self-hosted Langfuse on a small VM today, point your application at it, and re-evaluate in six weeks once you have actual traffic patterns. The total cost of that decision is one afternoon of platform-team time and the cost of a t3.medium. The upside is full trace visibility before your first user-visible hallucination.

If you are already deep into LangChain and your team is small, set the two LangSmith environment variables this afternoon and skip the self-host conversation. The time-to-first-trace is shorter than reading this article was.

If you are a regulated workload with a real eval-rigour requirement, pair Arize Phoenix with whichever operations layer your team already runs. Neither category fully replaces the other; production teams that mature past the first year almost always end up with one tool per axis.