Instrumenting and Evaluating LLM Applications with TruLens and OpenAI

A Coding Guide to Instrumenting, Tracing, and Evaluating LLM Applications Using TruLens and OpenAI Models

TruLens enables the creation of measurable evaluation pipelines by capturing inputs and intermediate steps as structured traces. This framework allows developers to move beyond black-box testing by attaching quantitative feedback functions to every stage of an LLM application.

Why This Matters

In real-world settings, trust and explainability are as critical as raw performance, yet LLMs are often deployed without granular visibility. Instrumentation transforms every model call into an inspectable artifact, allowing engineers to address failures in retrieval or generation through versioned experimentation and systematic leaderboards.

Key Insights

• TruLens feedback functions like groundedness_measure_with_cot_reasons provide Chain-of-Thought explanations to validate model outputs against context.
• Vector stores like Chroma index text embeddings from models such as OpenAI’s text-embedding-3-small to enable semantic search.
• Instrumentation adds tracing spans to application functions to capture latency, token usage, and retrieved contexts via OpenTelemetry conventions.
• Systematic comparison of prompt styles, such as base prompts versus strict citation enforcement, is facilitated through versioned runs and leaderboards.
• The evaluation pipeline utilizes feedback providers like TruOpenAI to compute quantitative scores for answer relevance and contextual alignment.

Working Examples

Core RAG class implementation featuring TruLens instrumentation for retrieval and generation spans.

class RAG: def __init__(self, *, gen_model: str, prompt_style: str = 'base', k: int = 4): self.gen_model = gen_model; self.prompt_style = prompt_style; self.k = k; @instrument(span_type=SpanAttributes.SpanType.RETRIEVAL, attributes={SpanAttributes.RETRIEVAL.QUERY_TEXT: 'query', SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: 'return'}) def retrieve(self, query: str): res = collection.query(query_texts=[query], n_results=self.k); return res; @instrument(span_type=SpanAttributes.SpanType.GENERATION) def generate(self, query: str, hits: list): context = format_context(hits); resp = oai_client.chat.completions.create(model=self.gen_model, messages=[{'role': 'system', 'content': 'helpful assistant'}]); return resp.choices[0].message.content

Practical Applications

• RAG System Optimization: Comparing multiple prompt versions using a shared leaderboard to identify the most reliable configuration for grounding answers in context.
• Pitfall: Hardcoding sensitive credentials like OPENAI_API_KEY; developers should use secure input methods like getpass to maintain security during instrumentation.
• Pitfall: Poor document chunking; failing to split knowledge sources into overlapping segments can lead to loss of semantic continuity during retrieval.

References:

• https://www.marktechpost.com/2026/02/22/a-coding-guide-to-instrumenting-tracing-and-evaluating-llm-applications-using-trulens-and-openai-models/