Langfuse Hands-On Workshop: From AI Agent Observability to Evaluation & Continuous Improvement

Date: Wednesday, November 25, 2026
Time: 12:00pm GST
Duration: 4 hours
Location: MediaOne hotel, Dubai
                  Media 1 Tower - Al Falak St - Exit 32
                  Dubai Media City - Dubai
Cost: FREE
Format: Instructor-led training with hands-on labs
Audience: AI engineers, developers, data scientists, ML engineers, technical architects

What You’ll Walk Away With

• A working agent fully instrumented with Langfuse
• Visibility into LLM calls, tool execution, latency, and errors
• Version-controlled prompts linked to traces
• A reusable evaluation dataset and experiment results
• A repeatable, evidence-based workflow for improving agent quality

Agenda:
Start time: 12.00 pm
Lunch and networking: 12.15-1.00pm
Workshop: 1.00pm-4.00pm

• The AI Engineering Loop — Why reliable agents are hard, and where observability, prompt management, and evaluation fit in the lifecycle.
• Meet the Agent — Set up the sample IT Support Agent and explore its architecture, tool-calling flow, and baseline behaviour.
• Tracing & Observability — Instrument the agent with Langfuse. Capture generations, tool calls, latency, and errors; debug from real traces.
• Prompt Management — Move prompts into Langfuse, version them, and link changes to agent behaviour.
  
• Break — Networking and Q&A.
• Monitoring in Production — Use traces to spot user dissatisfaction, unexpected responses, and quality issues in the wild.
• Build an Evaluation Dataset — Turn representative questions and expected behaviours into a reusable test set.
• Run Experiments — Evaluate the agent against the dataset, score and compare runs, and prove whether a change improves quality.
• The Improvement Loop — Connect tracing, prompts, monitoring, datasets, and experiments into one repeatable workflow.
• Wrap-Up — Key takeaways, questions, and applying this to your own applications.

Before You Arrive:

• Bring a laptop with Node.js/npm and Git installed, a Langfuse Cloud account, and your LLM API credentials.
• Complete the environment setup in advance if possible.

Materials: https://langfuse.com/workshop