OpenTelemetry (OTEL)

OpenTelemetry (OTEL) is a CNCF project providing specifications, APIs, and libraries that establish standardized methods for collecting distributed traces and metrics from applications.

Muster operates as an OpenTelemetry backend, accepting traces at the /api/public/otel (OTLP) endpoint. This integration extends compatibility beyond the native SDKs and integrations to frameworks, libraries, and languages including Java and Go through tools like OpenLLMetry and OpenLIT.

Muster maps the received OTel traces to its data model and supports additional attributes popular in the OTel GenAI ecosystem.

When to Use the OpenTelemetry Integration

• Prefer the native SDKs if: you are using Python or JavaScript/TypeScript.
• Use OTEL if: you have an existing OTEL setup, collector-based ingestion, or you are working in a language without a first-party SDK.

flowchart LR
  subgraph App["Application"]
    Logic["Operations, LLM calls, events"]
    OTel["OpenTelemetry instrumentation (SDKs, libraries)"]
    Logic --> OTel
  end

  OTel -->|"OTLP traces"| Endpoint["Muster OTLP endpoint /api/public/otel"]

  subgraph Muster["Muster"]
    Endpoint --> Map["OTel to data-model attribute mapping"]
    Map --> UI["Muster traces"]
  end

Important: Propagating Trace Attributes to All Spans

Certain trace-level attributes must propagate to all spans within a trace for proper aggregation and filtering:

• userId (via langfuse.user.id or user.id)
• sessionId (via langfuse.session.id or session.id)
• metadata (via langfuse.trace.metadata.*)
• version (via langfuse.version)
• release (via langfuse.release)
• tags (via langfuse.trace.tags)
• trace_name (via langfuse.trace.name)

Recommended: OpenTelemetry Baggage

Use OpenTelemetry Baggage with a BaggageSpanProcessor:

1. Set the desired attributes as baggage entries at trace start.
2. Set the attributes on the currently active span.
3. Configure a BaggageSpanProcessor to copy baggage entries to span attributes.
4. All downstream spans receive these attributes automatically.

Implementation guides are available for Python and JavaScript.

Security note: OpenTelemetry baggage is propagated across service boundaries and to third-party APIs. Do not include sensitive information in baggage when using this approach.

Alternative: SDK Helpers

If you are using the Muster (langfuse) SDKs, the convenience methods propagate_attributes() (Python) or propagateAttributes() (TypeScript) provide automatic attribute propagation.

Ingestion Options

OpenTelemetry-Native SDK v4

The quickest path is the OTEL-native langfuse SDK v4, a thin layer on the official OpenTelemetry client. It automatically converts spans into rich Muster observations and adds LLM-specific helpers for token usage, cost tracking, prompt linking, and scoring.

The SDK exports spans from other OTEL-instrumented libraries by default, focusing on LLM-relevant spans. Use custom filters in the advanced SDK configuration to export everything.

OpenTelemetry Endpoint

Muster receives traces at the /api/public/otel (OTLP) endpoint.

OTEL_EXPORTER_OTLP_ENDPOINT="https://app.getmuster.io/api/public/otel"
# Self-hosted: http(s)://<your-muster-host>/api/public/otel

OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic ${AUTH_STRING},x-langfuse-ingestion-version=4"

Authentication uses Basic Auth. Generate AUTH_STRING:

echo -n "pk-lf-1234567890:sk-lf-1234567890" | base64
# For GNU base64: base64 -w 0

For real-time Fast Preview, include the x-langfuse-ingestion-version: 4 header. If you use signal-specific headers, add the same value to OTEL_EXPORTER_OTLP_TRACES_HEADERS.

Signal-specific endpoint:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://app.getmuster.io/api/public/otel/v1/traces"

Muster supports OTLP over HTTP (JSON and protobuf). gRPC is not yet supported.

Custom OpenTelemetry SDKs

Use OpenTelemetry SDKs directly with the configuration above to export traces, extending language support beyond Python and JS/TS.

OpenTelemetry GenAI Instrumentation Libraries

Recommended libraries:

• OpenLIT
• OpenLLMetry
• Arize
• MLflow

Export From an OpenTelemetry Collector

graph LR
    A[Application with OTel Instrumentation] --> B[OpenTelemetry Collector]
    B --> C[Muster]

Use this configuration with the OpenTelemetry Collector:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
  memory_limiter:
    limit_mib: 1500
    spike_limit_mib: 512
    check_interval: 5s

exporters:
  otlphttp/muster:
    endpoint: "https://app.getmuster.io/api/public/otel"
    headers:
      Authorization: "Basic ${AUTH_STRING}"
      x-langfuse-ingestion-version: "4"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/muster]

Filtering Spans Sent to Muster

Use the OTel Collector filterprocessor to selectively forward spans. Note that filtering at the span level risks creating incomplete traces — root spans must be sent to ensure correct trace creation.

Example filtering by gen_ai.system:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  filter/openaisystem:
    error_mode: ignore
    traces:
      span:
        - 'attributes["gen_ai.system"] != "openai"'

exporters:
  otlphttp/muster:
    endpoint: "https://app.getmuster.io/api/public/otel"
    headers:
      Authorization: "Basic ${AUTH_STRING}"
      x-langfuse-ingestion-version: "4"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/openaisystem]
      exporters: [otlphttp/muster]

Attribute Mapping

Muster follows the OpenTelemetry GenAI semantic conventions and uses attributes within the langfuse.* namespace to map OTel span attributes directly to its data model. These attributes take precedence over generic conventions and are recommended for manual instrumentation.

Attribute keys containing __proto__, constructor, or prototype as a path segment are silently dropped during ingestion as a security measure.

How Metadata Mapping Works

SDK-emitted vs. standard OTel attributes:

• Muster (langfuse) SDKs: provide utility functions that automatically set langfuse.*.metadata.* prefixed attributes. Custom metadata appears at the first level and is filterable.
• Standard OTel SDKs: set attributes directly on spans. Without an explicit langfuse.trace.metadata.* or langfuse.observation.metadata.* prefix, attributes end up in the metadata.attributes catch-all and are not directly filterable.

Trace-Level Attributes

Applied to the trace record. May be set on any span in the trace.

Filtering by metadata key:

Muster supports filtering only on top-level metadata keys. Use the langfuse.trace.metadata prefix to map attributes to the top-level metadata object:

with tracer.start_as_current_span("Example") as span:
    span.set_attribute("langfuse.trace.metadata.user_name", "user-123")

Observation-Level Attributes

Applied to individual observations (spans) within a trace.

Filtering by metadata key:

with tracer.start_as_current_span("Example") as span:
    span.set_attribute("langfuse.observation.metadata.user_name", "user-123")

Troubleshooting

• For 4xx errors on a self-hosted Muster, upgrade to the latest version. The OpenTelemetry endpoint was introduced in upstream Langfuse v3.22.0 with significant improvements since.
• Muster supports OTLP over HTTP (JSON and protobuf). gRPC is not yet supported.

See also

• Upstream Langfuse OpenTelemetry docs