Skip to content

bug: OpenTelemetry Spans Not Displaying in Self-Hosted Instance Timeline #2821

Open
#3686
deepshekhardas/trigger.dev
#16
Open
bug: OpenTelemetry Spans Not Displaying in Self-Hosted Instance Timeline#2821
#3686
deepshekhardas/trigger.dev
#16

Description

@Aquesta
Aquesta
opened on Dec 29, 2025

Provide environment information

Self-hosted Instance: │ │

  • Trigger.dev Version: main (Docker image from 2024-12-26) │ │
  • Deployment: Docker Compose (official hosting/docker/webapp setup)
  • Instance URL: trigger.devneurox.tech │ │
  • Docker Images: │ │
  • Webapp: ghcr.io/triggerdotdev/trigger.dev:main (c3af7f05c1da) │ │
  • Supervisor: ghcr.io/triggerdotdev/supervisor:main (d3d1d21ad176)

Describe the bug

Issue: OpenTelemetry Spans Not Displaying in Self-Hosted Instance Timeline

Environment:

  • Trigger.dev SDK: v4.3.1
  • Self-hosted instance: trigger.devneurox.tech
  • Cloud instance: cloud.trigger.dev

Problem Description:

OpenTelemetry spans created with logger.trace() are not visible in the timeline UI on our self-hosted Trigger.dev instance, but they work correctly on the official cloud instance (cloud.trigger.dev).

Configuration:

trigger.config.ts:

import { defineConfig } from "@trigger.dev/sdk";
import { HttpInstrumentation } from "@opentelemetry/instrumentation-http";
import { FetchInstrumentation } from "@opentelemetry/instrumentation-fetch";

export default defineConfig({
  project: "proj_iahascmunqsxfplmrepp",
  runtime: "node",
  logLevel: "log",
  telemetry: {
    instrumentations: [
      new HttpInstrumentation(),
      new FetchInstrumentation(),
    ],
  },
  // ...
});

Code Example:

const marketData = await logger.trace("bybit-api-calls", async (span) => {
  span.setAttribute("symbol", input.symbol);
  span.setAttribute("endpoints", "getTicker, getOpenInterest, getLongShortRatio, getFundingRate");
  return await bybitAPI.getMarketData(input.symbol);
});

Expected Behavior:

Timeline should show individual spans with hierarchy and timing.

Actual Behavior:

Self-Hosted: Timeline shows only one solid green bar - no spans visible
Cloud (cloud.trigger.dev): Timeline correctly shows all spans ✅
Steps to Reproduce:

Deploy same code with [logger.trace()](vscode-file://vscode-app/Applications/Visual%20Studio%20Code.app/Contents/Resources/app/out/vs/code/electron-browser/workbench/workbench.html) spans to both self-hosted and cloud
Trigger task execution
Compare timeline view in dashboard
Additional Info:

Tasks execute successfully on both instances. Console logs show correct timing. Issue appears UI-specific to self-hosted.

Question: What self-hosted dashboard version is required for span visualization?

### Reproduction repo

N/A - Private repository. Can provide minimal reproduction example if needed.

### To reproduce

**Steps to reproduce:**

1. Configure Trigger.dev with OpenTelemetry instrumentations:
```typescript
telemetry: {
  instrumentations: [
    new HttpInstrumentation(),
    new FetchInstrumentation(),
  ],
}
2. Add [logger.trace() spans in your task code:
const result = await logger.trace("operation-name", async (span) => {
  span.setAttribute("key", "value");
  return await someOperation();
});

3 Deploy to self-hosted instance (trigger.devneurox.tech)
4 Deploy same code to cloud instance (cloud.trigger.dev)
5 Execute task on both instances
6 Compare timeline view in dashboard
Expected: Timeline shows individual spans with hierarchy and timing

Actual on self-hosted: Timeline shows only one solid green bar - no span breakdown visible

Actual on cloud: Timeline correctly displays all spans with proper hierarchy ✅

Note: Tasks execute successfully on both - issue is UI-only on self-hosted.

### Additional information

<img width="751" height="345" alt="Image" src="https://github.com/user-attachments/assets/6eb7b2a4-6265-4c92-a2bd-d6d72e3eaa02" />

<img width="647" height="418" alt="Image" src="https://github.com/user-attachments/assets/d506c4ab-7812-4d50-ba20-73e56df4a86c" />

Activity

  1. coderabbitai commented on Dec 29, 2025

    @coderabbitai
    coderabbitaibot
    on Dec 29, 2025 – with coderabbitai
    Contributor

    📝 CodeRabbit Plan Mode

    Generate an implementation plan and prompts that you can use with your favorite coding agent.

    • Create Plan
    Examples
    🔗 Similar Issues

    Possible Duplicates

    • 2521

    Related Issues

    👤 Suggested Assignees
    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
  auto_enrich:
    enabled: false

    💬 Have feedback or questions? Drop into our discord!

  2. caiofeuser commented on Jan 13, 2026

    @caiofeuser
    caiofeuser
    on Jan 13, 2026

    Same issue here

  3. lecondor-dev commented on Feb 2, 2026

    @lecondor-dev
    lecondor-dev
    on Feb 2, 2026

    +1

  4. ThullyoCunha
    added 2 commits that reference this issue on May 21, 2026

    feat(webapp): add RUNTIME_API_ORIGIN to decouple runner traffic from …

    eb32a7f

    feat(webapp): add RUNTIME_API_ORIGIN to decouple runner traffic from …

    da151f6
  5. deepshekhardas
    added a commit that references this issue on Jun 1, 2026

    fix(webapp): add overridesBySpanId and environmentId filter for self-…

    b0a7682

  6. matt-aitken commented on Aug 29, 2026

    @matt-aitken
    matt-aitken
    on Aug 29, 2026
    Member

    Thank you for the report. We investigated this.

    The single green bar is the root span of the run. The webapp writes this span when the run is triggered. All child spans (logger.trace, HTTP instrumentation) and all logger.* output are sent from the run container to the webapp's /otel endpoint. The run container gets this address from the OTEL_EXPORTER_OTLP_ENDPOINT variable on the supervisor. The Docker Compose default is http://webapp:3000/otel. If the run container cannot reach this URL, the run still completes, because the run lifecycle traffic goes through the supervisor, but no spans or logs arrive. The export failure is not logged by default.

    Please check the following:

    1. The supervisor's OTEL_EXPORTER_OTLP_ENDPOINT points to a URL that resolves from inside a run container. In a split webapp/worker setup this must be the public webapp URL followed by /otel, not the internal webapp hostname.
    2. Any reverse proxy in front of the webapp allows POST requests with application/x-protobuf bodies and does not set a small body-size limit.
    3. The webapp logs show POST /otel/v1/traces requests when a run executes.
    4. Are logger.log and logger.info lines also missing from the dashboard? They use the same exporter. If they are missing too, the export path is the cause.

    To see exporter errors, set TRIGGER_OTEL_LOG_LEVEL=error as an environment variable on your project. The errors then appear in the run container output.

    If you store task events in ClickHouse (EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2), also confirm that the clocks on the worker host and the webapp host are synchronised to within one minute. The trace query filters spans by start time relative to the run creation time.

    If none of this applies, please share your supervisor environment (with secrets removed) and the webapp log lines from one run, and we will look further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions