[OTEL] Add OpenTelemetry observability support

.github/workflows/release.yml

@@ -46,6 +46,7 @@ jobs:
           helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
           helm repo add strimzi https://strimzi.io/charts/
           helm repo add seaweedfs https://seaweedfs.github.io/seaweedfs/helm
+          helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
 
       - name: Run chart-releaser
         uses: helm/chart-releaser-action@cae68fefc6b5f367a0275617c9f83181ba54714f

.gitignore

@@ -4,3 +4,7 @@ charts/mlrun-ce/charts/*
 **/.DS_Store
 *.DS_Store
 **/__pycache__
+# Packaged chart tarballs (generated by make package)
+charts/mlrun-ce/mlrun-ce-*.tgz
+# MLRun project directories created by test scripts
+otlp-pro/

charts/mlrun-ce/Chart.yaml

@@ -1,6 +1,6 @@
 apiVersion: v1
 name: mlrun-ce
-version: 0.11.0-rc.32
+version: 0.11.0-rc.33
 description: MLRun Open Source Stack
 home: https://iguazio.com
 icon: https://www.iguazio.com/wp-content/uploads/2019/10/Iguazio-Logo.png

charts/mlrun-ce/README.md

@@ -14,6 +14,7 @@ The Open source MLRun ce chart includes the following stack:
 * Spark Operator - https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
 * Pipelines - https://github.com/kubeflow/pipelines
 * Prometheus stack - https://github.com/prometheus-community/helm-charts
+* OpenTelemetry Operator - https://github.com/open-telemetry/opentelemetry-operator (observability)
 
 ## Prerequisites
 
@@ -64,6 +65,33 @@ helm --namespace mlrun \
     mlrun/mlrun-ce
 ```
 
+### Installing with OpenTelemetry Enabled
+
+> **Note:** OpenTelemetry is **disabled by default**. Follow the standard [Installing the Chart](#installing-the-chart) steps, adding the OTel flags below.
+
+To install with OpenTelemetry enabled, append the following flags to the helm install command:
+
+```bash
+helm --namespace mlrun \
+    install my-mlrun \
+    --wait \
+    --set global.registry.url=<registry URL e.g. index.docker.io/iguazio> \
+    --set global.registry.secretName=registry-credentials \
+    --set opentelemetry-operator.enabled=true \
+    --set opentelemetry.namespaceLabel.enabled=true \
+    --set opentelemetry.collector.enabled=true \
+    --set opentelemetry.instrumentation.enabled=true \
+    mlrun/mlrun-ce
+```
+
+To verify the OpenTelemetry resources were created:
+
+```bash
+kubectl -n mlrun get opentelemetrycollectors
+kubectl -n mlrun get instrumentations
+kubectl -n mlrun get pods | grep opentelemetry
+```
+
 ### Installing MLRun-ce on minikube
 
 The Open source MLRun ce uses node ports for simplicity. If your kubernetes cluster is running inside a VM, 
@@ -89,6 +117,25 @@ following values:
 Additional configurable values are documented in the `values.yaml`, and the `values.yaml` of all sub charts. 
 Override those [in the normal methods](https://helm.sh/docs/chart_template_guide/values_files/).
 
+### Configuring OpenTelemetry (Observability)
+
+MLRun CE includes the OpenTelemetry Operator for collecting metrics and traces. When enabled, it deploys a single collector per namespace (deployment mode) — instrumented pods push OTLP data to the collector, which forwards metrics to Prometheus via the OTLP endpoint. Python auto-instrumentation is applied namespace-wide via a webhook, and the `mlrun.io/otel: "true"` label is applied to Jupyter and Nuclio function pods to mark them for metric enrichment and trigger OTel injection on restart.
+
+For a fresh install with OTel, see [Installing with OpenTelemetry Enabled](#installing-with-opentelemetry-enabled).
+
+To enable OTel on an existing installation:
+
+```bash
+helm --namespace mlrun upgrade my-mlrun \
+    --set opentelemetry-operator.enabled=true \
+    --set opentelemetry.namespaceLabel.enabled=true \
+    --set opentelemetry.collector.enabled=true \
+    --set opentelemetry.instrumentation.enabled=true \
+    mlrun/mlrun-ce
+```
+
+> **Note:** The above assumes a single-namespace installation. For multi-namespace (admin/non-admin) deployments, refer to the MLRun documentation.
+
 ### Working with ECR
 
 To work with ECR, you must create a secret with your AWS credentials and a secret with ECR Token while providing both secret names to the helm install command.
@@ -282,6 +329,6 @@ Refer to the [**Kubeflow documentation**](https://www.kubeflow.org/docs/started/
 
 This table shows the versions of the main components in the MLRun CE chart:
 
-| MLRun CE   | MLRun  | Nuclio | Jupyter | MPI Operator | SeaweedFS | Spark Operator | Pipelines | Kube-Prometheus-Stack |
-|------------|--------|--------|---------|--------------|-----------|----------------|-----------|-----------------------|
-| **0.11.0** | 1.11.0 | 1.15.9 | 4.5.0   | 0.2.3        | 4.17.0    | 2.1.0          | 2.15.0    | 72.1.1                |
+| MLRun CE   | MLRun  | Nuclio | Jupyter | MPI Operator | SeaweedFS | Spark Operator | Pipelines | Kube-Prometheus-Stack | OpenTelemetry Operator |
+|------------|--------|--------|---------|--------------|-----------|----------------|-----------|-----------------------|------------------------|
+| **0.11.0** | 1.11.0 | 1.15.9 | 4.5.0   | 0.2.3        | 4.17.0    | 2.1.0          | 2.15.0    | 72.1.1                | 0.78.1                 |

charts/mlrun-ce/admin_installation_values.yaml

@@ -77,3 +77,28 @@ strimzi-kafka-operator:
 
 kafka:
   enabled: false
+
+# OpenTelemetry Operator - enabled for CRD installation at cluster level
+opentelemetry-operator:
+  enabled: true
+  admissionWebhooks:
+    certManager:
+      enabled: false
+    autoGenerateCert:
+      enabled: true
+    # Only apply webhooks to namespaces with the opentelemetry label
+    namespaceSelector:
+      matchLabels:
+        opentelemetry.io/inject: "enabled"
+
+# OpenTelemetry CRs - disabled at admin level, enabled in user namespaces
+# Note: Controller namespace does NOT need the opentelemetry label since
+# no workloads are instrumented here - only the operator runs here
+opentelemetry:
+  namespaceLabel:
+    enabled: false
+  collector:
+    enabled: false
+  instrumentation:
+    enabled: false
+

charts/mlrun-ce/non_admin_cluster_ip_installation_values.yaml

@@ -96,3 +96,19 @@ kafka:
 
 kube-prometheus-stack:
   enabled: false
+
+# OpenTelemetry Operator - disabled, CRDs installed at controller level
+opentelemetry-operator:
+  enabled: false
+
+# OpenTelemetry CRs - enabled for user namespace
+# The namespace will be labeled with opentelemetry.io/inject=enabled
+# so the operator can inject sidecars into pods
+opentelemetry:
+  namespaceLabel:
+    enabled: true
+  collector:
+    enabled: true
+  instrumentation:
+    enabled: true
+

charts/mlrun-ce/non_admin_installation_values.yaml

@@ -82,3 +82,19 @@ kafka:
 
 kube-prometheus-stack:
   enabled: false
+
+# OpenTelemetry Operator - disabled, CRDs installed at controller level
+opentelemetry-operator:
+  enabled: false
+
+# OpenTelemetry CRs - enabled for user namespace
+# The namespace will be labeled and annotated for OTel deployment-mode collection
+# and namespace-wide Python auto-instrumentation.
+opentelemetry:
+  namespaceLabel:
+    enabled: true
+  collector:
+    enabled: true
+  instrumentation:
+    enabled: true
+

charts/mlrun-ce/requirements.lock

@@ -20,5 +20,8 @@ dependencies:
 - name: strimzi-kafka-operator
   repository: https://strimzi.io/charts/
   version: 0.48.0
-digest: sha256:e2b2d1b7531c4829aa25c8ce8d95506642ab59d0cb692a343d2e508a71525374
-generated: "2026-03-31T17:13:31.403112322Z"
+- name: opentelemetry-operator
+  repository: https://open-telemetry.github.io/opentelemetry-helm-charts
+  version: 0.78.1
+digest: sha256:50ed77fd11e450e243c05eadac99857b4b0aae92ae73ca9a6c00fc1cdc726f70
+generated: "2026-04-15T11:23:19.249332+03:00"

charts/mlrun-ce/requirements.yaml

@@ -25,3 +25,7 @@ dependencies:
     repository: "https://strimzi.io/charts/"
     version: "0.48.0"
     condition: strimzi-kafka-operator.enabled
+  - name: opentelemetry-operator
+    repository: "https://open-telemetry.github.io/opentelemetry-helm-charts"
+    version: "0.78.1"
+    condition: opentelemetry-operator.enabled

charts/mlrun-ce/templates/NOTES.txt

@@ -127,5 +127,44 @@ TimescaleDB is available at:
 {{- end }}
 {{- end }}
 
+{{- if index .Values "opentelemetry-operator" "enabled" }}
+{{- "\n" }}
+OpenTelemetry Operator is enabled!
+-  Operator manages OpenTelemetryCollector and Instrumentation CRs
+-  Namespace selector: opentelemetry.io/inject=enabled
+{{- if .Values.opentelemetry.collector.enabled }}
+{{- "\n" }}
+OpenTelemetry Collector (deployment mode):
+-  Collector CR: {{ include "mlrun-ce.otel.collector.fullname" . }}
+-  OTLP gRPC endpoint: {{ include "mlrun-ce.otel.collector.fullname" . }}-collector:{{ .Values.opentelemetry.collector.otlp.grpcPort }}
+-  OTLP HTTP endpoint: {{ include "mlrun-ce.otel.collector.fullname" . }}-collector:{{ .Values.opentelemetry.collector.otlp.httpPort }}
+-  Metrics export: collector pushes via OTLP to Prometheus at /api/v1/otlp/v1/metrics
+{{- end }}
+{{- if .Values.opentelemetry.instrumentation.enabled }}
+{{- "\n" }}
+OpenTelemetry Auto-Instrumentation:
+-  Instrumentation CR: {{ include "mlrun-ce.otel.instrumentation.fullname" . }}
+{{- if .Values.opentelemetry.instrumentation.python.enabled }}
+-  Python auto-instrumentation: enabled (namespace-wide via namespace annotation)
+{{- end }}
+{{- if .Values.opentelemetry.instrumentation.java.enabled }}
+-  Java auto-instrumentation: enabled
+{{- end }}
+{{- end }}
+{{- if .Values.opentelemetry.namespaceLabel.enabled }}
+{{- "\n" }}
+Namespace OTel configuration:
+-  Label: {{ .Values.opentelemetry.namespaceLabel.key }}={{ .Values.opentelemetry.namespaceLabel.value }}
+{{- if .Values.opentelemetry.instrumentation.enabled }}
+-  Python instrumentation annotation applied to all pods in namespace {{ .Release.Namespace }}
+{{- end }}
+{{- end }}
+{{- if or .Values.opentelemetry.collector.enabled .Values.opentelemetry.instrumentation.enabled }}
+{{- "\n" }}
+Pods labeled with mlrun.io/otel=true: Jupyter and Nuclio function pods (via functionDefaults).
+These Python-based pods receive OTel auto-instrumentation (runtime metrics, traces, HTTP metrics for Nuclio functions).
+{{- end }}
+{{- end }}
+
 Happy MLOPSing!!! :]
 {{- end }}

charts/mlrun-ce/templates/_helpers.tpl

@@ -413,3 +413,154 @@ TimescaleDB connection string for MLRun model monitoring
 postgresql://{{ .Values.timescaledb.auth.username | urlquery }}:{{ .Values.timescaledb.auth.password | urlquery }}@{{ include "mlrun-ce.timescaledb.fullname" . }}:{{ .Values.timescaledb.service.port }}/{{ .Values.timescaledb.auth.database }}
 {{- end }}
 
+{{/*
+=============================================================================
+OpenTelemetry helpers
+=============================================================================
+*/}}
+
+{{/*
+OpenTelemetry Collector name
+*/}}
+{{- define "mlrun-ce.otel.collector.name" -}}
+{{- default "otel" .Values.opentelemetry.collector.nameOverride | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+OpenTelemetry Collector fullname
+*/}}
+{{- define "mlrun-ce.otel.collector.fullname" -}}
+{{- if .Values.opentelemetry.collector.fullnameOverride }}
+{{- .Values.opentelemetry.collector.fullnameOverride | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- $name := default "otel" .Values.opentelemetry.collector.nameOverride }}
+{{- if contains $name .Release.Name }}
+{{- .Release.Name | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
+{{- end }}
+{{- end }}
+{{- end }}
+
+{{/*
+OpenTelemetry Instrumentation name
+*/}}
+{{- define "mlrun-ce.otel.instrumentation.name" -}}
+{{- default "otel-instrumentation" .Values.opentelemetry.instrumentation.nameOverride | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+OpenTelemetry Instrumentation fullname
+*/}}
+{{- define "mlrun-ce.otel.instrumentation.fullname" -}}
+{{- if .Values.opentelemetry.instrumentation.fullnameOverride }}
+{{- .Values.opentelemetry.instrumentation.fullnameOverride | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- $name := default "otel-instrumentation" .Values.opentelemetry.instrumentation.nameOverride }}
+{{- if contains $name .Release.Name }}
+{{- .Release.Name | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
+{{- end }}
+{{- end }}
+{{- end }}
+
+{{/*
+OpenTelemetry common labels
+*/}}
+{{- define "mlrun-ce.otel.labels" -}}
+{{ include "mlrun-ce.common.labels" . }}
+{{ include "mlrun-ce.otel.selectorLabels" . }}
+{{- end }}
+
+{{/*
+OpenTelemetry selector labels
+*/}}
+{{- define "mlrun-ce.otel.selectorLabels" -}}
+{{ include "mlrun-ce.common.selectorLabels" . }}
+app.kubernetes.io/component: opentelemetry
+{{- end }}
+
+{{/*
+OpenTelemetryCollector CR manifest for use in the CRD readiness job
+*/}}
+{{- define "mlrun-ce.otel.collector.manifest" -}}
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  name: {{ include "mlrun-ce.otel.collector.fullname" . }}
+  namespace: {{ .Release.Namespace }}
+  labels:
+    {{- include "mlrun-ce.otel.labels" . | nindent 4 }}
+spec:
+  mode: {{ .Values.opentelemetry.collector.mode }}
+  upgradeStrategy: {{ .Values.opentelemetry.collector.upgradeStrategy }}
+  managementState: managed
+  image: {{ (index .Values "opentelemetry-operator").manager.collectorImage.repository }}:{{ (index .Values "opentelemetry-operator").manager.collectorImage.tag }}
+  resources:
+    {{- toYaml .Values.opentelemetry.collector.resources | nindent 4 }}
+  config:
+    {{- toYaml .Values.opentelemetry.collector.config | nindent 4 }}
+{{- end }}
+
+{{/*
+Instrumentation CR manifest for use in the CRD readiness job
+*/}}
+{{- define "mlrun-ce.otel.instrumentation.manifest" -}}
+apiVersion: opentelemetry.io/v1alpha1
+kind: Instrumentation
+metadata:
+  name: {{ include "mlrun-ce.otel.instrumentation.fullname" . }}
+  namespace: {{ .Release.Namespace }}
+  labels:
+    {{- include "mlrun-ce.otel.labels" . | nindent 4 }}
+spec:
+  exporter:
+    endpoint: http://{{ include "mlrun-ce.otel.collector.fullname" . }}-collector:{{ .Values.opentelemetry.collector.otlp.httpPort }}
+  propagators:
+    {{- toYaml .Values.opentelemetry.instrumentation.propagators | nindent 4 }}
+  sampler:
+    type: {{ .Values.opentelemetry.instrumentation.sampler.type }}
+    argument: {{ .Values.opentelemetry.instrumentation.sampler.argument | quote }}
+  env:
+    - name: OTEL_SERVICE_NAME
+      valueFrom:
+        fieldRef:
+          fieldPath: metadata.name
+    - name: OTEL_METRICS_EXPORTER
+      value: otlp
+    - name: OTEL_TRACES_EXPORTER
+      value: otlp
+    - name: OTEL_LOGS_EXPORTER
+      value: none
+  {{- if .Values.opentelemetry.instrumentation.python.enabled }}
+  python:
+    image: {{ .Values.opentelemetry.instrumentation.python.image.repository }}:{{ .Values.opentelemetry.instrumentation.python.image.tag }}
+    resourceRequirements:
+      {{- toYaml .Values.opentelemetry.instrumentation.python.resources | nindent 6 }}
+    env:
+      - name: OTEL_PYTHON_LOG_CORRELATION
+        value: "true"
+      - name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
+        value: "false"
+      - name: OTEL_PYTHON_DISABLED_INSTRUMENTATIONS
+        value: "aws_lambda"
+  {{- end }}
+  {{- if .Values.opentelemetry.instrumentation.java.enabled }}
+  java:
+    image: {{ .Values.opentelemetry.instrumentation.java.image.repository }}:{{ .Values.opentelemetry.instrumentation.java.image.tag }}
+    resourceRequirements:
+      {{- toYaml .Values.opentelemetry.instrumentation.java.resources | nindent 6 }}
+    env:
+      - name: OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED
+        value: "true"
+  {{- end }}
+{{- end }}
+{{/*
+OTel pod label — marks a pod as OTel-monitored for metric enrichment and discovery.
+Namespace-level instrumentation annotation (set by namespace-label job) handles Python auto-instrumentation.
+Wrap usage with: {{- if and .Values.opentelemetry.collector.enabled .Values.opentelemetry.instrumentation.enabled }}
+*/}}
+{{- define "mlrun-ce.otel.podLabels" -}}
+mlrun.io/otel: "true"
+{{- end }}