Skip to content

[TEST] Make OTLP gRPC functional teardown deterministic - #4541

Draft
yigitcan-ozturk wants to merge 1 commit into
open-telemetry:mainfrom
yigitcan-ozturk:fix/otlp-grpc-functional-teardown
Draft

[TEST] Make OTLP gRPC functional teardown deterministic#4541
yigitcan-ozturk wants to merge 1 commit into
open-telemetry:mainfrom
yigitcan-ozturk:fix/otlp-grpc-functional-teardown

Conversation

@yigitcan-ozturk

Copy link
Copy Markdown
Contributor

Contributes to #4489.

Problem

func_otlp_grpc intermittently exits with SIGSEGV / heap-corruption signatures during TLS failure cases after the normal ctest suite has already passed. The failure still reproduces with gRPC 1.83.1, so the earlier OpenSSL NO_ATEXIT explanation is not sufficient on its own.

Hypothesis

The functional binary is short-lived. C++ channel wrappers release their own gRPC runtime references as the OpenTelemetry exporter/provider is destroyed, and the last release can initiate asynchronous gRPC global teardown immediately before main() returns. TLS failure cases leave enough EventEngine/background teardown work to expose that race intermittently.

Change

  • hold one explicit grpc_init() reference for the functional-test invocation;
  • after the test has destroyed its OpenTelemetry/gRPC objects, release that final reference with grpc_shutdown_blocking();
  • keep the change scoped to the functional test binary; no exporter/runtime production behavior changes.

This is intentionally a draft while CI and maintainer review validate whether the teardown barrier removes the intermittent failure. If the hypothesis is wrong, I will not promote this as a fix.

Validation target

The important signal is repeated maintainer-mode CMake coverage of the TLS functional cases, especially the paths that have previously failed with exit 134/139.

@codecov

codecov Bot commented Sep 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.46%. Comparing base (bcd4f2d) to head (ea78fa2).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4541      +/-   ##
==========================================
- Coverage   83.47%   83.46%   -0.01%     
==========================================
  Files         521      522       +1     
  Lines       20380    20440      +60     
==========================================
+ Hits        17011    17059      +48     
- Misses       3369     3381      +12     

see 10 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant