Add data node OpenSearch startup logic

[moesterheld]

Description

Adds startup logic for deciding which of the embedded OpenSearch versions should be started.

Documented here https://graylogdocumentation.atlassian.net/wiki/x/GACnPAE

/nocl internal, part of bundling two OS versions with data node

Motivation and Context

resolves https://github.com/Graylog2/graylog-plugin-enterprise/issues/14261

How Has This Been Tested?

manually, added tests

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactoring (non-breaking change)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have requested a documentation update.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.

[Copilot]

Pull request overview

Adds startup-time decision logic in the Data Node to choose between bundled OpenSearch distributions (current vs compatibility), while ensuring data-dir compatibility checks and the “latest available” version tracking remain accurate across restarts and upgrades.

Changes:

• Introduces OpensearchVersionSelectorImpl logic to select the appropriate OpenSearch distribution based on recorded node metadata, on-disk compatibility signals, and optional auto-update testing mode.
• Centralizes data directory inspection/compatibility reporting in OpensearchDataDirCompatibilityService, reused by both preflight checks and the REST controller.
• Adds Lucene 9 “compatibility readers” (via shaded Lucene 9) plus an on-disk verification marker to avoid repeated scans once a major version has successfully opened the data dir.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
graylog2-server/src/main/java/org/graylog2/cluster/nodes/DataNodeMetadataService.java	Adds setLatestAvailableVersion API to refresh only the latest-available OpenSearch version.
graylog2-server/src/main/java/org/graylog2/cluster/nodes/DataNodeMetadataServiceImpl.java	Implements non-upserting update for latest-available version refresh.
data-node/src/main/java/org/graylog/datanode/Configuration.java	Adds hidden auto_update_opensearch testing flag and accessor.
data-node/src/main/java/org/graylog/datanode/bootstrap/preflight/OpensearchDataDirCompatibilityCheck.java	Switches preflight compatibility check to use shared compatibility service + verification marker.
data-node/src/main/java/org/graylog/datanode/configuration/OpensearchVersionSelectorImpl.java	Implements selection logic for choosing which bundled OpenSearch distribution to start.
data-node/src/main/java/org/graylog/datanode/filesystem/index/DataDirVerificationMarker.java	Introduces marker file recording which OpenSearch version last opened the data dir successfully.
data-node/src/main/java/org/graylog/datanode/filesystem/index/IndicesDirectoryParseResult.java	Adds parse result wrapper including “required distribution” and a bound-check helper.
data-node/src/main/java/org/graylog/datanode/filesystem/index/IndicesDirectoryParser.java	Adds compatibility fallback readers (Lucene 9 shaded) and returns IndicesDirectoryParseResult.
data-node/src/main/java/org/graylog/datanode/filesystem/index/OpensearchDataDirCompatibility.java	Adds compatibility result model used by REST and preflight.
data-node/src/main/java/org/graylog/datanode/filesystem/index/OpensearchDataDirCompatibilityService.java	Centralizes directory validation + parsing + error/warning collection.
data-node/src/main/java/org/graylog/datanode/filesystem/index/RequiredOpensearchDistribution.java	Defines the “current vs compat” distribution tiers and matching by major.
data-node/src/main/java/org/graylog/datanode/filesystem/index/dto/NodeInformation.java	Makes version derivation from indices resilient to null index versions.
data-node/src/main/java/org/graylog/datanode/filesystem/index/dto/ShardInformation.java	Switches Lucene min-version storage to a string to support multiple Lucene classpaths.
data-node/src/main/java/org/graylog/datanode/filesystem/index/indexreader/Lucene9ShardStatsParser.java	Adds shaded-Lucene-9 shard reader for Lucene 8 segments (OpenSearch 1 / ES 7).
data-node/src/main/java/org/graylog/datanode/filesystem/index/indexreader/ShardStats.java	Changes min Lucene version type from Lucene Version to nullable string.
data-node/src/main/java/org/graylog/datanode/filesystem/index/indexreader/ShardStatsParserImpl.java	Uses string min-version extraction and documents fallback expectations.
data-node/src/main/java/org/graylog/datanode/filesystem/index/statefile/AbstractStateFileParser.java	Extracts shared Smile decoding for state files; subclasses provide codec-frame reading.
data-node/src/main/java/org/graylog/datanode/filesystem/index/statefile/Lucene9StateFileParser.java	Adds shaded-Lucene-9 state file reader to handle older codec frames.
data-node/src/main/java/org/graylog/datanode/filesystem/index/statefile/StateFileParserImpl.java	Refactors current Lucene-based state file parsing to extend AbstractStateFileParser.
data-node/src/main/java/org/graylog/datanode/opensearch/statemachine/tracer/OpensearchVersionTracer.java	Refreshes latest-available version independently and writes verification marker on AVAILABLE.
data-node/src/main/java/org/graylog/datanode/rest/CompatibilityResult.java	Simplifies List typing and keeps REST response structure consistent.
data-node/src/main/java/org/graylog/datanode/rest/IndicesDirectoryController.java	Switches REST endpoint to use shared compatibility service.
data-node/pom.xml	Adds shaded Lucene 9 dependency/version property for compatibility parsing.
data-node/src/test/java/org/graylog/datanode/bootstrap/preflight/OpensearchDataDirCompatibilityCheckTest.java	Updates tests for marker-based skipping + shared compatibility service behavior.
data-node/src/test/java/org/graylog/datanode/configuration/OpensearchVersionSelectorImplTest.java	Adds selector tests for recorded versions, bounds, marker short-circuiting, and auto-update behavior.
data-node/src/test/java/org/graylog/datanode/filesystem/index/IndicesDirectoryParserTest.java	Updates parser tests for parse result + required distribution selection.
data-node/src/test/java/org/graylog/datanode/filesystem/index/OpensearchDataDirCompatibilityServiceTest.java	Adds coverage for compatibility service errors/warnings and required distribution reporting.
data-node/src/test/java/org/graylog/datanode/filesystem/index/indexreader/Lucene9ShardStatsParserTest.java	Adds tests for shaded Lucene 9 shard reader across fixture types.
data-node/src/test/java/org/graylog/datanode/filesystem/index/indexreader/ShardStatsParserTest.java	Adjusts expectations for string-based min Lucene version.
data-node/src/test/java/org/graylog/datanode/filesystem/index/statefile/Lucene9StateFileParserTest.java	Adds tests validating shaded Lucene 9 state file parsing on fixtures.
data-node/src/test/java/org/graylog/datanode/opensearch/statemachine/tracer/InMemoryDataNodeMetadataService.java	Adds in-memory implementation of setLatestAvailableVersion for tests.
data-node/src/test/java/org/graylog/datanode/opensearch/statemachine/tracer/OpensearchVersionTracerTest.java	Extends tracer tests for latest-available refresh + verification marker recording.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[todvora]

I'd rather avoid the substring and long parsing and rather used proper semver handling if possible. Or simplify the code and provide the major version as long directly, as the 2.x.x format is not used anyhow anyway.

[todvora]

The OpensearchDistributionProvider and DatanodeDirectories are now very verbose and we should probably limit that.

[todvora]

Tested existing cluster, fresh setup and upgrade, all seems to be working perfectly fine 👍