fix(server): configure finite DNS cache TTL

[bitflicker64]

Purpose of the PR

• close [Bug][HStore][Kubernetes] Data writes remain unavailable after Store pod replacement due to stale DNS resolution #3124

Problem, end to end

1. A Store pod is replaced in Kubernetes and comes back on a new IP behind the
   same stable DNS name.
2. HugeGraph Server runs with HugeSecurityManager installed by default.
3. With a SecurityManager present and no explicit policy, Java 11 falls back to
   networkaddress.cache.ttl = -1, so InetAddressCachePolicy is FOREVER.
4. The Server therefore keeps resolving the Store name to the old pod IP, and
   HStore writes keep going to an address that no longer exists.
5. Nothing recovers this until the Server process is restarted.

The setting is only honoured as a security property. The ordinary
-Dnetworkaddress.cache.ttl system property has no effect here, which is why
that form is deliberately not used.

Scope

In scope — the Server JVM's DNS caching policy, end to end along one path:

hugegraph-server.sh builds the JVM arguments -> the JVM loads the packaged
security properties -> HugeGraphServerBootstrap validates the effective
policy and installs HugeSecurityManager -> control passes to
HugeGraphServer.

The guarantee this PR is trying to make: with the security check enabled, the
Server either starts with a finite positive DNS TTL, or it does not start at
all. Silently falling back to an indefinite cache is treated as a bug, so
every failure mode along that path fails closed.

Boundary — this makes the Server re-resolve the name within the TTL. It
does not decide what happens to connections that were already open against the
old address.

Out of scope

• Store-client channel invalidation after re-resolution, tracked by [Bug][HStore Client] gRPC stub pools bind every entry to one ManagedChannel #3125
• HStore readiness changes
• Helm chart changes
• Replacing the HugeSecurityManager sandbox so the security check can run on
  JDK 24+ (JEP 486). This PR only reports that incompatibility clearly instead
  of failing with a raw VM initialization error.

Main Changes

• Add a packaged conf/java-security.properties with
  networkaddress.cache.ttl=30, loaded through -Djava.security.properties
  only when HugeSecurityManager is enabled.
• Validate the effective networkaddress.cache.ttl in a small
  HugeGraphServerBootstrap before the server starts, so a missing, unreadable
  or infinite policy fails fast rather than silently restoring the indefinite
  cache. Because it reads the effective Security.getProperty, it also covers
  policies injected through _JAVA_OPTIONS or an operator override.
• Reject a security-enabled startup on JDK 24+ with an actionable message.
  JEP 486 removed the Security Manager there: -Djava.security.manager=allow
  is a fatal VM initialization error and System.setSecurityManager() always
  throws. Security-disabled startup on JDK 24+ is unchanged.
• Parse the java -version output by selecting the JVM banner line, anchored
  to its java version/openjdk version prefix, instead of head -1 or the
  first line containing version ". A preamble is printed whenever
  JAVA_TOOL_OPTIONS/_JAVA_OPTIONS is set, which is how APM agents are
  normally installed, and such an agent may print its own version "..."
  banner: reading either line leaves JAVA_VERSION unusable or reports the
  agent's version, which can reject a supported JDK or trip the JDK 24+ guard.
• Name the bundled policy file in the server log when it is missing or
  unreadable, so an upgrade that reuses an older conf/ reports the cause in the
  log start-hugegraph.sh points operators at. An operator override suppresses
  only this missing-file diagnostic, never validation: in daemon mode, where
  stderr only reaches hugegraph-server-stdout.log, the bootstrap mirrors its
  rejection of the effective policy — a missing, unreadable or infinite-TTL
  override included — into that same server log.
• Keep security-disabled startup unchanged, and preserve later operator JVM
  options including an explicit security-properties override.
• Run the focused Java runtime and launcher coverage in an independent RocksDB
  CI step, so it is not gated behind the broader startup prerequisites.

Verifying these changes

• Need tests and can be verified as follows (Java 11):
  • Distribution package build: passed.
  • test-java-security-properties.sh: passed.
  • Existing Server startup-script suite: 0 failed.
  • HugeGraphServerBootstrapTest: 2 passed, 0 failed.
  • Checkstyle: 0 violations. Apache RAT: 0 unapproved licenses.

Fail-closed behaviour was probed directly: an infinite TTL injected through
_JAVA_OPTIONS, and a -Dnetworkaddress.cache.ttl=30 system property paired
with an infinite security-properties file, are both rejected.

Each new assertion was confirmed non-vacuous by mutating the production code and
checking the test then fails:

Mutation	Result
Point the launcher at a missing bootstrap class	ClassNotFoundException passed the previous negative-only assertion, fails the new positive-sentinel one
Raise the JDK cap so the JDK 24+ guard never fires	boundary test fails
Restore head -1 version parsing	preamble regression test fails
Un-anchor the version match back to any version " line	agent-banner regression fails
Drop the missing-policy diagnostic	missing-file test fails
No-op the daemon-mode bootstrap error-log property	daemon-mode override tests fail

Does this PR potentially affect the following parts?

• Dependencies
• Modify configurations
• The public API
• Other affects
• Nope

Documentation Status

• Doc - TODO
• Doc - Done
• Doc - No Need

[codecov]

Codecov Report

❌ Patch coverage is 17.07317% with 34 lines in your changes missing coverage. Please review.
✅ Project coverage is 39.11%. Comparing base (b9710a7) to head (791947c).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
.../hugegraph/bootstrap/HugeGraphServerBootstrap.java	17.07%	29 Missing and 5 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3126      +/-   ##
============================================
- Coverage     39.19%   39.11%   -0.08%     
  Complexity      264      264              
============================================
  Files           770      771       +1     
  Lines         65779    65820      +41     
  Branches       8726     8732       +6     
============================================
- Hits          25779    25745      -34     
- Misses        37247    37323      +76     
+ Partials       2753     2752       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[imbajin]

Blocking: yes. Summary: The DNS TTL policy can silently disappear with a retained configuration directory, and its regression test can be skipped by unrelated startup prerequisites. Evidence: exact-head static review, a controlled missing-security-properties reproduction, workflow/log inspection, and git diff --check; codecov/project is failing.

[Copilot]

🟢 Ready to approve

The changes are narrowly scoped to Server startup wiring, include a packaged configuration, and add CI coverage validating both the security property value and argument ordering/overrides.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR addresses stale JVM DNS caching when HugeGraph Server runs with HugeSecurityManager enabled (Java 11 behavior), which can prevent HStore writes from recovering after a Kubernetes Store pod is replaced behind the same DNS name.

Changes:

• Add a packaged security properties file setting networkaddress.cache.ttl=30.
• Wire the file into Server startup via -Djava.security.properties=... only when HugeSecurityManager is enabled.
• Add a dedicated CI startup-time verification script and run it in Server CI before existing startup tests.

File summaries

File	Description
hugegraph-server/hugegraph-dist/src/assembly/travis/test-java-security-properties.sh	Adds a focused runtime + launcher-argument test to verify the security property value and startup wiring behavior.
hugegraph-server/hugegraph-dist/src/assembly/static/conf/java-security.properties	Introduces the packaged security properties file that sets a finite positive DNS cache TTL.
hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh	When security is enabled, adds -Djava.security.properties=... alongside HugeSecurityManager to ensure finite DNS caching.
.github/workflows/server-ci.yml	Runs the new security-properties verification script as part of the existing Server dist startup test step.

Review details

• Files reviewed: 3/4 changed files
• Comments generated: 0
• Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[imbajin]

Blocking: yes. Summary: The security-enabled launcher cannot start on documented JDK 24+, and its success-path regression can accept unrelated early JVM failures. Evidence: six independent exact-head review lanes; OpenJDK JEP 486 and static launcher/bootstrap analysis; HugeGraphServerBootstrapTest passed 2/2 on JDK 11; git diff --check passed; visible exact-head checks are green.

[bitflicker64]

Follow-up review found two issues in my own earlier commit, both fixed in 9438a94:

1. The JDK 24+ guard could be defeated by an env var. java -version prints Picked up JAVA_TOOL_OPTIONS: ... before the version line whenever JAVA_TOOL_OPTIONS/_JAVA_OPTIONS is set, so the existing head -1 parse left JAVA_VERSION unusable and every version-gated branch silently evaluated false — the new guard never fired, and -Djava.security.manager=allow was dropped on JDK 18-23. Now the version line is selected explicitly and a non-numeric result is rejected.
2. A missing conf/java-security.properties failed with the reason in the wrong log. The bootstrap's stderr goes to hugegraph-server-stdout.log in daemon mode, while start-hugegraph.sh points operators at hugegraph-server.log. The launcher now names the missing file in that log.

Both are covered by new regression tests, and each was confirmed non-vacuous by reverting the fix and checking the test fails.

[Copilot]

🟡 Not ready to approve

The new test-java-security-properties.sh reintroduces fragile java -version | head -1 parsing (can break under JAVA_TOOL_OPTIONS preambles), and the launcher logs a misleading “missing bundled properties” error even when an operator override is already provided.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (1)

hugegraph-server/hugegraph-dist/src/assembly/travis/test-java-security-properties.sh:63

• JAVA_MAJOR parsing uses head -1 on java -version output, which breaks when the JVM prints a preamble like Picked up JAVA_TOOL_OPTIONS: ... (the same issue this PR fixes in hugegraph-server.sh). With set -euo pipefail, a non-numeric JAVA_MAJOR can also make the -ge test abort the script unexpectedly.

JAVA_MAJOR=$($JAVA_BIN -version 2>&1 | head -1 | cut -d'"' -f2 |
             sed 's/^1\.//' | cut -d'.' -f1)
SECURITY_MANAGER_OPTION=""
if [[ "$JAVA_MAJOR" -ge 18 ]]; then
    SECURITY_MANAGER_OPTION="-Djava.security.manager=allow"
fi

• Files reviewed: 5/6 changed files
• Comments generated: 0 new
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[imbajin]

Blocking: yes. Summary: JVM version parsing can still select a tool/agent preamble, and invalid operator policy overrides suppress the actionable upgrade diagnostic. Evidence: six independent exact-head review lanes, focused shell reproductions, HugeGraphServerBootstrapTest passed 2/2, git diff --check passed, and visible exact-head checks are green.

[imbajin]

⚠️ This still accepts the first arbitrary line containing version ", not specifically the JVM banner. For example, APM agent version "7.2.0" before openjdk version "21.0.8" makes this pipeline return 7, so a supported JDK is rejected; a preamble containing version "24" can likewise trigger the new upper-bound branch. The test helper uses the same broad match, so it cannot catch this. Please anchor the match to the actual java/openjdk version line and add a tool-options/agent preamble regression.

[bitflicker64]

Fixed in 9d53f15. The match is now anchored to the JVM banner line itself — only lines starting with java version " or openjdk version " are considered — in both the launcher and the test helper's own JAVA_MAJOR parse.

Added two mock-JVM regressions where an agent banner precedes the JVM's:

• Elastic APM agent version "7.2.0" ahead of a JDK 21 banner must still emit -Djava.security.manager=allow (the unanchored match reads 7 and rejects the runtime as below the minimum).
• APM agent version "24.0.1" ahead of a JDK 11 banner must not trip the JDK 24+ guard and must reach the bootstrap without a security-manager option.

Both fail against the unanchored match (re-verified by mutating the packaged launcher back to /version "/).

[imbajin]

⚠️ Any non-empty -Djava.security.properties=... token suppresses the missing bundled-policy diagnostic without checking whether the replacement file exists, is readable, or supplies a valid positive TTL. The bootstrap then rejects the invalid policy, but daemon stderr goes to hugegraph-server-stdout.log while start-hugegraph.sh directs operators to hugegraph-server.log, which now lacks the cause. Please suppress this diagnostic only after validating the effective override, or mirror bootstrap failures into the advertised log, and cover missing/unreadable/invalid override files in daemon mode.

[bitflicker64]

Fixed in 791947c, via the second option: the bootstrap's rejection is mirrored into the advertised log rather than validated shell-side.

I deliberately did not add launcher-side validation of the override file. Only the JVM's own properties parsing decides what the override loads to — the escaped-key, line-continuation and file:-URL cases in the test suite are all files a plain shell check would mis-classify — so a shell validator could disagree with the bootstrap in both directions and reintroduce exactly this inconsistency. The bootstrap stays the single validator.

Mechanics: in daemon mode the launcher passes -Dhugegraph.bootstrap.error.log=${LOGS}/hugegraph-server.log, and the bootstrap appends the same fatal message it prints to stderr (including the effective java.security.properties value, so the broken override is named) to that file, best-effort, before the security manager is installed and without touching the logging framework. Stdout mode is unchanged since stderr is already operator-visible there.

New daemon-mode tests cover missing, unreadable and infinite-TTL operator overrides, asserting both the cause and the override path land in hugegraph-server.log; the unreadable fixture also carries invalid content so the case still fails closed where permission bits do not apply (root). No-op'ing the error-log property in the packaged launcher makes them fail.