[Feature] Orchestrate graph creation through PD: new graphs are not consistently available across Server replicas ("Could not rebind [g]")

[bitflicker64]

Summary

In distributed mode (PD + HStore), creating a graph returns 200 before the
graph is usable everywhere, and in the worst case before it is usable
anywhere. The creating Server registers its own Gremlin binding through an
event it does not await, and every other Server replica converges
independently through a PD metadata watch plus a local backend open. There is
no upper bound on that window and no signal when it closes. Any deployment
that load-balances several Servers turns the window into a user-visible 400:
a query lands on a replica that has not yet bound the graph and fails with
TinkerPop's "Could not rebind" error.

I propose fixing this in three phases: make the creating replica synchronous
(a one-line parity fix), give clients a way to await cluster-wide
availability, and finally move graph creation into PD so Servers become pure
converging replicas of PD state. The last step is the same direction as
#3118 / #3119, which make Server-side init-store skippable in distributed
mode because the storage side already owns its metadata.

Reproducible symptom

Setup: current master images, PD + HStore, two or more Servers behind one
load-balanced endpoint (in Kubernetes, a ClusterIP Service; any round-robin
proxy reproduces it). Timeline:

t0  client -> LB -> Server A : POST create graph "graph_164" (DEFAULT space)
t1  Server A : opens backend, writes config to PD meta, puts the graph in
               its local map, fires GRAPH_CREATE without awaiting it,
               returns 200
t2  client -> LB -> Server B : POST /gremlin with
               aliases {"g": "__g_DEFAULT-graph_164"}
t3  Server B : PD watch not yet delivered, or delivered but the backend
               open is still running; the Gremlin global bindings contain
               no __g_DEFAULT-graph_164
t4  client <- 400 : Could not rebind [g] to [__g_DEFAULT-graph_164] as
               [__g_DEFAULT-graph_164] not in the Graph or TraversalSource
               global bindings
tN  Server B : watch fires -> reads config from PD -> opens the backend
               graph -> injects the binding (also via a non-awaited event);
               only now has this one replica converged

Observed end to end with Hubble driving such a deployment: creating a graph
succeeds, and the immediately following index build fails with

[2104] execute indexing failed ... Could not rebind [g] to
[__g_DEFAULT-graph_164] as [__g_DEFAULT-graph_164] not in the Graph or
TraversalSource global bindings

The [2104] prefix is Hubble's error wrapper; the embedded text is the 400
body from the Gremlin endpoint. Retrying the same request after a few
seconds succeeds once the replica converges, which is exactly the signature
of the race rather than of a broken graph.

Where the window comes from (verified on master @ 1716c77)

All paths below are relative to
hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph unless
stated otherwise.

1. The creating Server does not await its own binding registration.
   core/GraphManager.java#createGraph(String, String, String, Map, boolean)
   (line 1222) on the PD path opens the graph (line 1334), persists the
   config to PD meta and publishes the add notification (lines 1346 to 1348,
   metaManager.addGraphConfig and metaManager.notifyGraphAdd), puts the
   graph into the REST-side map (line 1350), then fires
   this.eventHub.notify(Events.GRAPH_CREATE, graph) at line 1356 and moves
   on. EventHub.notify submits the listener invocation to an executor and
   returns a Future
   (hugegraph-commons/hugegraph-common/src/main/java/org/apache/hugegraph/event/EventHub.java,
   lines 166 and 192), so the 200 can be written before any listener has
   run. The non-PD local path already does this correctly:
   createGraphLocal calls notifyAndWaitEvent(Events.GRAPH_CREATE, graph)
   at line 1206, whose implementation (line 1774) blocks on future.get().
   The PD path lost that parity.

2. The Gremlin binding only exists after the event listener runs.
   auth/ContextGremlinServer.java listens for GRAPH_CREATE (lines 66 to
   73) and injectGraph (lines 126 to 141) registers the graph and a
   traversal source named G_PREFIX + name with G_PREFIX = "__g_" (line
   49) into TinkerPop's server-side GraphManager and global bindings. Every
   Gremlin request that aliases g hits those bindings: the REST
   /gremlin endpoint (api/gremlin/GremlinAPI.java, @Path("gremlin")
   line 42, forwarding at line 68) proxies to the embedded Gremlin Server
   (api/gremlin/GremlinQueryAPI.java line 61,
   ServerOptions.GREMLIN_SERVER_URL), where TinkerPop's
   HttpGremlinEndpointHandler rejects unknown aliases with exactly the
   observed message (TinkerPop 3.5.1, pinned by tinkerpop.version in
   hugegraph-server/pom.xml line 45; the format string is at
   HttpGremlinEndpointHandler.java line 299 in the 3.5.1 tag). The Cypher
   endpoint builds the same alias server-side
   (api/cypher/CypherAPI.java line 115, aliases.put("g", "__g_" + graphInfo)), so it fails identically. The binding name in the error,
   __g_DEFAULT-graph_164, is G_PREFIX plus spaceGraphName joined with
   DELIMITER = "-" (core/GraphManager.java lines 151 and 263), which
   confirms the failure is this lookup and not a client-side typo.

3. Other replicas converge through a watch with no bound and no signal.
   notifyGraphAdd is a KV put of the graph name onto a well-known key
   (hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/meta/managers/GraphMetaManager.java
   line 77). Every Server listens on that key through PD's watch mechanism
   (meta/MetaManager.java line 239 delegating to GraphMetaManager line
   168, backed by meta/PdMetaDriver.java line 111, which registers a PD
   client watch). The handler, core/GraphManager.java#graphAddHandler
   (line 2370), reads the config back from PD meta and calls
   createGraph(..., init=false) at line 2415, which opens the backend
   stores against PD/HStore and then fires the same non-awaited
   GRAPH_CREATE event from step 1. A replica's convergence latency is
   therefore watch delivery plus a full backend open plus asynchronous
   event dispatch. Nothing reports completion back to PD, the creating
   Server, or the client.

4. Convergence failures are silent and terminal. In graphAddHandler, a
   failed open is only logged when startIgnoreSingleGraphError is set
   (line 2427); under the current implementation there is no retry and no
   health signal, so a replica that hits this path continues serving while
   remaining diverged, and every probe still passes.

Why readiness probes cannot mask this

While testing distributed deployments on Kubernetes, the natural first
instinct is "gate the Service on readiness". It cannot work here:

• Readiness is evaluated per pod against a startup-time contract. Graphs
  are created at runtime, after every replica is already Ready, so no probe
  definition can reference them.
• Making readiness mean "this replica has bound every graph in PD meta"
  would remove every replica from the Service on each creation, trading a
  scoped 400 for a cluster-wide outage, and would still not cover the
  creating replica's own gap between the 200 and its asynchronous
  injectGraph.
• The failure is per graph, not per process. A replica that missed one
  graph (step 4 above) is healthy for every other graph; readiness is the
  wrong granularity to express that.

This is a data-plane consistency question, and only the creation protocol
itself can answer it.

Proposal, in phases

Phase 1, quick win: make the creating replica consistent at 200. Change
line 1356 of core/GraphManager.java from this.eventHub.notify(...) to
this.notifyAndWaitEvent(Events.GRAPH_CREATE, graph), restoring parity
with the local path at line 1206. After this, the Server that answered the
create can always serve the graph immediately, which makes sticky routing a
complete client-side workaround instead of a probabilistic one. Two details
worth deciding during review: notifyAndWaitEvent currently swallows
listener failures with a warning (line 1774), so a failed injection should
probably fail the create instead; and the wait occupies a REST worker for
the duration of the binding, which is the same cost the local path already
pays.

Phase 2: let clients await cluster-wide availability. Give each Server a
per-graph status it writes to PD meta after its own injectGraph
completes, and expose the aggregate through the REST API, for example
GET /graphspaces/{space}/graphs/{name} gaining a status with
CREATING / READY(n of m) / FAILED(replica) semantics, or a dedicated
status endpoint. Optionally add a blocking flag to the create call that
returns only when all currently registered Servers report READY. This also
turns the silent divergence of step 4 into something operators can see and
alert on.

Phase 3, the architectural direction: PD owns graph creation. The create
API (on any Server, or on PD directly) becomes a request to PD; PD
validates, persists the config, and records the desired state; Servers
reduce to pure converging replicas: watch, open, bind, report status.
The creating Server no longer has a privileged code path at all, which
removes the asymmetry that produced this race. This is the same direction
already taken for initialization: #3118 argued, and #3119 implements
(under review), that Server-side InitStore is unnecessary in PD/HStore
mode because storage-side metadata is owned by PD, gated by a dedicated
init_store.enabled option. Graph creation is the remaining place where a
single Server locally mutates cluster-level state and then hopes the rest
of the cluster catches up. Moving it behind PD finishes that thought.

Interactions and risks

• Ordering: the add notification is a plain KV put of the latest graph
  name onto one shared key (GraphMetaManager.java line 77). The phase 2
  status keys should be per graph, not piggybacked on that channel, so
  rapid consecutive creations cannot coalesce.
• Rollback: today addGraphConfig lands in PD before schema template
  preparation runs (prepareSchema, line 1365), so a late failure on the
  creating Server leaves a config other replicas will happily converge on,
  with the client seeing an error. Phase 3 needs an explicit creation
  state machine in PD (PENDING until commit, cleanup on failure) rather
  than inheriting this publish-then-finish order.
• Mixed-version clusters: phase 1 is purely local to one Server and safe
  to roll. Phase 2 must treat replicas that never write status keys (old
  Servers) as UNKNOWN and document that clients fall back to retry. Phase
  3 changes ownership and needs either a PD-side compatibility mode that
  still accepts Server-initiated creation, or a cluster version gate.
• Failure visibility: any phase that makes creation synchronous or
  status-reporting must decide what a partial failure returns; silently
  logging and returning 200, as the event path does today, is the current
  behavior being fixed, not a fallback to keep.

Context

While testing distributed deployments on Kubernetes I hit this window
directly. For reference, the deployment tooling is the Helm chart
contributed in PR #3132 (issue #3131); it fronts the Server replicas
with one load-balanced Service, which is what makes the window user
visible: Hubble's ordinary create-then-index flow trips it on the first
try. The chart README currently documents the window as a known limitation
with retry, sticky routing, and per-replica graph-list polling as
mitigations. Those are workarounds; every multi-Server deployment behind
any load balancer will rediscover this until the creation protocol closes
the window itself. I am happy to start with a PR for phase 1 and a
concrete API sketch for phase 2 if the direction sounds right.

[bitflicker64]

Possible implementation roadmap, splitting the proposal above into independently reviewable steps:

Current problem
      |
      v
Graph creation is eventually consistent across Server replicas.
CreateGraph returns before the graph is usable cluster-wide.

PR 1 (small, low risk)

• Make the creating Server wait for its own GRAPH_CREATE event: replace notify() with notifyAndWaitEvent() on the PD path, matching what the local path already does.
• Guarantees the creating Server can serve the graph the moment CreateGraph returns.

PR 2 (medium)

• Introduce graph readiness reporting: each Server reports READY / LOADING / FAILED for a graph to PD.
• Expose cluster-wide creation status through an API that clients can await, closing the cross-replica window.

PR 3 (larger, architectural)

• Move graph creation ownership entirely into PD; Servers become pure replicas converging on PD metadata.
• The special "creating Server" path disappears, and the failure modes in the risks section above (log-only convergence failure, config persisted before schema preparation) go away by construction.

PR 1 is independently useful and could land while the rest is discussed. I am happy to start on it if the direction sounds right.