Add automatic X-Opaque-Id headers to datastore msearch requests

[jwils]

Elasticsearch and OpenSearch propagate the X-Opaque-Id HTTP header into the tasks API, slow logs, and deprecation logs. This PR now makes ElasticGraph set useful opaque ids itself for datastore _msearch traffic instead of offering an opt-in Faraday middleware that applications wire manually.

What changed

• Removes the standalone ElasticGraph::Support::FaradayMiddleware::OpaqueId middleware and its opt-in datastore_client_customization_block design.
• Adds ElasticGraph::GraphQL::Client#extra_opaque_id_parts as the GraphQL customization hook.
• elasticgraph-graphql now automatically includes X-Opaque-Id on datastore _msearch requests using stable EG-owned parts:
  • elasticgraph-graphql
  • client=<client.name>
  • any client.extra_opaque_id_parts
  • query=<fingerprint>
• Threads those parts through QueryExecutor -> QuerySource -> DatastoreSearchRouter rather than relying on Faraday env customization.
• Adds coarse opaque ids for other EG-owned _msearch call sites:
  • elasticgraph-health_check -> elasticgraph-health_check
  • Apollo test resolver -> elasticgraph-apollo;resolver=product
  • elasticgraph-indexer#source_event_versions_in_index -> elasticgraph-indexer;purpose=source_event_versions;cluster=<cluster>
• Uses stable/readable values only; no per-request random suffix.

Why this shape

• Addresses the review feedback that the original PR was useful but did not really belong in ElasticGraph as generic opt-in support code.
• Keeps the feature in EG request-layer abstractions where EG already knows meaningful request metadata.
• Avoids _bulk for now. Indexer write traffic still needs a better request-identity model before we choose default opaque-id semantics there.

Notes

• A small internal ElasticGraph::Support::OpaqueID formatter remains for assembling and sanitizing header values.
• The shared X-Opaque-Id header name now lives in elastic_graph/constants.

Verification

• Updated unit/spec coverage for GraphQL routing, GraphQL::Client normalization, health-check _msearch, and indexer source-event-version lookups.
• ruby -c passed on the touched Ruby files.
• git diff --check is clean.
• I could not run the full Ruby test suite in this workspace because the local bundle is incomplete and is missing gems such as aws_lambda_ric, faker, httpx, and nokogiri.

[myronmarston]

Left some mild feedback. I'm also curious if you can describe how you plan to use these ids (as I've not seen them before)? During an incident will you list the tasks on nodes having problems (which would include the opaque ids, I guess?) and then use those opaque ids to cancel requests? Are the properties that matter uniqueness and ability to correlate it to a client of your application so that you can understand the consequences of cancelling a task?

[myronmarston]

Would it be useful to set OPAQUE_ID from the bulk calls as well?

[myronmarston]

Would it be useful to set OPAQUE_ID from the bulk calls as well?

I think this is still worth considering in a followup PR if you have any ideas for what to use.

[jwils]

Yep, I want to think through adding this, but haven't decided what is best to include yet. Will do this as a followup

[myronmarston]

It's not clear to me why cluster is useful here. At the point you're seeing these opaque ids (e.g. in an OpenSearch task list) you'll know what cluster you're hitting, right? Or are you expecting these opaque ids to wind up in a centralized logging system that contains opaque ids data from multiple clusters?

In terms of other potential useful bits of data to include...maybe include the count of operations per type? e.g. widget_ops=7;team_ops=12

That would indicate at a glance how large of request it was and which types the request is for.

[myronmarston]

Is it worth including of types being queried here? You could change the caller from datastore_msearch(recency_queries_by_type_name.values) to datastore_msearch(recency_queries_by_type_name) and then you'd have the queried types as keys and the queries themselves as values. The types could nicely go in the opaque id.

[myronmarston]

Would it be useful to set OPAQUE_ID from the bulk calls as well?

I think this is still worth considering in a followup PR if you have any ideas for what to use.