[DAB] postgres_endpoints: GET returns 404 for an endpoint that list-endpoints returns correctly

[nicklaw5]

Summary

After creating a new compute endpoint via the postgres_endpoints bundle resource (or equivalently POST .../endpoints), the single-resource lookup GET .../endpoints/{endpoint_id} reliably returns 404 NOT_FOUND for that endpoint — even though GET .../branches/{branch}/endpoints (list) correctly includes it with full status. This breaks the direct engine's post-create operation poll, and also breaks bundle deployment bind, leaving the bundle permanently unable to reconcile state for that resource.

This reproduces regardless of the endpoint's current_state — it was first observed while the endpoint was IDLE (auto-suspended), and continues to reproduce after the endpoint transitioned to ACTIVE. A second, pre-existing endpoint on the same branch (created directly rather than via this bug's code path) resolves fine via the same single-resource GET, so the failure is specific to this endpoint, not to endpoint state.

Environment

• databricks bundle CLI v1.6.0
• DATABRICKS_BUNDLE_ENGINE=direct
• Resource: postgres_endpoints, endpoint_type: ENDPOINT_TYPE_READ_ONLY

Repro steps

1. Deploy a bundle containing a new postgres_endpoints resource:

   resources:
     postgres_endpoints:
       read_replica_1:
         endpoint_id: read-replica-1
         parent: projects/my-project/branches/production
         endpoint_type: ENDPOINT_TYPE_READ_ONLY
         autoscaling_limit_min_cu: 0.5
         autoscaling_limit_max_cu: 0.5
         suspend_timeout_duration: 300s

   databricks bundle deploy --target production
   

2. The endpoint is created successfully (visible in the Databricks UI and via list-endpoints), but the CLI's post-create operation poll 404s:

   Error: cannot create resources.postgres_endpoints.read_replica_1: endpoint not found (404 NOT_FOUND)
   GET https://<workspace-host>/api/2.0/postgres/projects/<project-id>/branches/production/endpoints/read-replica-1/operations/<op-id>
   

   Deploy state is never updated to reflect that the endpoint exists.

3. Re-running deploy attempts to create the endpoint again, and now fails with a conflict instead:

   Error: cannot create resources.postgres_endpoints.read_replica_1: endpoint with such id already exists in the branch (400 BAD_REQUEST)
   POST https://<workspace-host>/api/2.0/postgres/projects/<project-id>/branches/production/endpoints?endpoint_id=read-replica-1
   

4. Attempting to reconcile via bundle deployment bind fails the same way, since it also depends on the single-resource GET:

   databricks bundle deployment bind read_replica_1 "projects/<project-id>/branches/production/endpoints/read-replica-1" --target production
   Error: failed to fetch the resource, err: endpoint not found
   

5. Confirmed directly via databricks postgres get-endpoint (reproduced 3x consecutively, several seconds apart — not a transient propagation delay):

   databricks postgres get-endpoint "projects/<project-id>/branches/production/endpoints/read-replica-1"
   Error: endpoint not found
   

6. Confirmed at the raw REST layer, bypassing the CLI entirely:

   GET /api/2.0/postgres/projects/<project-id>/branches/production/endpoints/read-replica-1
   → {"error_code":"NOT_FOUND","message":"endpoint not found"}
   

7. Meanwhile, list-endpoints on the same parent branch correctly returns full status for the endpoint:

   databricks postgres list-endpoints "projects/<project-id>/branches/production"
   [
     {"endpoint_id":"primary", "status": {"current_state": "ACTIVE", ...}},
     {"endpoint_id":"read-replica-1", "status": {"current_state": "IDLE", ...}}
   ]
   

8. The endpoint later transitioned to current_state: ACTIVE (e.g. after a connection resumed it). Repeating step 5 against the now-ACTIVE endpoint still returns 404 NOT_FOUND — ruling out endpoint state as the cause.

Expected behavior

GET .../endpoints/{endpoint_id} should return the same endpoint that list-endpoints returns for the parent branch. The CLI's create/poll and bind flows depend on this single-resource GET being consistent with the list endpoint.

Actual behavior

The single-resource GET 404s indefinitely for an endpoint that list-endpoints reports as existing — reproduced both while the endpoint was IDLE and after it became ACTIVE. This leaves the bundle's deployment state permanently unable to reconcile: deploy retries hit a create conflict, and bundle deployment bind cannot fetch the resource to bind it.

Impact

Once a postgres_endpoints resource hits this bug, the owning bundle becomes stuck — deploys touching that resource fail with either a 404 (during create/poll) or a 400 conflict (on retry), and there is no CLI-level way to recover without direct API/support intervention.

[janniklasrose]

I'm unable to reproduce this. Does the issue happen consistently every time you deploy a postgres_endpoint in a fresh bundle?

[nicklaw5]

Hi @janniklasrose. I only noticed the issue when attempting to add a read replica endpoint to an existing postgres project that only contained the original primary read-write endpoint.

Seemed to work fine when doing a complete fresh deploy with target set to development. Not sure of that helps at all.