Cache Vault KV version by mount path to reduce preflight calls (CONCOURSE_VAULT_ENABLE_KV_MOUNT_CACHE)

[IvanChalukov]

Changes proposed by this PR

closes #9514

This PR adds KV mount version caching for Vault so Concourse does not run mount preflight on every secret read.

What changed

• Cache KV mount version by mount path.
• Reuse cached value on subsequent reads under the same mount.
• Keep preflight behavior for cache misses.

Notes to reviewer

Enablement

• Feature flag env var: CONCOURSE_VAULT_ENABLE_KV_MOUNT_CACHE. Values are "true" or "false". Default behavior in this change: cache is disabled unless explicitly set (set to "true" to enable)
• For var_sources:

var_sources:
  - name: main
    type: vault
    config:
      url: https://vault:8200
      path_prefix: /concourse
      namespace: main
      lookup_templates:
        - /main/test-iamge/{{.Secret}}
      auth_backend: token
      client_token: ...
      insecure_skip_verify: true
      enable_kv_mount_cache: true

[Kump3r]

Looks good to me, I tested by configuring the compose yaml with an internal Vault we use trough approle auth. I used debug logs as I do not have server side access to the Vault and the web container doesn't have ngrep, wireshark etc. and the cache is in memory. As I already spoke to you what is TBD from my perspective is the docs, helm and bosh packaging stuff. Logs from testing in collapsed details section.

Details

Logs enabled:

docker logs fork-concourse-ivan-web-1 | grep kv-mount-cache
{"timestamp":"2026-04-20T08:30:12.879820721Z","level":"debug","source":"atc","message":"atc.credential-manager.kv-mount-cache-miss","data":{"name":"vault","secret-path":"test/test/concourse/test-vault/test-secret/","session":"7"}}
{"timestamp":"2026-04-20T08:30:13.005661930Z","level":"debug","source":"atc","message":"atc.credential-manager.kv-mount-cache-fetched-from-vault","data":{"is-v2":true,"mount-path":"test/","name":"vault","secret-path":"test/test/concourse/test-vault/test-secret/","session":"7"}}
{"timestamp":"2026-04-20T08:30:13.005750680Z","level":"debug","source":"atc","message":"atc.credential-manager.kv-mount-cache-stored","data":{"is-v2":true,"mount-path":"test/","name":"vault","session":"7"}}
{"timestamp":"2026-04-20T08:30:13.057563763Z","level":"debug","source":"atc","message":"atc.credential-manager.kv-mount-cache-hit","data":{"is-v2":true,"mount-path":"test/","name":"vault","secret-path":"test/test/concourse/test-secret/","session":"7"}}
{"timestamp":"2026-04-20T08:32:22.410623961Z","level":"debug","source":"atc","message":"atc.credential-manager.kv-mount-cache-hit","data":{"is-v2":true,"mount-path":"test/","name":"vault","secret-path":"test/test/concourse/test-vault/test-secret/","session":"7"}}
{"timestamp":"2026-04-20T08:32:22.694515545Z","level":"debug","source":"atc","message":"atc.credential-manager.kv-mount-cache-hit","data":{"is-v2":true,"mount-path":"test/","name":"vault","secret-path":"test/test/concourse/test-secret/","session":"7"}}

Logs disabled:

docker logs fork-concourse-ivan-web-1 | grep kv-mount-cache
{"timestamp":"2026-04-20T09:36:32.028043715Z","level":"debug","source":"atc","message":"atc.credential-manager.kv-mount-cache-disabled","data":{"name":"vault","secret-path":"test/test/concourse/test-vault/test-secret/","session":"7"}}
{"timestamp":"2026-04-20T09:36:32.324927382Z","level":"debug","source":"atc","message":"atc.credential-manager.kv-mount-cache-disabled","data":{"name":"vault","secret-path":"test/test/concourse/test-secret/","session":"7"}}
{"timestamp":"2026-04-20T09:38:11.160152428Z","level":"debug","source":"atc","message":"atc.credential-manager.kv-mount-cache-disabled","data":{"name":"vault","secret-path":"test/test/concourse/test-vault/test-secret/","session":"7"}}
{"timestamp":"2026-04-20T09:38:11.547221345Z","level":"debug","source":"atc","message":"atc.credential-manager.kv-mount-cache-disabled","data":{"name":"vault","secret-path":"test/test/concourse/test-secret/","session":"7"}}

Edit: let me know if you need help/reviews in the packaging stuff as well.

[taylorsilva]

For these tests, why is the cache disabled for the v2 client and enabled for the v1 client?

[taylorsilva]

nit: is this needed? A /secret mount doesn't appear to be used in the tests here.

[IvanChalukov]

It is a leftover from testing, it should be deleted as you did. Thanks for that!

[taylorsilva]

I'm not sure integration tests are very useful here. I think these tests pass regardless if the KV cache is enabled or not.

Update: yup, they pass even when the cache is disabled.

Integration tests take longer, so I'm always a bit hesitant to add them. As-is, these tests don't error if the mount cache is not being used.

[IvanChalukov]

Thanks for raising your concerns about the integration tests. I was unsure where the functionality should be validated and didn’t realize the tests would pass in both scenarios (enabled and disabled).

I’ll keep this in mind for future PRs.

[taylorsilva]

I've gone ahead and removed the integration-level tests and added unit tests instead that fail if the cache is not hit on subsequent calls.

Cleaned up the small nits I mentioned as well.

I didn't make any other changes. Everything else LGTM. Feature works as expected. TY for working on this Ivan!

[IvanChalukov]

@taylorsilva I've opened PRs for packaging stuff:

• bosh deployment introduction of env variable - Add CONCOURSE_VAULT_ENABLE_KV_MOUNT_CACHE concourse-bosh-release#195
• helm chart introduction of env variable - add CONCOURSE_VAULT_ENABLE_KV_MOUNT_CACHE concourse-chart#414
• documentation in The Vault credential manager and var_sources config - add Vault KV mount cache to doc docs#600

[taylorsilva]

You da bestttt 🙏 👏