feat: add shared PostgreSQL deployment for platform services

[maskarb]

Add a shared PostgreSQL instance that can be used by multiple platform services (Unleash, Langfuse, etc.).

Base configuration:

• Uses postgres:16 image with init scripts for database creation
• Credentials use db.* key format (db.host, db.port, db.name, db.user, db.password)
• ConfigMap with init scripts creates service databases on first startup

Kind overlay:

• Inherits base postgres:16 image and init scripts
• Includes dev credentials with default values
• Smaller PVC (1Gi) for local development

Local-dev overlay (CRC/OpenShift):

• Patches to RHEL PostgreSQL image (registry.redhat.io/rhel10/postgresql-16)
• Uses POSTGRESQL_* env vars for RHEL compatibility

Production overlay:

• Patches to RHEL PostgreSQL image
• Credentials managed separately (not in repo)

[github-actions]

Claude Code Review

Summary

This PR adds a shared PostgreSQL instance deployed via Kustomize overlays (base + kind/local-dev/production), supporting platform services like Unleash and Langfuse. The overall structure is sound and mirrors the existing Minio deployment pattern. The previous automated review was marked "outdated" — this is a fresh pass against the current diff.

The main concerns are: missing SecurityContext (project security standard), the RHEL overlay silently drops database initialization without a fallback, duplicate patch files between overlays, and a misleading comment about SQL parameterization.

---

Issues by Severity

🚫 Blocker Issues

None — safe to continue as a draft.

---

🔴 Critical Issues

1. RHEL overlays drop init-scripts without a replacement initialization path

postgresql-json-patch.yaml removes the init-scripts volume and the RHEL image does not honor /docker-entrypoint-initdb.d/, so the unleash database is never created on local-dev (CRC) or production deployments. Services will fail to connect to their expected databases.

Options to fix:

• Add a Kubernetes Job (init container or post-install hook) that creates databases after the pod is ready
• Document that service deployments are responsible for creating their own databases and remove the misleading init-script comment from the RHEL patches
• Use POSTGRESQL_ADMIN_PASSWORD + a custom entrypoint script supported by the RHEL image

2. No SecurityContext on the Pod or container

The project security standard (CLAUDE.md) requires SecurityContext on all workload pods. The PostgreSQL deployment sets neither pod-level nor container-level security context:

# Missing from spec.template.spec (pod level):
securityContext:
  runAsNonRoot: true
  fsGroup: 999  # postgres group in upstream image

# Missing from containers[0] (container level):
securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]

The upstream postgres:16 image runs as the postgres user (UID 999) so runAsNonRoot: true is viable. The RHEL image runs as UID 26 by default.

---

🟡 Major Issues

3. local-dev overlay provides no credentials Secret

kind/kustomization.yaml includes postgresql-credentials.yaml. local-dev/kustomization.yaml does not. kustomize build overlays/local-dev will succeed but the deployed pod will crash immediately because the postgresql-credentials Secret doesn't exist. There is no documentation anywhere in the overlay about this gap.

Fix: Add a local-dev/postgresql-credentials.yaml.example (similar to the base-level example) and document in the overlay kustomization that the user must create it before deploying.

4. Duplicate JSON patch files (DRY violation)

overlays/local-dev/postgresql-json-patch.yaml and overlays/production/postgresql-json-patch.yaml are byte-for-byte identical. Any future change must be made twice and the files will inevitably diverge.

Consider moving the patch to base/ (or a shared path) and referencing it from both overlays, or at minimum add a comment explaining the intentional duplication.

5. Floating image tag in base

postgres:16 is a floating tag — it will pull different minor versions over time, making deployments non-reproducible. Pin to a specific minor version, e.g. postgres:16.6.

---

🔵 Minor Issues

6. Misleading comment on init script SQL injection prevention

# Uses parameterized queries to avoid SQL injection
...
psql -c "CREATE DATABASE $db_name"

This is not a parameterized query — it is string interpolation with regex pre-validation. The regex validation (^[a-zA-Z_][a-zA-Z0-9_]*$) is effective and the approach is safe, but the comment is technically incorrect and could mislead future editors into weakening the validation thinking it's redundant. Replace the comment with: # Validated against allowlist regex above — only alphanumeric/underscore names permitted.

7. No NetworkPolicy for PostgreSQL

PostgreSQL is reachable from any pod in the namespace. Given this is a shared multi-service database, a NetworkPolicy restricting ingress to only the services that need it (Unleash, Langfuse, etc.) would be a meaningful security improvement.

8. Production overlay specifies no PVC storage class or size override

The base requests 10Gi with no storageClassName. Production clusters often require an explicit storage class for durability guarantees (e.g., gp3, ssd). This should be patched per environment rather than relying on the cluster default.

9. pg_isready probe uses shell variable expansion

command: ["/bin/sh", "-c", "pg_isready -U \"$POSTGRES_USER\""]

This works, but pg_isready does not actually authenticate — it only checks if the server accepts connections. For a true readiness check, consider also verifying the target databases exist (at least for the base/kind case where init scripts run):

pg_isready -U "$POSTGRES_USER" && psql -U "$POSTGRES_USER" -c "SELECT 1" -d unleash

---

Positive Highlights

• Clean overlay structure — correctly mirrors the Minio deployment pattern
• .gitignore correctly updated to block actual credential files from being committed
• Recreate strategy is appropriate for a single-replica stateful workload (avoids split-brain)
• PGDATA set to a subdirectory (/var/lib/postgresql/data/pgdata) — avoids the lost+found directory issue with upstream images
• Resource requests and limits are both set
• Readiness + liveness probes using pg_isready — appropriate for PostgreSQL
• Init script uses regex validation before interpolation — correctly prevents SQL injection even without true parameterization
• Init script is idempotent (checks before creating) — safe to re-run
• Kind credentials file includes a prominent DO NOT use in production warning
• Named container port (name: postgresql) allows the Service to use a named targetPort — follows best practices
• Example Secret template is well-documented with inline comments

---

Recommendations

Priority order before promoting from draft:

1. Fix RHEL database initialization (Critical Outcome: Reduce Refinement Time with agent System #1) — decide on the init strategy for RHEL environments before other services depend on this
2. Add SecurityContext (Critical Epic: RAT Architecture & Design #2) — straightforward addition, required by project standards
3. Document local-dev credentials gap (Major Epic: Data Source Integration #3) — prevents silent runtime failures for developers using CRC
4. Deduplicate JSON patch files (Major Epic: AI Agent Development #4) — prevents future maintenance drift
5. Pin image tag (Major Epic: Jira Integration & Workflow #5) — postgres:16.6 or similar
6. Items 6–9 can be addressed in follow-up PRs

---

🔍 View AI decision process (logs available for 90 days)

📋 View memory system files loaded (click to expand)

What Amber Loaded for Code Review

Amber automatically loaded these repository standards from the memory system:

1. CLAUDE.md - Master project instructions, development standards
2. backend-development.md - Go backend, K8s integration patterns
3. frontend-development.md - NextJS, Shadcn UI, React Query patterns
4. security-standards.md - Auth, RBAC, token handling
5. k8s-client-usage.md - User token vs service account patterns
6. error-handling.md - Consistent error patterns
7. react-query-usage.md - Data fetching patterns

Impact: This review used your repository's specific code quality standards, security patterns, and best practices from the memory system (PRs #359, #360) - not just generic code review guidelines.

[github-actions]

Claude Code Review

Summary

This PR adds a shared PostgreSQL deployment via Kustomize overlays (base + kind/local-dev/production) for platform services like Unleash and Langfuse. The overlay structure cleanly mirrors the existing Minio pattern. However, there is a critical blocker in the production overlay where component images were accidentally redirected to a personal registry, plus several correctness and security issues.

---

Issues by Severity

🚫 Blocker Issues

1. Production overlay redirects ALL component images to a personal registry

overlays/production/kustomization.yaml now points 5 of the platform's component images to quay.io/mskarbek/ instead of quay.io/ambient_code/. This was not mentioned in the PR description and appears to be unintentional testing residue:

# Every :latest-tagged image now pulls from a personal registry:
- name: quay.io/ambient_code/vteam_backend:latest
  newName: quay.io/mskarbek/vteam_backend      # ← should be quay.io/ambient_code/vteam_backend
- name: quay.io/ambient_code/vteam_claude_runner:latest
  newName: quay.io/mskarbek/vteam_claude_runner  # ← same
- name: quay.io/ambient_code/vteam_frontend:latest
  newName: quay.io/mskarbek/vteam_frontend       # ← same
- name: quay.io/ambient_code/vteam_operator:latest
  newName: quay.io/mskarbek/vteam_operator       # ← same
- name: quay.io/ambient_code/vteam_state_sync:latest
  newName: quay.io/mskarbek/vteam_state_sync     # ← same

The base kustomization uses :latest tags, so deploying production with this PR would pull all platform components from an uncontrolled personal registry. Only vteam_public_api remains on the org registry.

This must be reverted to quay.io/ambient_code/ for all affected images before merge.

---

🔴 Critical Issues

2. No SecurityContext on PostgreSQL pod or container

The project security standard (CLAUDE.md) requires SecurityContext on all workload pods. The PostgreSQL deployment has neither pod-level nor container-level security contexts:

# Missing from spec.template.spec:
securityContext:
  runAsNonRoot: true
  fsGroup: 999  # postgres user group in upstream image (26 in RHEL image)

# Missing from containers[0]:
securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]

The upstream postgres:16 image runs as UID 999, RHEL image as UID 26 — both support runAsNonRoot: true.

3. RHEL overlays silently drop database initialization

The kind overlay mounts init scripts at /docker-entrypoint-initdb.d/. The RHEL patches replace the entire container config but provide no equivalent — the RHEL image (registry.redhat.io/rhel10/postgresql-16) does not honor /docker-entrypoint-initdb.d/; its init hooks live under /usr/share/container-scripts/postgresql/custom/.

Result: on local-dev (CRC) and production deployments, the unleash database is never created, causing Unleash to fail on startup with no obvious error.

Fix options:

• Mount a custom init script into /usr/share/container-scripts/postgresql/custom/ in the RHEL patches
• Add a post-startup Kubernetes Job that creates databases after the pod is ready
• Document that each service is responsible for creating its own database via init container

---

🟡 Major Issues

4. local-dev overlay provides no credentials Secret

The kind overlay includes postgresql-credentials.yaml. The local-dev overlay does not. kustomize build overlays/local-dev produces valid YAML but the deployed pod crashes immediately because the postgresql-credentials Secret is absent. There is no comment or documentation about this gap in the overlay.

Fix: add a note in local-dev/kustomization.yaml directing users to the example template, and/or add a local-dev/postgresql-credentials.yaml.example.

5. Duplicate JSON patch files — will drift

overlays/local-dev/postgresql-json-patch.yaml and overlays/production/postgresql-json-patch.yaml are byte-for-byte identical. Any change requires editing both files, and they will inevitably diverge. Consider moving the patch to base/ or a shared directory and referencing it from both overlays.

6. Floating image tag in base

image: postgres:16  # ← floating — pulls different patch versions over time

Pin to a specific patch version for reproducible builds, e.g. postgres:16.6. The production RHEL image is correctly pinned (10.1) — apply the same discipline to the base image.

---

🔵 Minor Issues

7. Misleading SQL injection comment

# Uses parameterized queries to avoid SQL injection
...
psql -c "CREATE DATABASE $db_name"

This is not a parameterized query — it's string interpolation with regex pre-validation. The validation is effective and the approach is safe, but the comment is technically incorrect and could mislead future editors. Replace with: # Allowlist-validated: only alphanumeric/underscore names permitted by regex above.

8. No NetworkPolicy for PostgreSQL

PostgreSQL is reachable from any pod in the namespace. As a shared multi-service database, a NetworkPolicy restricting ingress to only the services that need it (Unleash, Langfuse) would meaningfully limit blast radius.

9. Production PVC has no storage class or size override

The base requests 10Gi with no storageClassName. Production clusters typically require an explicit storage class for durability guarantees (e.g. gp3-csi). Add a production PVC patch or document the expected cluster default.

10. PGDATA is set in base but not in RHEL patches

The base sets PGDATA=/var/lib/postgresql/data/pgdata (a subdirectory to avoid lost+found issues). The RHEL patches replace the entire env array and omit PGDATA. The RHEL image handles this internally and the volumeMount path is correctly patched to /var/lib/pgsql/data, but a brief comment explaining why PGDATA is intentionally absent from the RHEL patch would aid future maintainers.

---

Positive Highlights

• Clean overlay structure — correctly mirrors the Minio deployment pattern
• .gitignore updated before any actual secret files are added
• Recreate strategy is appropriate for a single-replica stateful workload
• PGDATA set to a subdirectory in base — avoids the lost+found mount issue
• Resource requests and limits are both set (unlike Minio)
• Readiness + liveness probes using pg_isready are appropriate for PostgreSQL
• Init script uses regex validation before interpolation — safe against SQL injection even without true parameterization
• Init script is idempotent — safe to re-run on pod restarts
• Kind credentials file has a prominent DO NOT use in production warning
• Named container port (name: postgresql) used in Service targetPort — Kubernetes best practice
• Example Secret template is well-documented with inline comments

---

Recommendations

Priority order before merge:

1. Revert personal registry references (Blocker Outcome: Reduce Refinement Time with agent System #1) — check git diff main -- overlays/production/kustomization.yaml and restore quay.io/ambient_code/ for all images
2. Fix RHEL database initialization (Critical Epic: Data Source Integration #3) — decide on init strategy for RHEL environments before other services depend on this
3. Add SecurityContext (Critical Epic: RAT Architecture & Design #2) — required by project standards, straightforward addition
4. Document local-dev credentials gap (Major Epic: AI Agent Development #4) — prevents silent runtime failures for CRC developers
5. Deduplicate JSON patch files (Major Epic: Jira Integration & Workflow #5) — prevents future maintenance drift
6. Pin image tag (Major Epic: Testing & Validation #6) — postgres:16.6 or current latest patch
7. Items 7–10 can be addressed in follow-up PRs

---

Review generated by Claude Code (claude-sonnet-4-6) against repository patterns in CLAUDE.md and .claude/context/.

---

🔍 View AI decision process (logs available for 90 days)

📋 View memory system files loaded (click to expand)

What Amber Loaded for Code Review

Amber automatically loaded these repository standards from the memory system:

1. CLAUDE.md - Master project instructions, development standards
2. backend-development.md - Go backend, K8s integration patterns
3. frontend-development.md - NextJS, Shadcn UI, React Query patterns
4. security-standards.md - Auth, RBAC, token handling
5. k8s-client-usage.md - User token vs service account patterns
6. error-handling.md - Consistent error patterns
7. react-query-usage.md - Data fetching patterns

Impact: This review used your repository's specific code quality standards, security patterns, and best practices from the memory system (PRs #359, #360) - not just generic code review guidelines.

[github-actions]

Claude Code Review

Summary

This PR adds a shared PostgreSQL deployment for platform services (Unleash, Langfuse, etc.) using a Kustomize overlay architecture. It correctly distinguishes between upstream postgres:16 (dev/e2e) and RHEL postgresql-16 (local-dev/production), migrates deprecated Kustomize syntax, and includes sensible init scripts for dev environments. The infrastructure approach is clean, but there are security and operational gaps to address before this is production-ready.

---

Issues by Severity

🚫 Blocker Issues

None — PR is in DRAFT, but the following criticals should be resolved before promoting.

---

🔴 Critical Issues

1. Missing SecurityContext on the PostgreSQL container

components/manifests/base/postgresql-deployment.yaml defines no securityContext for the container. The project standard (CLAUDE.md) requires:

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: false   # PostgreSQL needs temp writes
  capabilities:
    drop: ["ALL"]

PostgreSQL runs on port 5432 (> 1024, no NET_BIND_SERVICE needed) and typically as uid 999. This can be applied without breaking anything. Consider also adding a pod-level securityContext with runAsNonRoot: true and fsGroup: 999 to ensure the data volume is accessible.

2. No NetworkPolicy restricting database access

The postgresql Service is ClusterIP, but any pod in the ambient-code namespace can connect to port 5432. For a platform emphasizing multi-tenancy, there is no NetworkPolicy limiting connectivity to only the services that need it (Unleash, Langfuse). This is a lateral movement risk if any pod in the namespace is compromised.

Suggested policy allowing only labeled consumers:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgresql-allow-consumers
spec:
  podSelector:
    matchLabels:
      app: postgresql
  ingress:
  - from:
    - podSelector:
        matchLabels:
          db-consumer: postgresql
    ports:
    - port: 5432

3. Production/local-dev have no database initialization path

The base deployment comment says "production uses init containers per service", but no init containers exist anywhere in this PR. The local-dev and production overlays only apply the RHEL image patch — they don't include init scripts or any other mechanism to create the unleash (or future langfuse) database. Unleash will fail to start against an empty PostgreSQL instance.

---

🟡 Major Issues

4. Identical files duplicated across kind and e2e overlays

Three files are byte-for-byte identical between the two overlays:

• postgresql-credentials.yaml
• postgresql-init-scripts.yaml
• postgresql-init-scripts-patch.yaml

This creates a maintenance burden — a change to the init script requires updating two files. Consider a shared kind-base overlay or using Kustomize components (kind: Component) to deduplicate. At minimum, the comment in both postgresql-init-scripts.yaml files says "Only used in kind overlay" but the e2e overlay also includes it — that's a stale comment.

5. Unversioned base image tag

image: postgres:16 will resolve to whatever the latest 16.x patch is at pull time. This means the deployment is non-reproducible — a new minor release could introduce behavioral changes. Pin to a specific patch: postgres:16.6 (or whatever the current stable is).

6. langfuse database intentionally omitted but Langfuse is listed as a consumer

The PR description explicitly lists Langfuse as a use case. The init script has it commented out:

# create_db_if_not_exists "langfuse"

If Langfuse is a real consumer, this should either be uncommented or the PR description should clarify it will be added later. As written, this is incomplete for the stated goal.

7. No storageClassName on the PVC

The PVC relies on the cluster's default StorageClass. Production OpenShift clusters often have multiple storage classes (e.g., standard, encrypted, replicated). Without an explicit storageClassName, you cannot guarantee what backing storage PostgreSQL gets. This is especially important for a stateful workload.

---

🔵 Minor Issues

8. Weak dev credentials

postgres123 is a recognizable weak password. While dev-only credentials are expected to be simple, using something slightly less guessable (e.g., a random suffix) reduces the risk of someone accidentally using it in a non-dev context. The # DO NOT use these credentials in production! comment helps, but a stronger password adds defense-in-depth.

9. Resource limits may be insufficient for production

memory: 512Mi is a hard limit. Unleash with active feature flag evaluation plus Langfuse tracing under moderate load can easily exceed this. Consider bumping the production limit (or documenting that it should be patched per environment). A PostgreSQL OOM kill causes service disruption for all consumers simultaneously since this is a shared instance.

10. postgresql-credentials-secret.yaml.example missing connection string examples

The comment block in the .example file says:

# Connection strings for services (constructed from above)
# Format: postgres://username:password@postgresql:5432/database
# Services should use their specific database name

...but doesn't actually provide the db.connection-string keys that services would consume. If services are expected to build connection strings themselves from db.host/db.port/etc., that's fine — but it should be documented which approach is expected.

11. PGDATA subdirectory missing from RHEL patch

The base spec sets PGDATA=/var/lib/postgresql/data/pgdata (a subdirectory of the PVC mount) to avoid PostgreSQL's requirement that the data dir be empty on initialization. The RHEL patch replaces the full env array and does not include a PGDATA equivalent. RHEL's PostgreSQL image handles this internally, but if the behavior ever diverges from upstream, data will be written directly to the mount root. Low risk, but worth a comment explaining the intentional omission.

---

Positive Highlights

• SQL injection prevention in init script — The validate_db_name() function with ^[a-zA-Z_][a-zA-Z0-9_]*$ regex is well done. Combined with ON_ERROR_STOP=1 on all psql calls, this is solid defensive scripting.
• Idempotent database creation — SELECT 1 FROM pg_database WHERE datname = '$db_name' before CREATE DATABASE prevents re-initialization errors on pod restart.
• Recreate strategy — Correct choice for a single-replica stateful database. RollingUpdate would cause split-brain.
• Kustomize syntax migration — patchesJson6902 → patches and patchesStrategicMerge → patches migrations are correct and complete.
• .gitignore update — **/postgresql-credentials-secret.yaml prevents accidental credential commits.
• Health probes — pg_isready is the right liveness/readiness check; initialDelaySeconds: 30 for liveness is appropriate.
• RHEL/upstream separation — The design cleanly isolates RHEL-specific concerns to overlays rather than the base, keeping the base portable.
• db.* key naming — Consistent secret key format that other services can reliably reference.

---

Recommendations

1. (Critical) Add securityContext to the container spec in postgresql-deployment.yaml
2. (Critical) Add a NetworkPolicy to base/ or per-overlay to restrict database access
3. (Critical) Define how production databases get initialized — either add init containers to the RHEL JSON patch, or document the manual runbook and link it from the .example file
4. (Major) Deduplicate kind/e2e resources using a Kustomize component
5. (Major) Pin postgres:16 to a specific patch version
6. (Major) Uncomment langfuse database creation or remove it from the PR description scope
7. (Major) Add storageClassName patch for production overlay
8. (Minor) Add explicit runAsNonRoot: true pod security context + fsGroup
9. (Minor) Fix stale comment: postgresql-init-scripts.yaml in e2e says "Only used in kind overlay"

---

Review generated by Claude Code (claude-sonnet-4-6) against ambient-code/platform@postgres-deployment

---

🔍 View AI decision process (logs available for 90 days)

📋 View memory system files loaded (click to expand)

What Amber Loaded for Code Review

Amber automatically loaded these repository standards from the memory system:

1. CLAUDE.md - Master project instructions, development standards
2. backend-development.md - Go backend, K8s integration patterns
3. frontend-development.md - NextJS, Shadcn UI, React Query patterns
4. security-standards.md - Auth, RBAC, token handling
5. k8s-client-usage.md - User token vs service account patterns
6. error-handling.md - Consistent error patterns
7. react-query-usage.md - Data fetching patterns

Impact: This review used your repository's specific code quality standards, security patterns, and best practices from the memory system (PRs #359, #360) - not just generic code review guidelines.