bug: ECS CreateService livelocks under concurrent awsvpc + fixed hostPort (docker_utils port-status ReadTimeoutError)

[manishahluwalia]

Summary

When creating multiple ECS services concurrently (as Terraform/OpenTofu apply does), each using networkMode: awsvpc with a fixed (non-zero) hostPort in portMappings, LocalStack Pro's ECS task-placement subsystem livelocks.

• CreateService API bookkeeping can succeed (services appear in list-services)
• Tasks never progress past pendingCount
• Underlying Docker task containers are never started (observed for 7+ minutes)
• Clients (AWS CLI / Terraform AWS provider) eventually fail with:

ServerException: Read timeout on endpoint URL: "http://localhost:4566/"

LocalStack logs show a tight loop of container remove + port-status checks timing out on the Docker Unix socket (localstack.utils.docker_utils / docker_sdk_client), with host CPU ~1% (I/O serialization, not host starvation).

Secondary symptoms after client timeout + retry (downstream of the same livelock):

• The given port (Port(port=N, protocol='tcp')) is already reserved.
• The given port (Port(port=N, protocol='tcp')) is already in use.

Environment

• LocalStack Pro localstack/localstack-pro:2026.07.0 (also observed on older latest → 4.8.2.dev35, and on 2026.4.0-arm64)
• macOS Apple Silicon (arm64), Docker Desktop / Engine 29.2.1
• /var/run/docker.sock mounted; default Docker executor
• Services: ecs,ec2,logs,iam,sts

Minimal reproduction (no application code)

1. Start LocalStack Pro:

services:
  localstack-repro:
    image: localstack/localstack-pro:2026.07.0
    container_name: localstack-repro
    ports:
      - "4599:4566"
    environment:
      - LOCALSTACK_AUTH_TOKEN=<token>
      - DEBUG=1
      - SERVICES=ecs,ec2,logs,iam,sts
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

2. Create VPC/subnet/SG (or default VPC), ECS cluster, and 13 task definitions with awsvpc + fixed hostPort 8081–8093:

{
  "family": "repro-task-8081",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512",
  "containerDefinitions": [{
    "name": "repro-container-8081",
    "image": "public.ecr.aws/nginx/nginx:alpine",
    "essential": true,
    "portMappings": [{"containerPort": 8081, "hostPort": 8081, "protocol": "tcp"}]
  }]
}

3. Create all 13 services concurrently:

for p in 8081 8082 8083 8084 8085 8086 8087 8088 8089 8090 8091 8092 8093; do
  aws --endpoint-url=http://localhost:4599 ecs create-service \
    --cluster repro-cluster --service-name repro-service-$p \
    --task-definition repro-task-$p --desired-count 1 --launch-type FARGATE \
    --network-configuration "awsvpcConfiguration={subnets=[$SUBNET_ID],securityGroups=[$SG_ID],assignPublicIp=ENABLED}" &
done
wait

Observed result

All 13 concurrent calls hang ~180s then fail uniformly with client read timeout. Services exist in list-services, but zero corresponding Docker containers start even after 7+ minutes.

A single isolated create-service (no concurrency) completes and starts its container in ~10–40s. The failure mode is concurrency-triggered.

Root cause evidence from LocalStack logs

WARN --- l.utils.docker_utils : Unexpected error when attempting to determine container port status
...
TimeoutError: timed out
urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(...): Read timed out. (read timeout=60)

This repeats across concurrent task placements sharing what appears to be an undersized/serialized Docker Unix-socket client pool.

Notes

• Component: ECS provider / localstack.utils.docker_utils / docker_sdk_client
• Hypothesis: specifically tickled by awsvpc + fixed hostPort because LocalStack must inspect containers to verify bound host ports
• Workaround: reduce Terraform parallelism for aws_ecs_service against LocalStack (-parallelism=1/2)