Skip to content

DOC-1871: Add client VPC and security group setup to dedicated Privat… #504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
micheleRP merged 4 commits into from
Mar 2, 2026
Merged

DOC-1871: Add client VPC and security group setup to dedicated Privat… #504

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f733bb1
DOC-1871: Add client VPC and security group setup to dedicated Privat…
micheleRP Feb 17, 2026
9da9c85
DOC-1871: Fix AZ ID example to use AZ name format for --availability-…
micheleRP Feb 26, 2026
1adf9a3
DOC-1871: Apply review fixes across PrivateLink docs
micheleRP Feb 27, 2026
62a3d03
incorporate review feedback
micheleRP Mar 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 12 additions & 125 deletions modules/networking/pages/aws-privatelink.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
:page-aliases: deploy:deployment-option/cloud/aws-privatelink.adoc


NOTE: This guide is for configuring AWS PrivateLink using the Redpanda Cloud API. To configure and manage PrivateLink on an existing public cluster, you must use the Cloud API. See xref:networking:configure-privatelink-in-cloud-ui.adoc[Configure PrivateLink in the Cloud UI] if you want to set up the endpoint service using the Redpanda Cloud UI.
NOTE: This guide is for configuring AWS PrivateLink using the Redpanda Cloud API. To configure and manage PrivateLink on an existing public cluster, you must use the Cloud API. See xref:networking:configure-privatelink-in-cloud-ui.adoc[Configure PrivateLink in the Cloud UI] if you want to set up the endpoint service using the Redpanda Cloud Console.

The Redpanda AWS PrivateLink endpoint service provides secure access to Redpanda Cloud from your own VPC. Traffic over PrivateLink does not go through the public internet because a PrivateLink connection is treated as its own private AWS service. While your VPC has access to the Redpanda VPC, Redpanda cannot access your VPC.

Expand All @@ -22,11 +22,11 @@ After <<get-a-cloud-api-access-token,getting an access token>>, you can <<create
== Prerequisites

* Install `rpk`.
* Your Redpanda cluster and <<create-client-vpc,VPC>> must be in the same region, unless you configure <<cross-region-privatelink,cross-region PrivateLink>>.
* Your Redpanda cluster and <<set-up-the-client-vpc,VPC>> must be in the same region, unless you configure <<cross-region-privatelink,cross-region PrivateLink>>.
* In this guide, you use the link:/api/doc/cloud-controlplane/topic/topic-cloud-api-overview[Redpanda Cloud API] to enable the Redpanda endpoint service for your clusters. Follow the steps below to <<get-an-access-token,get an access token>>.
* Use the https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html[AWS CLI^] to create a new client VPC or modify an existing one to use the PrivateLink endpoint.

TIP: In Kafka clients, set `connections.max.idle.ms` to a value less than 350 seconds.
TIP: In Kafka clients, set `connections.max.idle.ms` to a value less than 350 seconds (350000 ms).

NOTE: Enabling PrivateLink changes private DNS behavior for your cluster. Before configuring connections, review <<dns-resolution-with-privatelink>>.

Expand All @@ -36,7 +36,7 @@ include::networking:partial$private-links-api-access-token.adoc[]

== Create new cluster with PrivateLink endpoint service enabled

. In the https://cloud.redpanda.com/[Redpanda Cloud UI^], go to **Resource groups** and select the resource group in which you want to create a cluster.
. In the https://cloud.redpanda.com/[Redpanda Cloud Console^], go to **Resource groups** and select the resource group in which you want to create a cluster.
+
Copy and store the resource group ID (UUID) from the URL in the browser.
+
Expand Down Expand Up @@ -87,7 +87,7 @@ Wait for the network to be ready before creating the cluster in the next step. Y
In the example below, make sure to set your own values for the following fields:
+
--
- `zones`: for example, `"usw2-az1","usw2-az2","usw2-az3"`
- `zones`: for example, `"us-west-2a","us-west-2b","us-west-2c"`
- `type`: `"TYPE_BYOC"` or `"TYPE_DEDICATED"`
- `tier`: for example, `"tier-1-aws-v2-arm"`
- `name`
Expand Down Expand Up @@ -146,7 +146,7 @@ Enabling PrivateLink on your VPC interrupts all communication on existing Redpan
To avoid disruption, consider using a staged approach to enable PrivateLink. See: xref:networking:byoc/aws/vpc-peering-aws.adoc#switch-from-vpc-peering-to-privatelink[Switch from VPC peering to PrivateLink].
====

. In the Redpanda Cloud UI, go to the cluster overview and copy the cluster ID from the **Details** section.
. In the Redpanda Cloud Console, go to the cluster overview and copy the cluster ID from the **Details** section.
+
[,bash]
----
Expand Down Expand Up @@ -204,7 +204,7 @@ When you have a PrivateLink-enabled cluster, you can create an endpoint to conne

=== Get cluster domain

Get the domain (`cluster_domain`) of the cluster from the cluster details in the Redpanda Cloud UI.
Get the domain (`cluster_domain`) of the cluster from the cluster details in the Redpanda Cloud Console.

For example, if the bootstrap server URL is: `seed-3da65a4a.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com:9092`, then `cluster_domain` is: `cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com`.

Expand All @@ -227,126 +227,13 @@ PL_SERVICE_NAME=`curl -X GET \
$PUBLIC_API_ENDPOINT/v1/clusters/$CLUSTER_ID | jq -r .cluster.aws_private_link.status.service_name`
----

=== Create client VPC
With the service name stored, set up your client VPC to connect to the endpoint service.

If you are not using an existing VPC, you must create a new one.

[CAUTION]
====
xref:networking:byoc/aws/vpc-peering-aws.adoc[VPC peering] and PrivateLink will not work at the same time if you set them up on the same VPC where your Kafka clients run. PrivateLink endpoints take priority.

VPC peering and PrivateLink can both be used at the same time if Kafka clients are connecting from distinct VPCs. For example, in a private Redpanda cluster, you can connect your internal Kafka clients over VPC peering, and enable PrivateLink for external services.
====

The client VPC must be in the same region as your Redpanda cluster, unless you have configured <<cross-region-privatelink,cross-region PrivateLink>>. To create the VPC, run:

[,bash]
----
# See https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html for
# information on profiles and credential files
PROFILE=<specific-profile-from-credential-file>

aws ec2 create-vpc --region $REGION --profile $PROFILE --cidr-block 10.0.0.0/20

# Store the client VPC ID from the command output
CLIENT_VPC_ID=<client_vpc_id>
----

You can also use an existing VPC. You need the VPC ID to <<modify-vpc-dns-attributes,modify its DNS attributes>>.

=== Modify VPC DNS attributes

To modify the VPC attributes, run:

[,bash]
----
aws ec2 modify-vpc-attribute --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
--enable-dns-hostnames "{\"Value\":true}"

aws ec2 modify-vpc-attribute --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
--enable-dns-support "{\"Value\":true}"
----

These commands enable DNS hostnames and resolution for instances in the VPC.

=== Create security group

You need the security group ID `security_group_id` from the command output to <<add-security-group-rules,add security group rules>>. To create a security group, run:

[,bash]
----
aws ec2 create-security-group --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
--description "Redpanda endpoint service client security group" \
--group-name "${CLUSTER_ID}-sg"
SECURITY_GROUP_ID=<security_group_id>
----

=== Add security group rules

The following example adds security group rules that work for any broker count by opening the documented per-broker port ranges.

NOTE: For PrivateLink, clients connect to individual ports for each broker in ranges 32000-32500 (Kafka API) and 35000-35500 (HTTP Proxy). Opening only a few ports by broker count can break producers/consumers for topics with many partitions. See xref:networking:cloud-security-network.adoc#private-service-connectivity-network-ports[Private service connectivity network ports].

[,bash]
----
# Allow Kafka API bootstrap (seed)
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
--group-id $SECURITY_GROUP_ID --protocol tcp --port 30292 --cidr 0.0.0.0/0

# Allow Schema Registry
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
--group-id $SECURITY_GROUP_ID --protocol tcp --port 30081 --cidr 0.0.0.0/0

# Allow HTTP Proxy bootstrap
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
--group-id $SECURITY_GROUP_ID --protocol tcp --port 30282 --cidr 0.0.0.0/0

# Allow Redpanda Cloud Data Plane API / Prometheus (if needed)
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
--group-id $SECURITY_GROUP_ID --protocol tcp --port 443 --cidr 0.0.0.0/0

# Private service connectivity broker port pools
# Kafka API per-broker ports
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
--group-id $SECURITY_GROUP_ID \
--ip-permissions 'IpProtocol=tcp,FromPort=32000,ToPort=32500,IpRanges=[{CidrIp=0.0.0.0/0}]'

# HTTP Proxy per-broker ports
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
--group-id $SECURITY_GROUP_ID \
--ip-permissions 'IpProtocol=tcp,FromPort=35000,ToPort=35500,IpRanges=[{CidrIp=0.0.0.0/0}]'
----

=== Create VPC subnet

You need the subnet ID `subnet_id` from the command output to <<create-vpc-endpoint,create a VPC endpoint>>. Run the following command, specifying the subnet availability zone (for example, `usw2-az1`):

[,bash]
----
aws ec2 create-subnet --region $REGION --profile $PROFILE --vpc-id $CLIENT_VPC_ID \
--availability-zone <zone> \
--cidr-block 10.0.1.0/24
SUBNET_ID=<subnet_id>
----

=== Create VPC endpoint

[,bash]
----
aws ec2 create-vpc-endpoint \
--region $REGION --profile $PROFILE \
--vpc-id $CLIENT_VPC_ID \
--vpc-endpoint-type "Interface" \
--ip-address-type "ipv4" \
--service-name $PL_SERVICE_NAME \
--subnet-ids $SUBNET_ID \
--security-group-ids $SECURITY_GROUP_ID \
--private-dns-enabled
----
include::networking:partial$private-links-aws-client-vpc-setup.adoc[]

== Access Redpanda services through VPC endpoint

After you have enabled PrivateLink for your cluster, your connection URLs are available in the *How to Connect* section of the cluster overview in the Redpanda Cloud UI.
After you have enabled PrivateLink for your cluster, your connection URLs are available in the *How to Connect* section of the cluster overview in the Redpanda Cloud Console.

include::networking:partial$private-links-access-rp-services-through-vpc.adoc[]

Expand All @@ -358,7 +245,7 @@ For more information about AWS cross-region PrivateLink support, see the https:/

=== Requirements

* The Redpanda cluster must be deployed across multiple availability zones (multi-AZ). This is an AWS limitation for cross-region PrivateLink.
* The Redpanda cluster must be deployed across multiple glossterm:availability zones[] (multi-AZ). This is an AWS limitation for cross-region PrivateLink.
* Cross-region PrivateLink is configured through the `supported_regions` field in the `aws_private_link` configuration. This field only appears in the API response for multi-AZ clusters.
* For BYOC clusters, the Redpanda agent IAM role must have `vpce:AllowMultiRegion` and `elasticloadbalancing:DescribeListenerAttributes` permissions.

Expand Down Expand Up @@ -407,7 +294,7 @@ aws ec2 create-vpc-endpoint \

== Test the connection

You can test the PrivateLink connection from any VM or container in the consumer VPC. If configuring a client isn't possible right away, you can do these checks using `rpk` or cURL:
You can test the PrivateLink connection from any VM or container in the client VPC. If configuring a client isn't possible right away, you can do these checks using `rpk` or cURL:

include::networking:partial$private-links-test-connection.adoc[]

Expand Down
38 changes: 34 additions & 4 deletions modules/networking/pages/configure-privatelink-in-cloud-ui.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Consider using the endpoint service if you have multiple VPCs and could benefit
* Your Redpanda cluster and VPC must be in the same region, unless you configure <<cross-region-privatelink,cross-region PrivateLink>>.
* Use the https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html[AWS CLI] to create a new client VPC or modify an existing one to use the PrivateLink endpoint.

TIP: In Kafka clients, set `connections.max.idle.ms` to a value less than 350 seconds.
TIP: In Kafka clients, set `connections.max.idle.ms` to a value less than 350 seconds (350000 ms).

== DNS resolution with PrivateLink

Expand All @@ -38,6 +38,36 @@ include::networking:partial$dns_resolution.adoc[]

NOTE: For help with issues when enabling PrivateLink, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda support^].

== Configure PrivateLink connection to Redpanda Cloud

When you have a PrivateLink-enabled cluster, create a VPC endpoint to connect your client VPC to your cluster.

=== Get cluster domain

Get the domain (`cluster_domain`) of the cluster from the bootstrap server URL in the *How to Connect* section of the cluster overview in the Redpanda Cloud Console.

For example, if the bootstrap server URL is: `seed-3da65a4a.cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com:9092`, then `cluster_domain` is: `cki01qgth38kk81ard3g.fmc.dev.cloud.redpanda.com`.

[,bash]
----
CLUSTER_DOMAIN=<cluster_domain>
----

NOTE: Use `<cluster_domain>` as the domain you target with your DNS conditional forward (optionally also `*.<cluster_domain>` if your DNS platform requires a wildcard).

=== Get name of PrivateLink endpoint service

You need the service name to <<create-vpc-endpoint,create a VPC endpoint>>. You can find the service name on the *Cluster settings* page after PrivateLink is enabled, or in the *How to Connect* section of the cluster overview.

[,bash]
----
PL_SERVICE_NAME=<vpc_endpoint_service_name>
----

With the service name stored, set up your client VPC to connect to the endpoint service.

include::networking:partial$private-links-aws-client-vpc-setup.adoc[]

== Access Redpanda services through VPC endpoint

After you have enabled PrivateLink for your cluster, your connection URLs are available in the *How to Connect* section of the cluster overview in the Redpanda Cloud Console.
Expand All @@ -46,7 +76,7 @@ include::networking:partial$private-links-access-rp-services-through-vpc.adoc[]

== Test the connection

You can test the connection to the endpoint service from any VM or container in the consumer VPC. If configuring a client isn't possible right away, you can do these checks using `rpk` or cURL:
You can test the connection to the endpoint service from any VM or container in the client VPC. If configuring a client isn't possible right away, you can do these checks using `rpk` or cURL:

include::networking:partial$private-links-test-connection.adoc[]

Expand All @@ -58,13 +88,13 @@ For more information about AWS cross-region PrivateLink support, see the https:/

=== Prerequisites

* The Redpanda cluster must be deployed across multiple availability zones (multi-AZ). This is an AWS limitation for cross-region PrivateLink.
* The Redpanda cluster must be deployed across multiple glossterm:availability zones[] (multi-AZ). This is an AWS limitation for cross-region PrivateLink.

=== Configure supported regions

NOTE: The *Supported regions* option only appears in the UI for multi-AZ clusters.

. In the Redpanda Cloud UI, select your https://cloud.redpanda.com/clusters[cluster^], and go to the cluster settings page.
. In the Redpanda Cloud Console, select your https://cloud.redpanda.com/clusters[cluster^], and go to the cluster settings page.
. In the AWS PrivateLink section, click *Edit* (or *Enable* if PrivateLink is not yet enabled).
. In the *Supported regions* section, click *Add* to add a region from which PrivateLink endpoints can connect to your cluster.
. Select an AWS region from the dropdown. The cluster's home region is automatically included and not shown in the list.
Expand Down
10 changes: 5 additions & 5 deletions modules/networking/pages/serverless/aws/privatelink-api.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,15 +288,15 @@ The following example shows how to add security group rules to allow access to R
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
--group-id $SECURITY_GROUP_ID --protocol tcp --port 9092 --cidr 0.0.0.0/0

# Allow Kafka API bootstrap (broker)
# Allow Kafka API broker 1
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
--group-id $SECURITY_GROUP_ID --protocol tcp --port 9093 --cidr 0.0.0.0/0

# Allow Kafka API bootstrap (broker)
# Allow Kafka API broker 2
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
--group-id $SECURITY_GROUP_ID --protocol tcp --port 9094 --cidr 0.0.0.0/0

# Allow Kafka API bootstrap (broker)
# Allow Kafka API broker 3
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
--group-id $SECURITY_GROUP_ID --protocol tcp --port 9095 --cidr 0.0.0.0/0

Expand All @@ -311,7 +311,7 @@ aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \

=== Create VPC subnet

You need the subnet ID `subnet_id` from the command output to <<create-vpc-endpoint,create a VPC endpoint>>. Run the following command, specifying the subnet availability zone (for example, `usw2-az1`):
You need the subnet ID `subnet_id` from the command output to <<create-vpc-endpoint,create a VPC endpoint>>. Run the following command, specifying the subnet Availability Zone name (for example, `us-west-2a`):
Copy link
Copy Markdown
Contributor

@paulohtb6 paulohtb6 Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

glossterm for az?

Copy link
Copy Markdown
Contributor Author

@micheleRP micheleRP Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added glossterm


[,bash]
----
Expand Down Expand Up @@ -344,7 +344,7 @@ include::networking:partial$private-links-access-rp-service-serverless.adoc[]

== Test the connection

You can test the PrivateLink connection from any VM or container in the consumer VPC. If configuring a client isn't possible right away, you can do these checks using `rpk` or cURL:
You can test the PrivateLink connection from any VM or container in the client VPC. If configuring a client isn't possible right away, you can do these checks using `rpk` or cURL:

include::networking:partial$private-links-test-connection-serverless.adoc[]

Expand Down
14 changes: 7 additions & 7 deletions modules/networking/pages/serverless/aws/privatelink-ui.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ Consider using the PrivateLink endpoint service if you have multiple VPCs and co
== Requirements

* Your Redpanda Serverless cluster and VPC must be in the same region.
* Use the https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html[AWS CLI] to create a new client VPC or modify an existing one to use the PrivateLink endpoint.
* Use the https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html[AWS CLI^] to create a new client VPC or modify an existing one to use the PrivateLink endpoint.

TIP: In Kafka clients, set `connections.max.idle.ms` to a value less than 350 seconds.
TIP: In Kafka clients, set `connections.max.idle.ms` to a value less than 350 seconds (350000 ms).

== DNS resolution with PrivateLink

Expand Down Expand Up @@ -134,15 +134,15 @@ The following example shows how to add security group rules to allow access to R
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
--group-id $SECURITY_GROUP_ID --protocol tcp --port 9092 --cidr 0.0.0.0/0

# Allow Kafka API bootstrap (broker)
# Allow Kafka API broker 1
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
--group-id $SECURITY_GROUP_ID --protocol tcp --port 9093 --cidr 0.0.0.0/0

# Allow Kafka API bootstrap (broker)
# Allow Kafka API broker 2
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
--group-id $SECURITY_GROUP_ID --protocol tcp --port 9094 --cidr 0.0.0.0/0

# Allow Kafka API bootstrap (broker)
# Allow Kafka API broker 3
aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \
--group-id $SECURITY_GROUP_ID --protocol tcp --port 9095 --cidr 0.0.0.0/0

Expand All @@ -157,7 +157,7 @@ aws ec2 authorize-security-group-ingress --region $REGION --profile $PROFILE \

=== Create VPC subnet

You need the subnet ID `subnet_id` from the command output to <<create-vpc-endpoint,create a VPC endpoint>>. Run the following command, specifying the subnet availability zone (for example, `usw2-az1`):
You need the subnet ID `subnet_id` from the command output to <<create-vpc-endpoint,create a VPC endpoint>>. Run the following command, specifying the subnet Availability Zone name (for example, `us-west-2a`):

[,bash]
----
Expand Down Expand Up @@ -192,7 +192,7 @@ include::networking:partial$private-links-access-rp-service-serverless.adoc[]

== Test the connection

You can test the connection to the endpoint service from any VM or container in the consumer VPC. If configuring a client isn't possible right away, you can do these checks using `rpk` or cURL:
You can test the connection to the endpoint service from any VM or container in the client VPC. If configuring a client isn't possible right away, you can do these checks using `rpk` or cURL:

include::networking:partial$private-links-test-connection-serverless.adoc[]

Expand Down
Loading