Skip to main content
    • Knowledge Base/
    Aws Privatelink Setup For Clickpipes

    AWS PrivateLink setup to expose private RDS for ClickPipes

    Setup steps to expose a private RDS via AWS PrivateLink to ClickPipes.

    Setup steps to expose a private RDS via AWS PrivateLink to ClickPipes.

    Requirements​

    The VPC must be located in one of our ClickPipes regions: us-east-1, us-east-2 or eu-central-1, additionally the ClickHouse Instance must be in the same region.

    Follow these steps to create a VPC endpoint service for your RDS instance. Repeat these steps if you have multiple RDS instances that require endpoint services (OR you may have different listener ports for different instances):

    1. Locate Your VPC and Create an NLB

      • Navigate to your target VPC and create a Network Load Balancer (NLB). Note that the NLB should be internal (private) and not internet-facing (public).

    2. Configure the Target Group

      • The target group should point to the RDS instance's endpoint IP and Port (typically 5432 for PostgreSQL or 3306 for MySQL).
      Note

      If you would like to automate the process of updating the target group with the new RDS endpoint IP, you can use AWS Lambda functions or other automation tools. One of the terraform modules that can be used for this purpose is this.

      • IMPORTANT: Make sure the RDS instance endpoint used in case of DB Cluster/Aurora is ONLY the WRITER Endpoint and NOT the common endpoint.
      • Ensure that the TCP protocol is used to avoid TLS termination by the NLB.

    3. Set the Listener Port

      • The listener port of the load balancer must match the port used by the target group (typically 5432 for PostgreSQL or 3306 for MySQL).

    4. Create the VPC Endpoint Service

      • In the VPC, create an endpoint service that points to the NLB.
      • Enable acceptance of connection requests from specific accounts.

    5. Authorize ClickPipes to Use the Endpoint Service

      • Grant permission to the ClickPipes account to request this endpoint service.
      • Configure allowed principals by adding the following principal ID: 
        arn:aws:iam::072088201116:root

    6. Disable "Enforce Security Group Inbound Rules on Private Link Traffic" on the NLB (if a security group is attached to the NLB)

      • Navigate to the NLB's settings and disable the "Enforce Security Group Inbound Rules on Private Link Traffic" setting if a security group is attached to the NLB.
      • If using Terraform, set the enforce_security_group_inbound_rules_on_private_link_traffic attribute to false for the NLB
      • This setting is required to allow traffic from the ClickPipes VPC to the NLB.

    Initiating connection​

    When it's done, share details such as AWS region, VPC service name and availability zone. ClickPipes team will initiate VPC endpoints creation in ClickPipes VPC. This will require connection request acceptance on your side.

    Creating ClickPipes​

    ClickPipes team will provide the DNS name to be used in the ClickPipe creation process. You can now create your ClickPipe.

    Dynamically updating the RDS endpoint IP​

    When the RDS endpoint IP changes (in case of restarts/failovers/updates), you need to update the NLB target group with the new IP. You can automate this process using AWS Lambda functions or other automation tools.

    · 3 min read