Private Amazon EKS Clusters and Ingress Patterns: A Network Design Deep Dive
Private Amazon EKS Clusters and Ingress Patterns: A Network Design Deep Dive
Private EKS clusters fail when teams treat the private endpoint as the whole design. This deep dive covers API access, VPC endpoints, internal ALB/NLB ingress, private DNS, and operational failure modes.
TL;DR
Private Amazon EKS clusters are not just EKS clusters with the public endpoint disabled. They are VPC-bound operating environments where API access, node bootstrap, image pulls, IAM credential exchange, load balancer reconciliation, DNS, and operator access must all work without assuming public internet reachability. The reliable pattern is to separate control-plane access from workload ingress, keep nodes in private subnets, choose public, restricted-public-plus-private, or private-only API endpoints deliberately, expose applications through internal ALB/NLB or connected networks, and pre-build VPC endpoints, private DNS, and break-glass access before cutover.
Private EKS Is a Network Architecture, Not a Checkbox
The most common private Amazon EKS design mistake is collapsing four different traffic paths into one phrase: "make the cluster private." That phrase hides the hard parts. The Kubernetes API endpoint, node bootstrap, controller access to AWS APIs, image pulls, pod identity, DNS, and application ingress do not become private in the same way, and they do not fail in the same way.
Amazon EKS gives you endpoint access controls for the Kubernetes API server. The AWS Load Balancer Controller gives you Kubernetes-native reconciliation for Application Load Balancers and Network Load Balancers. Route 53 private hosted zones and Resolver endpoints give you private names. VPC endpoints and PrivateLink give nodes and pods private paths to AWS service APIs. Those pieces are related, but they are not interchangeable.
The better mental model is this:
- Control-plane access answers: "How do operators, CI jobs, nodes, and Fargate pods reach the Kubernetes API?"
- Workload egress answers: "How do nodes, controllers, and pods call AWS APIs without public internet access?"
- Application ingress answers: "How do clients reach services running inside the cluster?"
- DNS answers: "Which resolver returns the private address, from which network, for which name?"
The original post correctly emphasized that private EKS clusters need deliberate ingress design. The deeper correction is that ingress is only one lane. A private-only EKS API can be perfectly configured while the AWS Load Balancer Controller cannot call Elastic Load Balancing, pods cannot call STS, nodes cannot pull images from ECR, and on-premises clients cannot resolve the private application hostname.
Key design metrics to track before production cutover:
- 3 endpoint modes for the Kubernetes API: public only, public plus private, and private only.
- At least 2 Availability Zones for typical ALB/NLB-backed ingress, unless you are handling a documented single-subnet edge case.
- 10 common AWS service dependencies to review in private clusters: EKS, EKS Auth, EC2, ECR API, ECR Docker, S3, STS, Elastic Load Balancing, CloudWatch Logs, and SSM.
- 4 failure domains to test independently: API access, image pull, AWS API calls from controllers and pods, and DNS resolution from every client network.
AWS documents the EKS cluster endpoint modes, private cluster requirements, and private subnet guidance. Treat those documents as separate checklists, not as one generic "private cluster" feature.
1. Decide Who Can Reach the Kubernetes API
The EKS Kubernetes API endpoint is the administrative entry point used by kubectl, controllers, CI/CD, node bootstrap paths, and managed integrations. It is not the same thing as application ingress. Disabling public endpoint access does not expose an app privately; it only changes who can speak to the control plane.
EKS supports three useful endpoint postures:
| API endpoint mode | What it means | Good fit | Main risk |
|---|---|---|---|
| Public only | The default for new clusters. Kubernetes API requests use the public endpoint, protected by IAM, Kubernetes RBAC, and optional public CIDR restrictions. | Early development, temporary migration, clusters where private connectivity is not ready. | Nodes and automation can depend on NAT or internet-reachable paths. Public CIDR mistakes are high impact. |
| Public and private | Requests from inside the VPC use the private endpoint; approved external networks can still use the public endpoint. | Transitional production posture. CI and operators move private first while emergency public access remains CIDR-restricted. | Hybrid nodes and clients can resolve differently. Public and private controls are separate. |
| Private only | Kubernetes API traffic must originate from the VPC or a connected network. | Regulated or internal-only environments with VPN, Direct Connect, Transit Gateway, SSM, or in-VPC automation already working. | You can lock yourself out if RBAC, IAM, DNS, or network access is not ready before disabling public access. |
An eksctl cluster config can make the endpoint choice explicit:
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: private-platform
region: us-east-1
version: "1.30"
vpc:
clusterEndpoints:
publicAccess: false
privateAccess: true
subnets:
private:
us-east-1a:
id: subnet-0aaa1111
us-east-1b:
id: subnet-0bbb2222
us-east-1c:
id: subnet-0ccc3333
managedNodeGroups:
- name: system-private
privateNetworking: true
instanceTypes: ["m7i.large"]
desiredCapacity: 3
The equivalent AWS CLI update for an existing cluster is blunt and should be run only after testing the private path:
aws eks update-cluster-config \
--region us-east-1 \
--name private-platform \
--resources-vpc-config endpointPublicAccess=false,endpointPrivateAccess=true
For a safer migration, start with private access enabled and public access restricted to known CIDRs:
aws eks update-cluster-config \
--region us-east-1 \
--name private-platform \
--resources-vpc-config \
endpointPrivateAccess=true,endpointPublicAccess=true,publicAccessCidrs=203.0.113.10/32
That posture lets in-VPC automation, nodes, and operators validate the private endpoint before the public endpoint disappears.
What Changes When Private Access Is Enabled
When private endpoint access is enabled, EKS creates and manages a Route 53 private hosted zone for the cluster endpoint and associates it with the cluster VPC. That hosted zone is not visible as a normal hosted zone in your Route 53 account. EKS also requires the VPC DNS attributes enableDnsHostnames and enableDnsSupport to be true and the DHCP options to include AmazonProvidedDNS.
Check the VPC before blaming Kubernetes:
aws ec2 describe-vpcs \
--vpc-ids vpc-0123456789abcdef0 \
--query 'Vpcs[0].{VpcId:VpcId,CidrBlock:CidrBlock}'
aws ec2 describe-vpc-attribute \
--vpc-id vpc-0123456789abcdef0 \
--attribute enableDnsSupport
aws ec2 describe-vpc-attribute \
--vpc-id vpc-0123456789abcdef0 \
--attribute enableDnsHostnames
If a private-only cluster cannot be reached, test DNS first from inside the VPC:
aws eks describe-cluster \
--region us-east-1 \
--name private-platform \
--query 'cluster.endpoint' \
--output text
dig +short EXAMPLE.gr7.us-east-1.eks.amazonaws.com
curl -vk https://EXAMPLE.gr7.us-east-1.eks.amazonaws.com/readyz
A DNS answer that does not resolve to private addresses, a timeout to port 443, and an RBAC denial are three different failures. Fix them in that order: name resolution, network reachability, then identity and Kubernetes authorization.
2. Keep Nodes Private, But Do Not Starve Them
Private worker nodes are the normal EKS production shape: nodes run in private subnets, application load balancers live in the appropriate public or private subnets, and outbound dependency access is explicitly designed. For private-only clusters, the stricter version is that nodes and pods have no direct outbound internet path and must use VPC endpoints, internal mirrors, or controlled egress.
The trap is assuming "private subnets" means "nothing needs outbound connectivity." Nodes still need to:
- register with the Kubernetes API
- pull images from a registry
- call EC2 APIs through the VPC CNI and controllers
- exchange credentials through STS or EKS Auth, depending on IAM model
- send logs and metrics if observability is expected
- let controllers create, tag, and update AWS resources
AWS's private cluster guidance calls out that nodes need private endpoint access to register and that pod identity and IRSA have separate AWS API requirements. For self-managed nodes without internet access, bootstrap configuration may need the cluster API endpoint and certificate authority passed explicitly so the bootstrap path does not have to call the EKS API from inside the isolated VPC.
For Amazon Linux 2023-style node configuration, the shape is:
---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
cluster:
name: private-platform
apiServerEndpoint: https://EXAMPLE.gr7.us-east-1.eks.amazonaws.com
certificateAuthority: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0t...
cidr: 10.100.0.0/16
Do not copy Outposts-only bootstrap flags into a standard regional EKS design. Flags such as --enable-local-outpost and --cluster-id belong to EKS local clusters on AWS Outposts, not to ordinary private EKS clusters running in a Region. For standard private EKS, the decision is usually managed node group versus self-managed node bootstrap, private endpoint reachability, DNS, and AWS API access.
Baseline VPC Endpoints for Private EKS
The exact endpoint set depends on what runs in the cluster, but the baseline is predictable:
REGION=us-east-1
VPC_ID=vpc-0123456789abcdef0
SUBNET_IDS="subnet-0aaa1111 subnet-0bbb2222 subnet-0ccc3333"
ENDPOINT_SG=sg-0123456789abcdef0
for service in \
ec2 \
ecr.api \
ecr.dkr \
eks \
eks-auth \
elasticloadbalancing \
logs \
ssm \
ssmmessages \
ec2messages \
sts
do
aws ec2 create-vpc-endpoint \
--region "$REGION" \
--vpc-id "$VPC_ID" \
--vpc-endpoint-type Interface \
--service-name "com.amazonaws.${REGION}.${service}" \
--subnet-ids $SUBNET_IDS \
--security-group-ids "$ENDPOINT_SG" \
--private-dns-enabled
done
aws ec2 create-vpc-endpoint \
--region "$REGION" \
--vpc-id "$VPC_ID" \
--vpc-endpoint-type Gateway \
--service-name "com.amazonaws.${REGION}.s3" \
--route-table-ids rtb-0aaa1111 rtb-0bbb2222
Security groups on interface endpoints matter. If endpoint ENIs do not allow port 443 from node and pod CIDRs, private DNS will resolve cleanly and the connection will still fail. That is one of the more misleading private-cluster failure modes because the application sees an HTTPS timeout, not a clean AWS API error.
STS Must Be Regional
IRSA and many AWS SDK credential flows call AWS STS. In private VPC designs, create the STS interface endpoint and make workloads use regional STS endpoints. Older SDK defaults or explicit sts.amazonaws.com calls can bypass the regional endpoint strategy.
For Kubernetes workloads, make the regional behavior explicit:
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
spec:
template:
spec:
serviceAccountName: payments-api
containers:
- name: app
image: 111122223333.dkr.ecr.us-east-1.amazonaws.com/payments-api:2026-06-13
env:
- name: AWS_REGION
value: us-east-1
- name: AWS_STS_REGIONAL_ENDPOINTS
value: regional
Do the same check for controllers, not only application pods. The AWS Load Balancer Controller needs Elastic Load Balancing and EC2 API access. ExternalDNS may need Route 53. Cluster Autoscaler or Karpenter needs its own service dependencies. Observability agents may need CloudWatch Logs, X-Ray, OTLP collectors, or vendor-private connectivity.
3. Design Operator Access Before You Disable Public Access
A private-only cluster is only operable if humans and automation can reach it privately. There are four common patterns.
| Pattern | How it works | Strength | Watch out for |
|---|---|---|---|
| VPN or Direct Connect | Engineers or CI reach the VPC through private routed connectivity, often through Transit Gateway. | Natural for enterprises with existing network controls. | DNS forwarding and route propagation must be tested from every source network. |
| SSM Session Manager | Operators start a session to an EC2 instance in the VPC and run kubectl there. | Avoids inbound SSH from the internet and works well for break-glass access. | Requires SSM, SSMMessages, EC2Messages, IAM, and endpoint security groups to work. |
| Bastion host | A hardened EC2 instance in a controlled subnet runs kubectl or acts as a jump point. | Simple to understand and easy to isolate. | SSH exposure, patching, key management, and audit logs must be handled deliberately. |
| In-VPC automation runner | CI/CD agents run in private subnets and deploy directly. | Best long-term production path. | You need a separate plan for emergency human access when CI is down. |
A minimal SSM-oriented break-glass host can be deployed without public SSH:
Resources:
EksOpsInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Roles:
- !Ref EksOpsRole
EksOpsHost:
Type: AWS::EC2::Instance
Properties:
ImageId: ami-0123456789abcdef0
InstanceType: t3.small
IamInstanceProfile: !Ref EksOpsInstanceProfile
SubnetId: subnet-0aaa1111
SecurityGroupIds:
- sg-ops-host
Tags:
- Key: Name
Value: private-eks-ops-host
The instance role then needs IAM permission to describe the cluster and an EKS access entry or Kubernetes RBAC mapping that authorizes the required groups. The cluster security group must allow inbound TCP 443 to the private endpoint from the ops host security group or connected network CIDR.
Validate the access path with a command sequence you can run during an incident:
aws ssm start-session --target i-0123456789abcdef0
aws eks update-kubeconfig \
--region us-east-1 \
--name private-platform
kubectl auth can-i list pods -A
kubectl get --raw='/readyz?verbose'
If that sequence is not tested before public access is removed, the private cluster is not production-ready.
4. Choose Ingress by Protocol and Client Location
Ingress for a private EKS cluster is not automatically private. It depends on load balancer scheme, subnet selection, security groups, DNS, and the network path from clients to the load balancer.
Use this decision rule:
- Choose an internal ALB for HTTP/HTTPS applications that need host routing, path routing, redirects, TLS termination, health checks, and pod-level targets.
- Choose an internal NLB for TCP, TLS pass-through, gRPC over TCP where L7 ALB features are not desired, static IP patterns, PrivateLink producer services, or protocol preservation.
- Choose no load balancer for pure in-cluster service-to-service traffic; use ClusterIP, service mesh, or Gateway API inside the cluster boundary.
- Choose public ALB/NLB only when the application is intentionally internet-facing, even if the EKS API endpoint is private.
The AWS Load Balancer Controller provisions ALBs for Kubernetes Ingress resources and NLBs for Services of type LoadBalancer. It uses subnet tags and annotations to decide where to place load balancers and how to configure target groups. AWS best-practice guidance also notes that ELB provisioning for ingress requires subnet planning across Availability Zones.
Internal ALB for HTTP Ingress
For private application ingress, the ALB scheme must be internal and the DNS name should be private. This example routes directly to pod IPs through the AWS VPC CNI:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: platform-api
namespace: platform
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/subnets: subnet-0aaa1111,subnet-0bbb2222,subnet-0ccc3333
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:111122223333:certificate/abcde-12345
alb.ingress.kubernetes.io/healthcheck-path: /healthz
alb.ingress.kubernetes.io/success-codes: "200-399"
alb.ingress.kubernetes.io/load-balancer-attributes: routing.http.drop_invalid_header_fields.enabled=true
spec:
rules:
- host: api.platform.internal.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: platform-api
port:
number: 8080
---
apiVersion: v1
kind: Service
metadata:
name: platform-api
namespace: platform
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: platform-api
ports:
- name: http
port: 8080
targetPort: 8080
target-type: ip sends traffic directly to pod IPs. That avoids NodePort routing and can reduce unnecessary hops, but it depends on pod IPs being routable in the VPC through the AWS VPC CNI. target-type: instance routes to node NodePorts and can be useful for older designs, but it adds another network layer to reason about.
Internal NLB for TCP and TLS Pass-Through
For TCP services, expose a Kubernetes Service through an internal NLB:
apiVersion: v1
kind: Service
metadata:
name: broker
namespace: messaging
annotations:
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: TCP
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: Environment=prod,Owner=platform
spec:
type: LoadBalancer
loadBalancerClass: service.k8s.aws/nlb
selector:
app.kubernetes.io/name: broker
ports:
- name: tls
port: 9093
targetPort: 9093
protocol: TCP
The current AWS Load Balancer Controller NLB docs describe internal NLBs as the secure default for recent controller versions and require an explicit annotation for internet-facing NLBs. In production, do not rely on defaults alone. Put the scheme in the manifest so review tools and humans can see the intended exposure.
Be cautious about changing load balancer type or target type on an existing Service. Controller docs warn that changing certain reconciliation annotations on an existing Service can cause misconfigured or leaked AWS resources. For material changes, create a new Service, move DNS, then delete the old Service after target health and traffic are confirmed.
5. DNS Is the Contract Between Private Networks
Private ingress is not useful if only the cluster can resolve the name. DNS needs to be designed from each client location:
- pods inside the EKS cluster
- EC2 instances in the same VPC
- other VPCs connected by Transit Gateway or peering
- on-premises networks connected through VPN or Direct Connect
- CI/CD runners
- emergency operator workstations
Route 53 private hosted zones answer DNS queries only for associated VPCs or hybrid networks that resolve through Route 53 Resolver inbound endpoints. This is different from public DNS, and it is also different from the EKS-managed private hosted zone used for the cluster API endpoint.
A practical pattern is to use a private hosted zone for internal application names:
aws route53 create-hosted-zone \
--name internal.example.com \
--vpc VPCRegion=us-east-1,VPCId=vpc-0123456789abcdef0 \
--caller-reference private-eks-internal-20260613 \
--hosted-zone-config Comment="Private names for EKS internal ingress",PrivateZone=true
Then map the internal ALB name to an application record:
{
"Comment": "Alias platform API to the internal ALB",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "api.platform.internal.example.com.",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "Z35SXDOTRQ7X7K",
"DNSName": "internal-k8s-platform-api-123456789.us-east-1.elb.amazonaws.com.",
"EvaluateTargetHealth": true
}
}
}
]
}
If you automate records with ExternalDNS, scope it tightly:
apiVersion: apps/v1
kind: Deployment
metadata:
name: external-dns
namespace: platform-dns
spec:
template:
spec:
serviceAccountName: external-dns
containers:
- name: external-dns
image: registry.k8s.io/external-dns/external-dns:v0.15.0
args:
- --source=ingress
- --domain-filter=internal.example.com
- --provider=aws
- --registry=txt
- --txt-owner-id=private-platform
- --aws-zone-type=private
For hybrid DNS, do not forward private queries to the VPC CIDR plus two resolver address from on-premises DNS servers. AWS Route 53 Resolver guidance recommends inbound Resolver endpoints for on-premises or connected networks. In practice, that means your corporate DNS forwards internal.example.com to the inbound Resolver endpoint IPs, and Route 53 answers using the private hosted zone associated with the VPC.
Private DNS failure is often mistaken for ingress failure. Test the name from every place that will use it:
dig api.platform.internal.example.com
curl -vk https://api.platform.internal.example.com/healthz
kubectl run dns-test \
--rm -it \
--image=public.ecr.aws/amazonlinux/amazonlinux:2023 \
--restart=Never \
-- bash -lc 'dnf -y install bind-utils curl && dig api.platform.internal.example.com && curl -sk https://api.platform.internal.example.com/healthz'
If DNS works in a pod but fails from the corporate network, fix Resolver forwarding. If DNS works from the corporate network but curl times out, inspect routes, security groups, NACLs, and load balancer target health. If curl reaches the load balancer and returns a 503, inspect Kubernetes endpoints and target group registration.
6. Operational Failure Modes Worth Testing
Private EKS designs usually fail at integration boundaries. A test plan should include failure injection and clean diagnostic commands, not just a successful deployment.
| Symptom | Likely cause | Fast check | Fix |
|---|---|---|---|
kubectl times out | No route to private endpoint, cluster security group missing 443, or DNS not returning private IPs. | dig <cluster-endpoint> and curl -vk <endpoint>/readyz from the same network. | Fix VPC DNS, routes, VPN/TGW propagation, or cluster security group ingress. |
| Nodes never join | Private endpoint disabled, bootstrap cannot discover endpoint, node role unauthorized, or VPC endpoints missing. | journalctl -u kubelet, node bootstrap logs, aws eks describe-cluster from the node path. | Enable private endpoint access, pass bootstrap endpoint/CA where needed, fix node IAM and endpoint access. |
| Pods cannot assume roles | STS endpoint missing, SDK calls global STS, trust policy or service account annotation wrong. | Inspect pod env, AWS SDK logs, VPC Flow Logs to STS, aws sts get-caller-identity from a debug pod. | Add STS endpoint, force regional STS, fix IRSA or Pod Identity binding. |
| Image pulls hang | ECR API, ECR Docker, or S3 endpoint path missing; endpoint security group blocks 443. | kubectl describe pod, container runtime logs, VPC endpoint metrics. | Add ecr.api, ecr.dkr, and S3 gateway endpoint; allow node and pod CIDRs to endpoint SG. |
| AWS Load Balancer Controller cannot reconcile | Missing EC2 or Elastic Load Balancing endpoint, IAM policy issue, subnet tags missing. | Controller logs and events on Ingress or Service. | Add endpoints, correct IAM, tag subnets, verify controller flags for private clusters. |
| ALB exists but targets are unhealthy | Pod readiness path mismatch, security group rule missing, wrong target type, pod not listening. | Target group health description and kubectl get endpoints. | Fix health check annotations, pod readiness, backend SG rules, or target type. |
| Internal DNS works only in one VPC | Private hosted zone not associated with all required VPCs or Resolver forwarding is incomplete. | aws route53 list-hosted-zones-by-vpc and resolver query logs. | Associate the zone, use Route 53 Profiles where appropriate, or configure inbound/outbound Resolver endpoints. |
| Private-only cluster cannot be administered during incident | No tested SSM/bastion/VPN/CI path or missing RBAC for break-glass principal. | Attempt the documented break-glass runbook quarterly. | Build and test an emergency access path before disabling public endpoint access. |
A private cluster readiness script should verify each boundary:
set -euo pipefail
CLUSTER=private-platform
REGION=us-east-1
APP_HOST=api.platform.internal.example.com
aws eks describe-cluster --region "$REGION" --name "$CLUSTER" \
--query 'cluster.resourcesVpcConfig.{private:endpointPrivateAccess,public:endpointPublicAccess,cidrs:publicAccessCidrs}'
kubectl get nodes -o wide
kubectl -n kube-system logs deploy/aws-load-balancer-controller --tail=80
kubectl get ingress -A
kubectl get service -A --field-selector spec.type=LoadBalancer
dig +short "$APP_HOST"
curl -skI "https://${APP_HOST}/healthz"
Run that from an operator path, not only from your laptop on the public internet.
Why This Design Works
The private EKS architecture that survives production incidents is intentionally boring: private endpoint access for the control plane, private nodes, explicit AWS service endpoints, internal load balancers for internal applications, Route 53 private hosted zones for application names, and a tested operator path through VPN, Direct Connect, SSM, or in-VPC automation.
The important choice is separation. Do not use the EKS API endpoint posture to reason about application exposure. Do not use a private ALB to prove that nodes can pull images. Do not use successful kubectl get pods from a bastion to prove that on-premises clients can resolve private DNS. Each path has its own DNS, route table, security group, IAM, and service dependency story.
That separation also makes trade-offs clearer:
- If public API access remains enabled, narrow it with CIDR allow lists and move automation to the private endpoint.
- If the cluster is private-only, keep a tested private operator path and an RBAC-approved break-glass role.
- If workloads have no internet egress, pre-provision the AWS service endpoints and internal image sources they need.
- If ingress is internal, pair internal load balancer scheme with private DNS and connected-network routing.
- If hybrid clients need private names, use Route 53 Resolver endpoints instead of ad hoc DNS forwarding to VPC resolver addresses.
The result is not merely "more secure." It is more diagnosable. When a private Amazon EKS cluster fails, you can locate the failure in one lane: Kubernetes API access, AWS service egress, application ingress, or DNS.
Implementation Checklist
Use this as the pre-cutover review for private Amazon EKS clusters:
- Enable EKS private endpoint access and decide whether public endpoint access is disabled or CIDR-restricted.
- Confirm VPC DNS support, DNS hostnames, and DHCP options are compatible with EKS private endpoint resolution.
- Place nodes in private subnets across at least two Availability Zones, and tag subnets for the AWS Load Balancer Controller.
- Create interface and gateway endpoints for the AWS services used by nodes, controllers, and workloads.
- Configure endpoint security groups to allow HTTPS from node and pod CIDRs or security groups.
- Force regional STS behavior for workloads that use IRSA or SDK credential calls.
- Install the AWS Load Balancer Controller with private-cluster-aware flags where needed, and verify it can call EC2 and Elastic Load Balancing APIs.
- Use
alb.ingress.kubernetes.io/scheme: internalfor private HTTP ingress andservice.beta.kubernetes.io/aws-load-balancer-scheme: internalfor private NLB services. - Create Route 53 private hosted zones for internal application names and associate them with every VPC that needs resolution.
- Use Route 53 Resolver inbound endpoints for on-premises or hybrid DNS integration.
- Test operator access through VPN, Direct Connect, SSM, bastion, Cloud9, CloudShell VPC environments, or private CI runners before disabling public endpoint access.
- Document a break-glass sequence that proves DNS, TCP 443, IAM, and Kubernetes RBAC.
Frequently Asked Questions
Q: How do I access a private Amazon EKS cluster? A: Use a network path that originates inside the cluster VPC or a connected network. Common choices are VPN, Direct Connect, Transit Gateway, SSM Session Manager to an in-VPC host, a hardened bastion host, Cloud9 in the VPC, CloudShell VPC environments where supported, or CI runners deployed in private subnets.
Q: Is private-only API endpoint access always better than restricted public endpoint access? A: Not automatically. Private-only access is stronger when your private operator and automation paths are reliable, but it is risky if CI/CD, DNS, RBAC, and emergency access are not ready. A restricted public endpoint plus private access is often the safer migration state.
Q: Should private EKS applications use an internal ALB or internal NLB? A: Use an internal ALB when HTTP routing features matter: host rules, path rules, redirects, TLS termination, health checks, and direct pod IP targets. Use an internal NLB for TCP, TLS pass-through, static IP-oriented clients, PrivateLink producer services, or protocols that should not be interpreted at layer 7.
Q: What DNS setup is required for private EKS ingress? A: Use Route 53 private hosted zones for application records and associate the zones with the VPCs that need them. For on-premises or hybrid clients, forward private zones to Route 53 Resolver inbound endpoint IPs instead of forwarding directly to the VPC resolver address.
Q: Why can nodes join a private EKS cluster while workloads still fail? A: Node registration proves only that nodes can reach the Kubernetes API and are authorized to join. Workloads may still fail because ECR, S3, STS, EKS Auth, Elastic Load Balancing, CloudWatch Logs, Route 53, or third-party dependency paths are missing or blocked by endpoint security groups.
Related Internal Guides
- Building an Internal Developer Platform on EKS
- Autoscaling EKS Clusters with Karpenter
- Kubernetes Gateway API vs Ingress
Resources
- Original Blogger post
- Amazon EKS cluster API server endpoint
- Amazon EKS private clusters with limited internet access
- Amazon EKS interface VPC endpoints
- Amazon EKS VPC and subnet best practices
- AWS Load Balancer Controller Ingress annotations
- AWS Load Balancer Controller Service annotations
- AWS Load Balancer Controller NLB guide
- Route 53 private hosted zones
- Route 53 Resolver for hybrid DNS
Comments
Post a Comment