The ticket
Finance: NAT data processing is huge. Flow logs: app subnets talking to S3 public prefixes via NAT. There is no gateway endpoint on the app route table. Private instances were never supposed to need a NAT just to reach S3.
Two VPC endpoint families. Gateway endpoint: Amazon S3 and DynamoDB only. You add a prefix-list route to the VPC route table; traffic to those services stays on the AWS network. Official docs: no additional charge for using gateway endpoints. Interface endpoint (AWS PrivateLink): an ENI in your subnet, security group, optional private DNS, used for most other AWS APIs (KMS, Secrets Manager, CloudTrail, EC2 API, …). You pay for interface endpoints (hours + data processing — do not invent a price). NAT to the public S3 endpoint from a private subnet is the expensive/wrong default when a gateway endpoint exists.
Account 111122223333 · OUs Security / Workloads · hub VPC 10.50.0.0/16 · app VPC 10.51.0.0/16 · SG sg-web · NACL acl-app · bucket tc-lab-app-logs · role AppEC2Role · TGW tgw-lab · VGW public 203.0.113.60 · on-prem FortiGate WAN 203.0.113.10. Dummy account ID. Not yours.
Mental model
| Gateway endpoint | Interface endpoint | |
|---|---|---|
| Services | S3, DynamoDB | PrivateLink — most other AWS services |
| How it attaches | Prefix list in the route table | ENI in a subnet + SG |
| This lab | App RT: pl-s3 → vpce-s3 for tc-lab-app-logs | KMS / Secrets Manager / CloudTrail ENIs in 10.51.0.0/16 |
| Wrong default | 0.0.0.0/0 → NAT just so S3 works | Leaving private DNS off and still calling the public API |
Top path is the design. Bottom path is the bill and the extra public exposure.
vpce-s3 (Gateway) + interface endpoints
How to choose
| Destination | Use | Do not |
|---|---|---|
| S3 / DynamoDB from a private subnet | Gateway endpoint + RT prefix list | NAT “because it needs internet” |
| KMS, Secrets Manager, STS, EC2 API | Interface endpoint + SG | Open the app SG to 0.0.0.0/0 so the public API works |
| True internet (patches from the public web) | NAT or a pull-through that you meant | Pretend S3 is that problem |
Runbook
Side A — gateway
Create the S3 gateway endpoint in app VPC
10.51.0.0/16. Associate the app private route tables. Confirm a prefix-list route, not a default route via NAT.Side B — interface
For KMS/Secrets: interface endpoints in the private subnets, SG allowing 443 from
sg-app, private DNS enabled.Side C — prove
From the instance (no public IP): write an object to
tc-lab-app-logs.aws ec2 describe-route-tablesshowsvpce-. NAT bytes to S3 should fall.
aws ec2 describe-vpc-endpoints --filters Name=vpc-id,Values=vpc-app \
--query 'VpcEndpoints[].{Id:VpcEndpointId,Type:VpcEndpointType,Service:ServiceName,State:State}'
# vpce-s3 Gateway com.amazonaws.ap-south-1.s3
# vpce-kms Interface com.amazonaws.ap-south-1.kms
aws ec2 describe-route-tables --filters Name=vpc-id,Values=vpc-app \
--query 'RouteTables[].Routes'
# DestinationPrefixListId: pl-xxxx GatewayId: vpce-s3
aws s3 cp /etc/os-release s3://tc-lab-app-logs/probe/os-release
# from the private instance — no public IP requiredFour failures
1 · NAT for S3 by habit
Works, costs, and teaches the wrong default. Add the gateway endpoint; then drop S3 from the NAT path.
2 · Gateway endpoint, wrong route table
You attached it to the public RT. The private app RT still points at NAT or has no S3 route.
3 · Interface endpoint, private DNS off
SDK still calls the public hostname and fails in a private subnet.
4 · Endpoint policy deny + IAM allow
Both must allow. The endpoint policy is another ceiling, like an SCP for that path.
How to prove it
1) describe-vpc-endpoints shows Gateway for S3. 2) The app RT has the prefix-list route. 3) A private instance wrote to tc-lab-app-logs. 4) You did not open 0.0.0.0/0 to “make S3 work.”
Traps
| Belief | Fact |
|---|---|
| S3 needs a NAT | S3 has a gateway endpoint |
| Gateway endpoints exist for KMS | KMS is interface / PrivateLink |
| Endpoint policy grants IAM | It only filters, like an SCP |
Knowledge check
Judgment items. One best answer. Reasons send you back to the matching section.
AWS security class series: Shared + IAM · Org / OU · SG vs NACL · Role not key · KMS + Secrets · VPC endpoint · Trail + GuardDuty · TGW hub · S3 public · Interview
Sources
Related: AWS session factory · Migration series.