The Hidden Manual Work Behind Kubernetes Pod Restarts

Ever deployed what looks like a simple Kubernetes setup, only to discover that "manual steps" are required every time a pod restarts? I encountered this exact scenario in production, and what I found reveals a fundamental architectural flaw that's probably more common than anyone wants to admit.

The Setup

We hosted multiple client websites through a single nginx reverse proxy in Kubernetes. Everything was containerized, configs lived in ConfigMaps, and logs flowed properly through our pipeline.

The Architecture:

• Multiple client websites (client1.example.com, client2.example.com, etc.)
• Client DNS records pointing to an AWS Elastic IP
• Single Kubernetes pod with three containers:
  • Nginx: Handling reverse proxy duties for all clients
  • Filebeat: Collecting logs and acting as a message queue
  • Logstash: Filtering and processing logs
• Persistent Volume for nginx logs
• ConfigMap managing all client configurations

Here's the traffic flow:

┌─────────────────────────────────────────────────────────────────┐
│                        Internet                                 │
│                                                                 │
│  client1.example.com ──┐                                        │
│  client2.example.com ──┼─→ DNS Points to AWS Elastic IP         │  
│  client3.example.com ──┘     (Static Public IP: 203.0.113.10)   │
└─────────────────────────────────┬───────────────────────────────┘
                                  │
                ┌─────────────────▼────────────────┐
                │         AWS EC2 Instance         │
                │    (Elastic IP: 203.0.113.10)    │
                │                                  │
                │  ┌─────────────────────────────┐ │
                │  │     Kubernetes Cluster      │ │
                │  │                             │ │
                │  │  ┌─────────────────────────┐│ │
                │  │  │      Single Pod         ││ │
                │  │  │                         ││ │
                │  │  │ ┌─────────┐ ┌────────┐  ││ │
                │  │  │ │  nginx  │ │filebeat│  ││ │
                │  │  │ │(reverse │ │  (MQ)  │  ││ │
                │  │  │ │ proxy)  │ │        │  ││ │
                │  │  │ └─────────┘ └────────┘  ││ │
                │  │  │      │      ┌────────┐  ││ │
                │  │  │      │      │logstash│  ││ │
                │  │  │      │      │(filter)│  ││ │
                │  │  │      ▼      └────────┘  ││ │
                │  │  │ [PVC Logs]              ││ │
                │  │  │      │                  ││ │
                │  │  │      ▼                  ││ │
                │  │  │ [ConfigMap: nginx.conf] ││ │
                │  │  └─────────────────────────┘│ │
                │  └─────────────────────────────┘ │
                └──────────────────────────────────┘

Traffic hits the Elastic IP, nginx routes based on host headers to different backend APIs, and logs flow through the pipeline.

The Problem

Need to add a new client or update routing rules? The process seemed straightforward:

1. Update the ConfigMap with new nginx configuration
2. Restart the pod to pick up changes
3. Done.

Except every single pod restart triggered manual intervention that operations teams had to perform.

The Hidden Manual Process

Here's what actually happens when you run kubectl rollout restart:

Phase 1: Pod Termination

Your pod gets terminated and Kubernetes schedules a new one. The problem starts here:

• If the new pod lands on the same EC2 node: 5-10 minutes of downtime
• If it lands on a different node: Extended downtime requiring manual intervention

Phase 2: Manual Recovery (When Pod Moves Nodes)

Step 1: IP Migration

# Find where the pod ended up
kubectl get pod nginx-pod-xyz -o wide
# The pod is now on node-2, but Elastic IP is still on node-1

# Manual intervention required
aws ec2 disassociate-address --association-id eipassoc-old-one
aws ec2 associate-address --instance-id i-new-node --allocation-id eipalloc-12345678

Step 2: Volume Reattachment

# Volume stuck in "attaching" state
aws ec2 detach-volume --volume-id vol-0123456789abcdef0 --force
# Wait...
aws ec2 attach-volume --volume-id vol-0123456789abcdef0 --instance-id i-new-node --device /dev/xvdf

Step 3: Recovery Validation

# Pod might be stuck waiting for volumes
kubectl delete pod nginx-pod-xyz --force
# Wait for everything to come back up...

The Real Cost:

• 15-30 minutes of complete downtime for all client sites
• Manual intervention required for every configuration change
• Client complaints during business hours
• Emergency calls when automated deployments fail

The Root Cause

The issue is architectural: we've tightly coupled network identity with compute identity.

• Network identity: The Elastic IP that clients use to reach us
• Compute identity: The specific EC2 node where our pod runs

When the pod moves, the network identity must manually follow. This violates a fundamental cloud-native principle: infrastructure should be independent of application lifecycle.

Why Not Application Load Balancer?

You might ask, "Why not use ALB?" Fair question, but ALB creates different problems:

ALB Issues:

• Dynamic IPs: ALB addresses change frequently, breaking client DNS setups
• Client IP complexity: Requires X-Forwarded-For header handling
• Whitelisting problems: Partners can't whitelist IPs that keep changing
• Layer 7 overhead: Unnecessary HTTP processing for simple reverse proxy

NLB Advantages:

• Static IP assignment: Can attach existing Elastic IPs directly
• Transparent client IP: Original client IPs pass through automatically
• Layer 4 performance: Lower latency for TCP traffic
• Purpose-built: Designed exactly for this use case

The Solution: Decouple Network from Compute

The goal is simple: make the Elastic IP the permanent home of our network identity, independent of where our pod runs.

Step 1: Create Network Load Balancer Service

apiVersion: v1
kind: Service
meta
  name: nginx-static-ip-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    # Your existing Elastic IP allocation IDs
    service.beta.kubernetes.io/aws-load-balancer-eip-allocations: "eipalloc-12345678,eipalloc-87654321"
    # Client IP preservation
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true
    # Health check configuration
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/health"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "10"
spec:
  type: LoadBalancer
  selector:
    app: nginx-reverse-proxy
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 443

Step 2: Add Health Checks

# Add to your ConfigMap
server {
    listen 80;
    server_name health.check;
    
    location /health {
        access_log off;
        return 200 "healthy\n";
        add_header Content-Type text/plain;
    }
}

Step 3: Enhanced Pod Configuration

apiVersion: apps/v1
kind: Deployment
meta
  name: nginx-reverse-proxy
spec:
  replicas: 1
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: nginx
        image: nginx:latest
        readinessProbe:
          httpGet:
            path: /health
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health
            port: 80
          initialDelaySeconds: 15
          periodSeconds: 10
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]
        # Your existing containers remain the same

The New Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Internet                                 │
│  client1.example.com ──┐                                        │
│  client2.example.com ──┼─→ DNS Points to SAME Elastic IPs       │  
│  client3.example.com ──┘     (Still: 203.0.113.10)              │
└─────────────────────────────────┬───────────────────────────────┘
                                  │
                ┌─────────────────▼────────────────┐
                │      Network Load Balancer       │
                │   (Elastic IPs: 203.0.113.10)    │ ← PERMANENT
                │         Health Checks +          │
                │      Auto Target Discovery       │
                └─────────────────┬────────────────┘
                                  │
                ┌─────────────────▼────────────────┐
                │         AWS EC2 Cluster          │
                │                                  │
                │  Pod can restart ANYWHERE        │
                │  without manual intervention     │
                │                                  │
                │  ┌─────────────────────────────┐ │
                │  │      Single Pod             │ │
                │  │ ┌─────────┐ ┌────────────┐  │ │
                │  │ │  nginx  │ │filebeat+   │  │ │
                │  │ │(reverse │ │logstash    │  │ │
                │  │ │ proxy)  │ │            │  │ │
                │  │ └─────────┘ └────────────┘  │ │
                │  └─────────────────────────────┘ │
                └──────────────────────────────────┘

NLB Limitations to Consider

Security Group Rules: Each NLB adds inbound rules to node security groups, which can exhaust the default limit causing deployments to fail.

Target Limits: With cross-zone load balancing, you're limited to 500 targets per load balancer.

Connection Limits: NLB supports about 55,000 simultaneous connections with client IP preservation enabled.

Cost: Cannot share a single NLB across multiple Kubernetes services, potentially increasing costs.

Operational Complexity: Introduces another component to monitor and troubleshoot.

Why This Solution Still Wins

Despite limitations, the NLB approach addresses the fundamental architectural problem. The limitations are manageable:

• Security group limits can be increased via AWS support
• Target limits are unlikely to be hit with single-pod architectures
• Connection limits are generous for most use cases
• Operational complexity is offset by eliminating manual intervention

Most importantly, this solution follows cloud-native principles: infrastructure lifecycle independent of application lifecycle.

The Bottom Line

The original architecture suffered from tight coupling between network identity and compute identity. Every time the application moved, infrastructure had to manually follow.

The NLB solution decouples these concerns:

• Network identity (Elastic IP) lives permanently with the load balancer
• Application identity (your pod) can freely move within the cluster
• AWS and Kubernetes handle the mapping automatically

While NLB has limitations, they're far outweighed by eliminating the operational burden of manual intervention. Your clients keep the same IP addresses, you get zero-downtime deployments, and manual steps become unnecessary.

This is what proper cloud-native architecture looks like: infrastructure that serves applications, not the other way around.