Why AWS Load Balancers Made Our Logs Useless

The Problem

In a recent project, we deployed a logging pipeline using Nginx, Filebeat, and Logstash containers on AWS. Multiple clients pointed their DNS records to our AWS public IP, routing through a Network Load Balancer to our Nginx reverse proxy.

Everything seemed to work perfectly—websites loaded, APIs responded—until I checked our analytics dashboard.

Our application included geolocation features that tracked user locations based on IP addresses. Instead of seeing global traffic distribution, every single request appeared to originate from the same 172.31.x.x private IP address. The geolocation API failed completely, showing 100% of traffic as "Unknown Location."

Our Architecture

The request flow looked straightforward:

Client DNS → AWS Public IP → Network Load Balancer → Nginx Reverse Proxy → Backend APIs

Nginx handled multiple client websites based on the Host header, routing requests to appropriate backend servers. From a connectivity standpoint, everything worked perfectly.

Debugging: Nginx Configuration

My initial assumption was that this had to be an Nginx configuration issue. I added the real_ip module settings:

http {
    # Configure real IP extraction
    real_ip_header X-Forwarded-For;
    set_real_ip_from 172.31.0.0/16;  # VPC CIDR
    set_real_ip_from 10.0.0.0/8;     # Custom VPC CIDR
    real_ip_recursive on;
    
    # Custom log format to verify real IPs
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "XFF: $http_x_forwarded_for"';
}

server {
    listen 80;
    server_name client1.example.com client2.example.com;
    access_log /var/log/nginx/access.log main;
    
    location / {
        proxy_pass http://backend_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

I verified everything was working:

# Check if real IP module is available
nginx -V 2>&1 | grep -o with-http_realip_module

# Test and reload configuration
sudo nginx -t && sudo nginx -s reload

# Monitor logs
sudo tail -f /var/log/nginx/access.log

The configuration was valid, the module loaded, but the geolocation API continued receiving private IP addresses.

The Culprit: AWS Load Balancer Settings

That's when I realized the issue wasn't in Nginx—it was in AWS. The 172.31.x.x IP wasn't from misconfigured Nginx; it was the Network Load Balancer's own IP address.

I checked the AWS console and found the problem: "Preserve client IP addresses" was disabled in the target group settings.

This meant the NLB was replacing original client IPs with its own elastic network interface IP. No amount of Nginx configuration could recover information that AWS had already stripped away.

The Solution

The fix required changes in AWS console:

After enabling this setting, the geolocation API immediately began receiving real client IP addresses.

When Client IP Preservation Won't Work

AWS automatically disables client IP preservation in certain scenarios:

• Cross-VPC targets: Nginx server in different VPC than the NLB
• PrivateLink connections: Traffic through VPC endpoints
• Hairpinning: Client and server on same network
• Remote targets: Targets outside the NLB's VPC

In these cases, you must use Proxy Protocol v2:

server {
    listen 80 proxy_protocol;
    
    # Extract IP from proxy protocol
    real_ip_header proxy_protocol;
    set_real_ip_from 0.0.0.0/0;  # Safe with proxy protocol
    
    # Use proxy protocol variables in logs
    log_format proxy '$proxy_protocol_addr - [$time_local] "$request" '
                     '$status $body_bytes_sent "$http_referer"';
    
    access_log /var/log/nginx/access.log proxy;
    
    location / {
        proxy_pass http://backend_servers;
        # Forward the real client IP
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
    }
}

Why This Issue is Unique to AWS

I had deployed identical Nginx setups on Hetzner Cloud, DigitalOcean, and local environments without encountering this behavior. The same configuration that failed on AWS worked perfectly elsewhere.

AWS Network Load Balancer has unique design decisions:

• Default behavior prioritizes performance over IP transparency
• Target type and protocol combinations affect IP preservation differently
• Requires explicit configuration for IP transparency

Other platforms handle this differently:

• Hetzner Cloud: Preserves source IPs naturally
• DigitalOcean: Uses standard X-Forwarded-For headers by default
• Local environments: Direct Nginx receives client IPs without intermediate load balancers

Key Lessons Learned

1. AWS defaults prioritize performance over transparency - Always review infrastructure settings when client metadata is critical
2. Infrastructure changes require application updates - Enabling client IP preservation changes your security model
3. Test thoroughly beyond basic connectivity - Verify that all metadata flows correctly through your entire stack
4. Document infrastructure dependencies - When clients depend on your infrastructure, your configuration choices affect their functionality
5. Monitor for silent failures - Set up alerting to detect when client IP preservation stops working

Verification and Testing

After implementing the fix, verify it works:

# Monitor nginx logs for real IPs
sudo tail -f /var/log/nginx/access.log

# Test geolocation API
curl -H "Host: yourclient.com" http://your-aws-ip/api/geolocation

# Verify nginx configuration
nginx -T | grep real_ip

This debugging experience reinforced that modern applications rely on complex infrastructure stacks where optimal functionality requires careful configuration at every layer. Sometimes the solution isn't in the code you're debugging—it's in the cloud infrastructure settings you might have overlooked.