Cannot pass client's real ip through nginx reverse proxy

[netsho]

Hello,

I need your help on how to enable the passing of the client's real ip to Authentik. Right now, the client IP is the IP of the nginx reverse proxy.

Authentik is running as a podman container, and is behind nginx reverse proxy container. Everything is working LAN only, no exposition to the public.

I already tried multiple solutions i found on the authentik documentation (TRUSTED_PROXY_CIDRS) and other issues pointing to the same problem (#7216, #7504, #7219), but i couldn't resolve it. Your help is much appreciated.
This is my Authentik podman-compose of the server:

 image: ghcr.io/goauthentik/server:2025.2.4
    container_name: authentik_server
    networks:
      - nginx-proxy-network
    restart: unless-stopped
    command: server
    environment:
      PUID: 1000
      PGID: 1000
      AUTHENTIK_REDIS__HOST: authentik_redis
      AUTHENTIK_POSTGRESQL__HOST: authentik_db
      AUTHENTIK_POSTGRESQL__USER: ${PG_USER}
      AUTHENTIK_POSTGRESQL__NAME: ${PG_DB}
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_LISTEN__TRUSTED_PROXY_CIDRS: "10.89.0.0/24,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16" # Tried multiple values here, none worked
      VIRTUAL_HOST: authentik.homelab.com
      CERT_NAME: homelab
    volumes:
      - /etc/authentik/config/media:/media
      - /etc/authentik/config/custom-templates:/templates
    env_file:
      - .env
    depends_on:
      authentik_db:
        condition: service_healthy
      authentik_redis:
        condition: service_healthy
    deploy:
      resources:
        limits:
          memory: 1536M
          cpus: "1"
    tmpfs:
      - /tmp    # Directs temporary files to memory instead of disk

I'm using jwilder nginx reverse proxy image which generates automatically the nginx configuration. NGINX and Authentik Server are in the same podman network (nginx-proxy-network). I have verified against the documentation (https://docs.goauthentik.io/docs/install-config/reverse-proxy), and i don't see anything missing.

# authentik.homelab-services.com/
upstream authentik.homelab-services.com {
    # Container: authentik_server
    #     networks:
    #         nginx-proxy-network (reachable)
    #     IPv4 address: 
    #     IPv6 address: (none usable)
    #     exposed ports (first ten): (none)
    #     default port: 80
    #     using port: 9000
    server 10.89.0.11:9000; # Authentik server ip adress in the podman network subnet
    keepalive 2;
}
server {
    server_name authentik.homelab-services.com;
    access_log /var/log/nginx/access.log vhost;
    http2 on;
    listen 443 ssl ;
    ssl_session_timeout 5m;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;
    ssl_certificate /etc/nginx/certs/homelab-services.crt;
    ssl_certificate_key /etc/nginx/certs/homelab-services.key;
    set $sts_header "";
    if ($https) {
        set $sts_header "max-age=31536000";
    }
    add_header Strict-Transport-Security $sts_header always;
    location / {
        proxy_pass http://authentik.homelab-services.com;
        set $upstream_keepalive true;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade_keepalive;
    }
}

Podman network inspect nginx-proxy-network:

"3b26bb2a3b387eb8e424b40a7cd8f5cbf8cb7e1e4b8b190ac6e6d66dac35f8c6": {
                    "name": "nginx-proxy",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.89.0.4/24",
                                        "gateway": "10.89.0.1"
                                   }
                              ]
                         }
                    }
               }
"b9ebe4ba2ba11275248463c786a4d1e539d45137ac7e659d445913bf9d2c52ac": {
                    "name": "authentik_server",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.89.0.11/24",
                                        "gateway": "10.89.0.1"
                                   }
                              ]
                         }
                    }
               }

I'll be happy to provide any more context/information.
Thank you in advance for your help.

[ImmanuelVonNeumann]

There are three possible issues I can imagine in this situation:

1. Nginx does not get the real ip from podman
2. Nginx does not pass the real ip to authentik
3. Authentik does not accept the Proxy Header

To debug whether you have an issue with 1. or 2. I suggest you create a whoami-container to check which headers are being sent from nginx to said container (I suggest https://github.com/traefik/whoami )
Ideally X-Forwarded-For is set and the IP is within AUTHENTIK_LISTEN__TRUSTED_PROXY_CIDRS.

If the X-Forwarded-For Header ist set but to the wrong ip: You likely have an issue with how podman forwards requests to nginx (rarely this could also be missconfiguration)
If the X-Forwarded-For Header is not set, you have an issue with your nginx configuration.
If the Source IP is not withing your Trusted Proxy CIDRs then you have an issue with your authentik configuration.

PS: I also found this similar issue #5529 . Maybe there is a similar cause.

[netsho]

After creating a whoami container, setting it up behind the nginx reverse proxy and debugging using curl, i saw that the X-Forwarded-For header is set to the IP of the nginx container. So, it's likely an issue with podman.
The issue #5529 isn't applied to podman, as userland-proxy is already disabled in podman.
After digging for a while in podman issues/discussions, i came by this containers/podman#23845 which is a similar issue to mine.
There are some solutions in the discussion (like nginx socket activation), i'm trying to apply them.
I'll keep this discussion open until I post about the solution for future references.

[eriksjolund]

Yes, nginx will see the real source IP address when using socket activation. Probably you will need to convert the compose file to quadlet files. There is no support for socket activation in docker-compose. I'm not sure whether podman-compose supports socket activation. In any case, using quadlet files is recommended by the Podman project. Quadlet files are compatible with socket activation.

[eriksjolund]

Some guides I wrote:

Socket activation of containers

HTTP reverse proxy which mentions some tips about how to connect from a container on the custom network to the proxy (with the help of NetworkAlias=whoami.example.com or AddHost=whoami.example.com:host-gateway)

podman-nginx-socket-activation contains some examples. Unfortunately, I didn't write an example of a system user service using rootless Podman with a custom network.

[netsho]

Hello,
Thank you for your help.

I have been trying to understand the socket activation for the last week. I'm not able to make it work (but i guess I overcomplicated it).
I'm using this nginx reverse proxy image to automate the creation of the configuration files.
The containers and this reverse proxy need to run on the same custom network, so it can detect new containers and create the corresponding configuration.

I started with the example3:

• created the socket as by the example
• copied the generated configuration files from the nginx container (default.conf and others) using podman cp into a directory on the host
• updated the service file to mount this directory to the podman run command on ExecStart

[Unit]
Description=Podman Nginx Reverse Proxy (HTTP)
Wants=network-online.target
After=network-online.target
Requires=user@1000.service
After=user@1000.service
RequiresMountsFor=/run/user/1000/containers

[Service]
User=netsho
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStop=/usr/bin/podman rm -f -i --cidfile=/run/user/1000/%N.cid
ExecStopPost=-/usr/bin/podman rm -f -i --cidfile=/run/user/1000/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman run \
     --cidfile=/run/user/1000/%N.cid \
     --cgroups=split \
     --rm \
     --env "NGINX=3;" \
      -d \
     --replace \
     --name systemd-%N \
     --sdnotify=conmon \
     --network none \
     -v /etc/nginx/certs:/etc/nginx/certs:ro \
     -v /etc/nginx/socket-activation-nginx/nginx:/etc/nginx:ro \
     -v /run/user/1000/podman/podman.sock:/tmp/docker.sock:ro
     docker.io/library/nginx

• Did the same for https: I created another socket which listens on 443 and another .service (i tried the same .service for both sockets, but couldn't make it work).
• The sockets are started, but curl -s localhost:80 | head -4 doesn't return anything, and the service fails after that. Same for https socket.
• Kept searching to resolve it, couldn't find anything relevant.
• I then tried with example4, didn't work.

I reviewed your response, and I thought to use quadlet files instead (recommended by podman), so I followed the example1:

• I followed the tutoriel step by step, it worked.
• After that, I needed to update the .container file. I came by podlet tool which generates quadlet file from compose file.
• I generated the quadlet file from my nginx compose file, and updated it as to match the example.

[Unit]
Requires=nginx-http.socket
After=nginx-http.socket

[Container]
ContainerName=nginx-proxy-quadlet-http
Environment=PUID=1000 PGID=1000 DISABLE_IPV6=true TRUST_DOWNSTREAM_PROXY=false HTTPS_METHOD=nohttp ENABLE_HTTP_ON_MISSING_CERT=false SSL_POLICY=Mozilla-Modern
Image=docker.io/library/nginx
Environment=NGINX=3;
Network=nginx-proxy-network
#Network=none
PublishPort=80:80
Volume=/etc/nginx/certs:/etc/nginx/certs:ro
Volume=/etc/nginx/socket-activation-nginx/nginx:/etc/nginx:ro

[Service]
Restart=always

[Install]
WantedBy=default.target

• When ran with the custom network so it can reach the containers, running curl -s localhost:80 | head -4 doesn't output anything, it keeps running.
• I switched to Network=none, ran curl -s localhost:80 | head -4 and i got "301 Moved Permanently", which is great because the default.conf is configured as to redirect http to https. But the problem here is i couldn't reach my containers (for example: https://homepage.homelab-services.com).

I also tried Http reverse proxy, but then the daemon didn't run, maybe doing it wrong.

Now I'm at a dead end. I'm thinking that using the generated config files wouldn't work with activated socket, but I can't come up with any other solution.

Thank you in advance for your response.

Have a good day

[eriksjolund]

I'm replying a short answer first. I'll try to investigate more later (within a few weeks).

Some suggestions/ideas:

Use quadlet files as it is simpler and less error-prone.
(The reason I needed to use *.service files in some of the examples is that I wanted to use
the systemd directive User= but doing that is experimental and unsupported by the Podman project)

Remove PublishPort=80:80 as it is not needed when using socket activation. (I think it could cause problems)
Don't use Network=none instead use Network=nginx-proxy-network

Start with trying to get it to work with just HTTP (TCP/80) first. I think you then need to configure nginx to not use HTTPS (TCP/443).

If you would like nginx to serve both TCP/80 and TCP/443 then you need to
use

Environment=NGINX=3;4;

instead of

Environment=NGINX=3;

Then combine it with something like

[Socket]
ListenStream=80
ListenStream=443

[Install]
WantedBy=sockets.target

The order of the ListenStream= statements can be important. It might be

ListenStream=443
ListenStream=80

Advanced tip for later:
Network=none can sometimes be used if the container is running stand-alone without using a custom network. To achieve something similar for custom networks you could add Internal=true as is done in
https://github.com/eriksjolund/podman-caddy-socket-activation/tree/main/examples/example2#using-internaltrue
but wait with that until you get the basic functionality working.