ArgoCD Ingress Fix - Traefik Bad Gateway

Environment

Cluster: RKE2 managed by Rancher
Ingress Controller: Traefik (kube-system namespace)
ArgoCD Version: v3.4.2 (Helm chart argo-cd-9.5.14)
Namespace: infrastructure
Hostname: argo.snarfnet.net

Problem

After deploying ArgoCD, accessing https://argo.snarfnet.net returned a 502 Bad Gateway from Traefik.

Root Cause

Two issues were identified:

1. Service TargetPort Mismatch

The ArgoCD server was listening on port 8080, but the Kubernetes service had targetPort: 8081. This was corrected by patching the service to point both ports (80 and 443) to targetPort 8080.

2. Traefik Protocol Mismatch (Primary Issue)

The ArgoCD service defined two ports:

ports:
  - name: http
    port: 80
    targetPort: 8080
  - name: https
    port: 443
    targetPort: 8080

The Ingress resource routed traffic to port 80, but Traefik's Kubernetes provider saw the port named https (443) on the service and automatically selected it, connecting to the backend using HTTPS:

"servers":[{"url":"https://10.42.1.76:8080"}]

However, ArgoCD was configured to run in insecure mode (server.insecure: true), meaning it only served plain HTTP on port 8080. Traefik's HTTPS connection to an HTTP backend resulted in the Bad Gateway.

Working services (Gitea, Jenkins, etc.) did not have this problem because they only exposed a single HTTP port with no https named port to confuse Traefik.

Fix

Removed the https (port 443) entry from the argocd-server service, leaving only the HTTP port:

spec:
  ports:
    - name: http
      port: 80
      targetPort: 8080

This forced Traefik to use http:// when connecting to the backend, which matched ArgoCD's insecure mode.

After the change, Traefik's internal service config showed:

"servers":[{"url":"http://10.42.1.76:8080"}]

Permanent Fix for Helm Upgrades

To prevent the Helm chart from recreating the 443 port on future upgrades, use one of these approaches:

Option A: Annotate the Ingress

Add this annotation to the argo-ing Ingress resource so Traefik always uses HTTP regardless of service port names:

metadata:
  annotations:
    traefik.ingress.kubernetes.io/service.serversscheme: http

Option B: Helm Values

Configure the chart to not expose the HTTPS service port (check chart documentation for exact key, as it varies by version):

configs:
  params:
    server.insecure: true

server:
  service:
    type: ClusterIP

Debugging Steps That Led to the Fix

Verified the pod was running and healthy (1/1 Ready)
Confirmed the pod was listening on port 8080 via /proc/net/tcp6
Tested direct pod connectivity from another pod in the cluster — returned HTTP 200
Queried Traefik's internal API at http://127.0.0.1:9000/api/http/services
Discovered Traefik was using https:// to connect to the backend
Compared with working services (Gitea, Jenkins) which all used http://
Identified the https named port on the service as the cause

Key Takeaway

Traefik's Kubernetes Ingress provider infers the backend protocol from the service port name. A port named https causes Traefik to connect using HTTPS, regardless of what port number the Ingress backend specifies. When running ArgoCD in insecure mode behind a TLS-terminating reverse proxy, ensure the service does not expose an https named port, or use the traefik.ingress.kubernetes.io/service.serversscheme annotation to override the behavior.

3.5 KiB Raw Blame History