Hi! Trying to run Hydra behind Apisix Gateway (and ingress controller) in production with TLS termination at apisix gateway. I feel like I've tried every possible combination of TLS settings and am very close to dropping Hydra altogether after having invested days into this - please help me determine how stupid I am. In short: What must I do to serve Hydra in HTTP mode behind an API Gateway (Apisix), so that Hydra finally stops complaining? There's two core issues from the hydra logs: 1.

Could not serve http connection audience=application error=map[message:X-Forwarded-Proto header is missing]

2.

error=map[message:can not serve request over insecure http]

Currently I'm running k3s on a local VM with self-signed certs (CA and Apisix). TLS termination works fine on Apisix. My only goal is to run it for machine to machine access tokens. Apisix has a plugin system, which lets me configure "openid-connect" where I have to set the "discovery" URL to point to Hydra. I tried configuring Hydra to run without TLS, because I'm running it behind an API Gateway anyway, so this shouldn't be an issue. But every try I made resulted in

x-forwarded-proto header is missing.

I tried some proxy-rewrite trickery with Apisix, but Hydra was unfazed by that. No https x proto header for me. Every proper route I took to wire Apisix discovery URL to Hydra still resulted in those two errors. Tried multiple "allow_termination_from" variations and setting "serve.tls" key+crt - with and without. Here's the relevant Apisix plugin config:

kind: ApisixRoute
metadata:
  name: secure-app
  namespace: default
spec:
  http:
    - name: app-route
      match:
        hosts:
          - secure-app.k3s.local
        paths:
          - /*
      backends:
        - serviceName: secure-app
          servicePort: 8080
      plugins:
        - name: openid-connect
          enable: true
          config:
            client_id: ...
            client_secret: ...
            discovery: <http://hydra-public.default.svc.cluster.local:4444/.well-known/openid-configuration>

Now, the tricky part seems to be how to configure hydra (which I've installed via Helm Chart v0.45.0 with Hydra v2.2.0). Here's the latest relevant excerpt from my helm values.file after many different tries (won't list all of them).

hydra:
  config:
    dsn: memory
    ttl:
      access_token: 8h
    urls:
      self:
        issuer: <https://hydra-public.default.svc.cluster.local/>
      login: <http://wrong-on-purpose/login>
      consent: <http://wrong-on-purpose/consent>
    secrets:
      system:
        - FUJEF0ef09FezJiQ252b3Vqc2wg98sdg0=
    log:
      level: debug

I even tried throwing in linkerd in hope of Hydra not having "insecure http", but to no avail. In the end I tried running Hydra with its own TLS cert by adding/mounting this self signed cert:

serve:
      tls:
        enabled: true
        key:
          path: /etc/secrets/hydra-tls/tls.key
        cert:
          path: /etc/secrets/hydra-tls/tls.crt
        allow_termination_from: []

Only to realize that I will have 2x TLS termination, which Apisix can't handle in this plugin. I don't deem this a viable route. It's just too far out there and was already a huge pain to arrive at that point. Hydra must run in HTTP mode. But I simply can't find the missing link. I have no clue at all anymore, what I'm doing so horribly wrong trying to set up Hydra to run in plain HTTP mode. The helm chart docs mention ``hydra.dangerousForceHttp`` but this seems to be outdated. Its not in the chart itself and I couldn't get it to recognize this option. Everything I tried even after correct TLS termination in Apisix and supposedly correct forwarding to Hydra resulted in those two issues mentioned at the beginning (x-forwarded-proto missing, can't seve over insecure http). Please a help a really tired dev here.

Never heard of apisix, but from googling it (https://apisix.apache.org/docs/apisix/plugins/openid-connect/) there is an

authorization_params

option. You could hack that to sent

x-forwarded-proto: https

.

@refined-kangaroo-48640 Well, it wasn't so easy as just to set the "authorization_params", but you had the right direction... I delved deep into the plugins' code (openid-connect) and hardcoded the header "X-Forwarded-Proto: https" into it. It finally works. I can say confidently, that this is no Hydra issue now, nor my stupidity. I assume this is some kind of regression that I will need to report to the Apisix Issue tracker. Anyway, thanks for your help!