Is there a way to debug

upstream

requests on Oathkeeper? Oathkeeper doesn't seem to be handling

strip_path

on the rules For example;

"upstream": {
      "url": "<http://account.dev.svc.cluster.local>",
      "strip_path": "/account",
    }

The access log for the

account

service above is then showing

request_path":"/account/api/schema.json"

which of course returns a 404 as the actual path should be

/api/schema.json

- the

/account

should be stripped as per the upstream config above.

first of all, what version of oathkeeper are you running?

The latest that's provided via the helm charts -

v0.38.19-beta.1

I'm currently looking in the code to find why it might not work, as it should 😅

Thank you, yeah it's puzzled me too, I can't see any reason why it wouldn't. We're using Istio, with envoy sidecars, so as a plan b, I can always apply a rewrite filter to remove the path there instead. I'll try setting

preserve_host: true

- We're not actually setting that,

false

is the default that's being set

hm ok I don't know... there is extensive tests and it works for us, so I suspect a) typo, although I could not spot one b) it works, but some other part of your system messes with the request

I'm suspecting the same too, the code in oathkeeper that's handling the replace looks simple enough that it can't really go wrong 😅

hm yeah the logs are not too great to debug something like that, true...

@User Are you (or anyone at Ory) familiar with using Oathkeeper with Istio?

maybe @User might have some idea

So I think I've found the issue, but don't quite know what the solution is at the moment. It is our environment rather than anything Ory which you'll be pleased to hear. We use Istio / EnvoyFilters to incept inbound traffic and check auth with Oathkeeper. Our envoyfilter looks like this; https://gist.github.com/adamstrawson/1901d6475dec93a92eb2dbbe81494e83 It appears that when you use EnvoyFilters with Oathkeeper,

upstream

gets ignored. You can pass through any values you want in oathkeeper

upstream

and they're ignored. (eg if I do an

upstream.url: <http://foo.com>

which is invalid and should fail, the request still ends up at the intended destination. I think what is happening is Istio runs the filters from EnvoyFilter, calls

/decisions

to determine if the request can pass, and then envoy sends the request down the original intended route (rather than oathkeeper proxying the request)

ah ok I see...

Yeah, yeah in this case it may be istio, as all traffic is tunneled via sidecar to sidecar. In this case I'd expect it to override oathkeeper. You would need to disable the sidecar at the target and terminate mtls at oathkeeper for it to behave as expected

We're not using mtls (yet), the request path goes Istio Gateway -> (oathkeeper) -> Service Sidecar too, rather than sidecar to sidecar in this scenario, not sure if that matters much. I would doubt disabling sidecars on the targets would be an option for us as it'll be across several services, (the sidecars are handling network policies, tracing etc)