https://www.ory.sh/ logo
#talk-oathkeeper
Title
# talk-oathkeeper
t

tall-angle-41306

04/20/2022, 5:00 PM
Is there a way to debug
upstream
requests on Oathkeeper? Oathkeeper doesn't seem to be handling
strip_path
on the rules For example;
Copy code
"upstream": {
      "url": "<http://account.dev.svc.cluster.local>",
      "strip_path": "/account",
    }
The access log for the
account
service above is then showing
request_path":"/account/api/schema.json"
which of course returns a 404 as the actual path should be
/api/schema.json
- the
/account
should be stripped as per the upstream config above.
Hey @User @User Sorry for the tag, I've hit a wall with this and it has become a blocker for us progressing
Full rule below;
Copy code
[
  {
    "upstream": {
      "url": "<http://account.dev.svc.cluster.local>",
      "strip_path": "/account",
      "preserve_host": false
    },
    "id": "dev-account-allow.auth",
    "match": {
      "url": "<http|https>://dev.foo.io/account/<.*>",
      "methods": [
        "GET",
        "POST",
        "PUT",
        "DELETE",
        "PATCH"
      ]
    },
    "authenticators": [
      {
        "handler": "anonymous"
      }
    ],
    "authorizer": {
      "handler": "allow"
    }
  }
]
A request to
<https://dev.foo.io/account/api/schema.json>
should result in an upstream request to
<http://account.dev.svc.cluster.local/api/schema.json>
I can confirm that the above rule matches fine, but the upstream service receives a request of
<http://account.dev.svc.cluster.local/account/api/schema.json>
s

steep-lamp-91158

04/21/2022, 8:53 AM
first of all, what version of oathkeeper are you running?
t

tall-angle-41306

04/21/2022, 8:54 AM
The latest that's provided via the helm charts -
v0.38.19-beta.1
s

steep-lamp-91158

04/21/2022, 9:03 AM
I'm currently looking in the code to find why it might not work, as it should 😅
did you try already setting
preserve_host
to true? I could only find examples that I know work with
preserve_host
being true
t

tall-angle-41306

04/21/2022, 9:06 AM
Thank you, yeah it's puzzled me too, I can't see any reason why it wouldn't. We're using Istio, with envoy sidecars, so as a plan b, I can always apply a rewrite filter to remove the path there instead. I'll try setting
preserve_host: true
- We're not actually setting that,
false
is the default that's being set
I can confirm setting
preserve_host: true
doesn't change the behaviour
s

steep-lamp-91158

04/21/2022, 9:24 AM
hm ok I don't know... there is extensive tests and it works for us, so I suspect a) typo, although I could not spot one b) it works, but some other part of your system messes with the request
did you already increase the log level and enable leak_sensitive_values and then check the oathkeeper logs?
t

tall-angle-41306

04/21/2022, 9:25 AM
I'm suspecting the same too, the code in oathkeeper that's handling the replace looks simple enough that it can't really go wrong 😅
I'd set log level to debug and trace, but it didn't really give much insight
s

steep-lamp-91158

04/21/2022, 9:31 AM
hm yeah the logs are not too great to debug something like that, true...
t

tall-angle-41306

04/21/2022, 12:55 PM
@User Are you (or anyone at Ory) familiar with using Oathkeeper with Istio?
s

steep-lamp-91158

04/21/2022, 1:20 PM
maybe @User might have some idea
t

tall-angle-41306

04/21/2022, 1:39 PM
So I think I've found the issue, but don't quite know what the solution is at the moment. It is our environment rather than anything Ory which you'll be pleased to hear. We use Istio / EnvoyFilters to incept inbound traffic and check auth with Oathkeeper. Our envoyfilter looks like this; https://gist.github.com/adamstrawson/1901d6475dec93a92eb2dbbe81494e83 It appears that when you use EnvoyFilters with Oathkeeper,
upstream
gets ignored. You can pass through any values you want in oathkeeper
upstream
and they're ignored. (eg if I do an
upstream.url: <http://foo.com>
which is invalid and should fail, the request still ends up at the intended destination. I think what is happening is Istio runs the filters from EnvoyFilter, calls
/decisions
to determine if the request can pass, and then envoy sends the request down the original intended route (rather than oathkeeper proxying the request)
s

steep-lamp-91158

04/21/2022, 1:57 PM
ah ok I see...
w

worried-kitchen-94392

04/21/2022, 2:25 PM
Yeah, yeah in this case it may be istio, as all traffic is tunneled via sidecar to sidecar. In this case I'd expect it to override oathkeeper. You would need to disable the sidecar at the target and terminate mtls at oathkeeper for it to behave as expected
t

tall-angle-41306

04/21/2022, 3:31 PM
We're not using mtls (yet), the request path goes Istio Gateway -> (oathkeeper) -> Service Sidecar too, rather than sidecar to sidecar in this scenario, not sure if that matters much. I would doubt disabling sidecars on the targets would be an option for us as it'll be across several services, (the sidecars are handling network policies, tracing etc)
Just to close the loop on this, I've got this all working now, with Istio / EnvoyFilters running as before. Since we can't use
upstream.strip_path
the path rewrite is now being handled within a Istio VirtualService
Thanks both for all the help today! Much appreciated
5 Views