https://www.ory.sh/ logo
Join Slack
Powered by
Hello. Question around Oathkeeper: > Does Oat...
# ory-selfhosting
r
Hello. Question around Oathkeeper:
Does Oathkeeper have any restrictions on which IPs it can trigger requests to?
Scenario: We have a 
hydrator
mutator set up for Oathkeeper running within Docker. The API endpoint we want to call is running on the host, so we set up the URL as:
Copy code
mutators:  
  noop:  
    enabled: true  
  hydrator:  
    enabled: true  
    config:  
      api:  
        url: <http://host.docker.internal>:{some-port}/some-endpoint
With this set up, Oathkeeper fails to call the endpoint, failing with an error:
Copy code
prohibited IP address: 0.250.250.254 is not a permitted destination (denied by: 0.0.0.0/8)
(I'm assuming that this is the IP 
host.docker.internal
is resolving to) However, if we expose the same service publicly (e.g. using ngrok)
Copy code
ngrok http {some-port}

> <https://some-random-ngrok-endpoint.ngrok.app>
and update the config:
Copy code
mutators:  
  noop:  
    enabled: true  
  hydrator:  
    enabled: true  
    config:  
      api:  
        url: <https://some-random-ngrok-endpoint.ngrok.app/some-endpoint>
then the endpoint gets called.
g
The resolved ip looks weird as It does not belong to common private IP ranges (like 192.168.x.x, 10.x.x.x, or 172.16.x.x), and it's not a valid public address either.
You can try to exec into the container to confirm the ip address just again to ascertain it’s properly resolved docker exec -it <oathkeeper-container> nslookup host.docker.internal
r
Here is the result of 
nslookup
from within the Oathkeeper container:
Copy code
/ $ nslookup host.docker.internal
Server:		127.0.0.11
Address:	127.0.0.11:53

Non-authoritative answer:

Non-authoritative answer:
Name:	host.docker.internal
Address: 0.250.250.254
However, I don't think this is unusual, since this is related to Docker networking. I think there's an application level check that's preventing calls to this host. I say this because using curl/wget to make a request to this host works; it's only Oathkeeper that's unable to make this call.
g
Just to understand, curl to 0.250.250.254 responds from the host?
r
Curl to 
<http://host.docker.internal>:{some-port}/some-endpoint
works when I perform it from within the Docker container However, Oathkeeper is unable to make the same API call. (
host.docker.internal
is how Docker recommends accessing the host from the container - ref https://docs.docker.com/desktop/features/networking/#i-want-to-connect-from-a-container-to-a-service-on-the-host)
g
Yes I understand. I just wanted to confirm you could access. Since you could use the dns name recommend, then oath keeper could be indeed blocking the calls. Do you have a gist of your docker compose for example you could share with me to run similar setup locally and see if I can replicate the same issue
r
Yes, I'll be able to do this later today once I have some free time. Thanks for your feedback so far.
Hello. Took me a while to create the reproducible example, which in turn allowed me to glean more information: this seems to be an issue with Oathkeeper 0.40.8 or later. Furthermore, this seems to be happening for me because I use Orbstack. Root cause: Ory has defined an "allow-list" of IP address to which requests can be made: https://github.com/ory/x/blob/8f55c13831d0ff51ffdb8c9211d3a987c13bad2c/httpx/ssrf.go#L91 When Orbstack sets up networking, it assigns 
host.internal.docker
to the IP address we were seeing in the logs (
0.250.250.254
). Since this IP is not included in that allow-list, this request gets denied. Therefore, I think the combination of Ory starting to have the IP address restriction, combined with using Orbstack, is the cause of this problem.
m
Hey Fred, not sure how to work around that - we don't use orbstack internally so we don't have a lot of experience with it. Can you add the orbstack IP to the allow-list - I guess that would require you to fork /x , also not ideal... 🤔
r
Hi @magnificent-energy-493 👋🏽 Yeah, not very ideal 🤔 I guess if Oathkeeper had a way to override the list of IPs to which requests can be made that would be ideal? I saw there were previous requests where other users have encountered this limitation, e.g because of IP ranges defined in their K8s clusters (e.g. https://github.com/ory/x/issues/805) For now, my workaround is to expose my local setup via 
ngrok
I've also asked the Orbstack team whether there's a way to configure the IP that's assigned to that domain: https://github.com/orgs/orbstack/discussions/1829
Or perhaps a config to disable the IP checks? Which a user can enable locally?
m
Yea, tbh I don't know Oathkeeper well enough to help here 😓 Maybe you can find something here in the community resources: https://github.com/ory/awesome-ory
25 Views
Open in Slack
PreviousNext
Powered by