https://www.ory.sh/ logo
Join Slack
Powered by
hello, originally asked this in <#C02PSRUGUAV|self...
# talk-oathkeeper
n
hello, originally asked this in #self-hosted, suggested i move here - i’m running into an annoying issue with my oathkeeper setup in production that doesn’t occur locally in development, i’m using the bearer_token authenticator to validate a legacy token that gets converted using the id_token mutator, locally this works fine and the subject is set correctly in the generated token, but for some reason in production the subject always comes through with an empty string
i have verified that if i pass in an invalid (e.g. expired) token, authentication fails, so it appears to be executing the authenticator correctly
and if i call the same check_session_url manually in postman i get
Copy code
{
  "sub": "291677"
}
back as my response, so i think that should let oathkeeper set the subject in the id_token
is there anything obvious i might be missing? or a good way to debug what is happening inside the authenticator? when i had a previous issue with the authenticator i tried turning up logging, but it doesn’t give any additional information about the authenticator itself, just additional tracing information around the request
m
Hey Andrew, So the request goes through correctly but returns an empty response? Are you able to share you configuration and the full response? Please redact any sensitive information.
n
here is the parsed JWT data received by my upstream application from the oathkeeper id_token (using jwt.io to read)
i have noop as an allowed authenticator because i also need some un-auth’d requests to pass through, but i don’t think that’s the issue since if i pass in an expired token oathkeeper does give me an error response
it’s odd, because the empty string in the subject is what i get when a request passes through the noop authenticator, but because it only succeeds when i provide a valid token it seems to be executing the bearer_token authenticator, not the noop one
d
Hello Andrew. Could you please tell more about your setup? What do you use as ingress controller? Because it can be configuration mismatch on the level above (e.g you’re using Nginx or envoy as load balancer for your microservices)
Configuration of oathkeeper looks good and should work fine
Some load balancers can be configured to prevent request to be modified
You can read discussion that looks like your case https://ory-community.slack.com/archives/C01340V8KSM/p1646302955836719
n
the header is being modified, it just doesn’t contain what it should. it’s a valid id_token generated by oathkeeper, but the subject is simply an empty string even though when i call that check session service separately from oathkeeper with the same source authorization token i’m providing to oathkeeper it does return the sub in the response as i expect
so i think my problem is different
i’m also running oathkeeper as a proxy, so i don’t think the gateway would have any control over whether oathkeeper is able to modify the request being passed through (i hope)
importantly, virtually the same configuration works locally (different only in match and upstream URLs, all other logic the same), but not production, so there does seem to be an environment-related issue, i just can’t figure out what it would be
also probably importantly, the same oathkeeper instance does work for a different service using a very similar rule configuration
Copy code
- id: public-api
  upstream:
    url: "<http://public-api-rpc.service.consul:9001>"
    preserve_host: true
    strip_path: /public-api
  match:
    methods:
      - GET
      - POST
      - PUT
      - DELETE
    url: "http<s?>://<(<http://oathkeeper.production.qtz.sh/public-api|oathkeeper.production.qtz.sh/public-api>)|(api.quartzy.com)><.*>"
  authenticators:
    - handler: bearer_token
      config:
        check_session_url: "<http://authentication-backend.service.consul:8002/access-token/authenticate>"
        preserve_path: true
        token_from:
          header: Access-Token
    - handler: noop
  authorizer:
    handler: allow
  mutators:
    - handler: id_token
      config:
        claims: '{"scopes": []}'
that service does seem to receive a token with the correct subject assigned
i can’t see any obvious differences why one would work and the other not, unless oathkeeper is receiving a different response than i expect from the check_session service in the first case, but so far debugging that has been impossible from what i have found
d
I’ll try to reproduce it and come back in a few hours
n
thank you, also if you know of a good way to debug on my side please let me know
oh, i think i might have found the problem, just saw a message in the docs 
Please note that Gzipped responses from check_session_url are not currently supported, and will fail silently.
so i suspect this is my issue, will confirm and report back shortly
yes, now almost certain that’s the problem, when i made the service request through postman instead of the browser and forced the 
Accept-Encoding
header to exclude gzip from the options, the request succeeded
d
Thanks for update. Is everything works fine now?
n
with oathkeeper, yes, i just need to figure out the right solution in our server configuration to allow gzip for all requests except the one oathkeeper uses to validate tokens
there is likely an improvement that can be made to oathkeeper here, ideally probably to support gzip, but if not at least remove gzip from the Accept-Encoding header of the request it forwards to the check_session_url, but at least for now i know what i need to do to work around this limitation
that sounded kind of negative, overall i’m very happy with oathkeeper and the overall platform from ory, just may be something to consider
Open in Slack
PreviousNext
Powered by