https://www.ory.sh/ logo
Join Slack
Powered by
So, I see that I can use the `ory_session_...` to ...
# talk-oathkeeper
h
So, I see that I can use the 
ory_session_...
to make a request to determine who the HTTP request belongs to in my Go backend. I need to get the 
identity.id
in my backend. However, is there a way to pass the 
identity.id
to the backend from oathkeeper instead of having to make an additional request in my backend?
s
You can use a 
mutator
in the Oathkeeper configuration to send that data to your backend via an http header, either in a header directly or a JWT token. E.g.,
Copy code
authenticators:
  cookie_session:
    enabled: true
    config:
      ...
      subject_from: 'identity.id'
mutators:
  header:
    enabled: true
    config:
      headers:
        X-User: '{{ print .Subject }}'
d
So, I did try that and I did not see my header showing up. Regardless whether I called it 
X-User
or 
Subject
Copy code
# Oathkeeper Config
authenticators:
      cookie_session:
        enabled: true
        config:
          check_session_url: <http://host.k3d.internal:4000/sessions/whoami>
          preserve_path: true
          extra_from: "@this"
          subject_from: "identity.id"
mutators:
      header:
        enabled: true
        config:
          headers:
            Subject: "Testing"

# Rules
- id: get-all-objects
      version: v0.39.0
      match:
        url: <.*>/v1/objects
        methods:
          - GET
      authenticators:
        - handler: cookie_session
      authorizer:
        handler: allow
      mutators:
        - handler: header
      errors:
        - handler: json
s
Any chance that you're using the API instead of the proxy? If so, you may have to configure your proxy to forward the header. E.g., for Envoy's ext_authz, 
authorization_response
allowed_upstream_headers
. I'm just asking because I don't see an upstream property in the access rule, but that may have just been omitted for the post.
h
That might be the issue. My Oathkeeper envoy filter has this for the http service:
Copy code
...
authorization_request:
  allowed_headers:
    patterns:
    - exact: accept
    - exact: authorization
    - exact: cookie
    - exact: content-type
    - exact: x-forwarded-for
    - exact: x-forwarded-proto
    - exact: x-forwarded-host
authorization_response:
  allowed_upstream_headers:
    patterns:
    - exact: authorization
So, I may need to add the header I am using, should probably go in the request section?
s
I believe the headers that the Oathkeeper header mutator adds or changes need to be added to the 
allowed_upstream_headers
section. E.g.,
Copy code
authorization_response:
  allowed_upstream_headers:
    patterns:
    - exact: authorization
    - exact: subject
    - exact: x-user
h
I am going to give that a shot and respond with my findings.
I am still not receiving those headers after modifying the envoy filter 😕
s
Hmm, I'm not sure what else is going on. That configuration seems to be working on my end. I'm including snippets of it in case anything stands out on your end. Oathkeeper access rule:
Copy code
- match:
    methods:
      - GET
    url: <https://api.ldev-njlaw.local/api/v1/ping>
  authenticators:
    - handler: noop
  authorizer:
    handler: allow
  mutators:
    - handler: header
      config:
        headers:
          X-User: my-special-user
Envoy filter configuration:
Copy code
- name: envoy.filters.http.ext_authz
    typed_config:
      '@type': <http://type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz|type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz>
      allowed_headers:
        patterns:
        - exact: accept
          ignore_case: true
        - exact: authorization
          ignore_case: true
        - exact: cookie
          ignore_case: true
        - exact: content-type
          ignore_case: true
        - exact: x-forwarded-for
          ignore_case: true
        - exact: x-forwarded-proto
          ignore_case: true
        - exact: x-forwarded-host
          ignore_case: true
      filter_enabled_metadata:
        filter: envoy.filters.http.rbac
        path:
        - key: istio_ext_authz_shadow_effective_policy_id
        value:
          string_match:
            prefix: istio-ext-authz
      http_service:
        authorization_response:
          allowed_upstream_headers:
            patterns:
            - exact: authorization
              ignore_case: true
            - exact: x-user
              ignore_case: true
        path_prefix: /decisions
        server_uri:
          cluster: outbound|4456||oathkeeper-api.ldev-njlaw.svc.cluster.local
          timeout: 600s
          uri: <http://oathkeeper-api.ldev-njlaw.svc.cluster.local>
      transport_api_version: V3
Test command:
Copy code
☸ minikube (ldev-njlaw) in ~ on ☁  (us-east-1) took 3m4s
❯ curl -i <https://api.ldev-njlaw.local/api/v1/ping>\?logHeaders\=true
HTTP/2 200
vary: Origin,Access-Control-Request-Method,Access-Control-Request-Headers
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: 0
strict-transport-security: max-age=31536000 ; includeSubDomains
x-frame-options: DENY
content-type: text/plain;charset=UTF-8
content-length: 0
date: Wed, 04 Oct 2023 18:37:49 GMT
x-envoy-upstream-service-time: 5
server: istio-envoy
set-cookie: ENVOY_LB_HASH="40819514c5b9a7d5"; Path=/api/v1; HttpOnly
Backend (upstream) logs:
Copy code
2023-10-04 18:32:38.433  INFO 8 --- [nio-8080-exec-9] c.x.cco.controller.SecurityController    : Request headers:
  host: api.ldev-njlaw.local
  user-agent: curl/7.81.0
  accept: */*
  x-forwarded-proto: https
  x-request-id: de8f5997-0968-4983-b611-6e8f21110ba3
  x-envoy-attempt-count: 1
  x-envoy-internal: true
  x-forwarded-client-cert: By=<spiffe://cluster.local/ns/ldev-njlaw/sa/cco-backend;Hash=298623e4d5aaebe42f01c018f1d4e74e12e37507812bdde4cc59c406cb393f65;Subject=>"";URI=<spiffe://cluster.local/ns/ldev-njlaw-gateway/sa/ingress>
  x-user: my-special-user
  x-b3-traceid: 82f2edce9dbfe9ba763d0f89cf95dc1c
  x-b3-spanid: 9f49acad71f6b63f
  x-b3-parentspanid: 763d0f89cf95dc1c
  x-b3-sampled: 0
h
Sorry for the delay. Going to take a look at this today or tomorrow. Thank you for providing this info to compare.
I've copy and pasted some your config that is slightly different in the envoy filter and it does not change my result. I've also 
noop
the authentication so there's no cookie being passed, mainly just want to see the 
X-User
passed. I am curling just like yours and my backend's metadata only includes this information:
Copy code
Metadata: map[:
    authority:[api-service:80]
    content-type:[application/grpc]
    grpcgateway-accept:[*/*]
    grpcgateway-user-agent:[curl/7.88.1]
    user-agent:[grpc-go/1.52.3]
    x-b3-parentspanid:[76a6c73ce5c395b9]
    x-b3-sampled:[0]
    x-b3-spanid:[918882f222597bf5]
    x-b3-traceid:[b0834dfa7a76d71c71234567e5c394e9]
    x-envoy-attempt-count:[1]
    x-forwarded-client-cert:[
        By=<spiffe://cluster.local/ns/app/sa/default>;
        Hash=...;
        Subject="";
        URI=<spiffe://cluster.local/ns/app/sa/default>
    ]
    x-forwarded-for:[10.90.9.0, 127.0.0.6]
    x-forwarded-host:[api.localhost:8080]
    x-forwarded-proto:[http]
    x-request-id:[9e9e9f5b-00a3-40b0-905b-b1481123987]
]
s
If you enable HTTP debug logging on the envoy sidecar, is there anything revealing? E.g., 
istioctl pc log <podname> --level http=debug
then 
curl ...
and 
kubectl logs <podname> istio-proxy | tail -n 200
You should see the request to Oathkeeper:
Copy code
2023-10-11T21:44:05.882566Z     debug   envoy http external/envoy/source/common/http/conn_manager_impl.cc:1089  [Tags: "ConnectionId":"82655","StreamId":"10727663888983783972"] request headers complete (end_stream=true):
':authority', 'api.ldev-njlaw.local'
':path', '/api/v1/ping?logHeaders=true'
':method', 'GET'
'user-agent', 'curl/7.81.0'
'accept', '*/*'
'x-forwarded-for', '10.244.0.1'
'x-forwarded-proto', 'https'
'x-envoy-internal', 'true'
'x-request-id', 'b9baf88d-3229-4fea-8d8d-6be6d0d2cbed'
'x-envoy-decorator-operation', 'cco-backend.ldev-njlaw.svc.cluster.local:8080/*'
'x-envoy-peer-metadata-id', 'router~10.244.2.53~ingress-gateway-v1-19-1-istio-5f8cc4b894-fj95r.ldev-njlaw-gateway~ldev-njlaw-gateway.svc.cluster.local'
'x-envoy-peer-metadata', 'ChQKDkFQUF9D...=='
'x-envoy-attempt-count', '1'
'x-b3-traceid', 'cb96702cfc6ca7b386cd4044bb42fb3d'
'x-b3-spanid', '86cd4044bb42fb3d'
'x-b3-sampled', '0'
        thread=23
2023-10-11T21:44:05.882572Z     debug   envoy http external/envoy/source/common/http/conn_manager_impl.cc:1072  [Tags: "ConnectionId":"82655","StreamId":"10727663888983783972"] request end stream     thread=23
Followed by Oathkeeper's response with the added header:
Copy code
2023-10-11T21:44:05.885768Z     debug   envoy http external/envoy/source/common/http/async_client_impl.cc:123   async http request response headers (end_stream=true):
':status', '200'
'x-user', 'my-special-user'
'date', 'Wed, 11 Oct 2023 21:44:05 GMT'
'content-length', '0'
'x-envoy-upstream-service-time', '2'
        thread=23
And in the Oathkeeper logs (taken from a later request, so the timestamps don't line up with above), the 
http_response
element should show the added header:
Copy code
time=2023-10-11T21:58:11Z level=info msg=completed handling request http_request=map[headers:map[accept:*/* content-length:0 x-b3-parentspanid:91b05f6d0f23bcb7 x-b3-sampled:0 x-b3-spanid:01e5d7e8868ed64e x-b3-traceid:a5c
5c33aa5c4e531fdecd6e17e2187d9 x-envoy-expected-rq-timeout-ms:600000 x-envoy-internal:true x-forwarded-for:10.244.0.1,10.244.2.246 x-forwarded-proto:https] host:api.ldev-njlaw.local method:GET path:/api/v1/ping query:Valu
e is sensitive and has been redacted. To see the value set config key "log.leak_sensitive_values = true" or environment variable "LOG_LEAK_SENSITIVE_VALUES=true". remote:127.0.0.6:60717 scheme:http] http_response=map[hea
ders:map[x-user:my-special-user] size:0 status:200 text_status:OK took:213.113µs]
h
Let me try that out.
I must be logging the wrong thing, because I don't even see my api path in the logs. I am setting the log level through istioctl on my oathkeeper pod, curling my api endpoint, and then reading the logs from oathkeeper's istio-proxy. All I am seeing are health checks and prometheus status. 😕
Actually. I see the API request to my backend but I do not see the 2nd log from Oathkeeper at all. So, no 
request response headers (end_stream=true)
in any oathkeeper istio-proxy logs. API Istio Proxy Logs
Copy code
2023-10-12T16:42:05.560888Z     debug   envoy http external/envoy/source/common/http/conn_manager_impl.cc:972   [C155][S12258433574594450538] request headers complete (end_stream=false):
':method', 'POST'
':scheme', 'http'
':path', '/api/read'
':authority', 'api:80'
'content-type', 'application/grpc'
'user-agent', 'grpc-go/1.52.3'
'te', 'trailers'
'grpc-timeout', '10000m'
'x-forwarded-for', '10.42.1.1, 127.0.0.6'
'grpcgateway-user-agent', 'curl/7.88.1'
'grpcgateway-accept', '*/*'
'x-forwarded-host', 'api.localhost:8080'
'x-forwarded-proto', 'http'
'x-request-id', '28d8a32f-044e-477d-9d25-123b98be1234'
'x-envoy-decorator-operation', 'api.appsvc.cluster.local:80/*'
'x-envoy-peer-metadata', 'CicKDkFQ...'
'x-envoy-peer-metadata-id', 'sidecar~10.42.2.90...'
'x-envoy-expected-rq-timeout-ms', '10000'
'x-envoy-attempt-count', '1'
'x-b3-traceid', 'b0b0bc80ab92cbc6b34a216991c31234'
'x-b3-spanid', 'b34a216991c31234'
'x-b3-sampled', '0'
        thread=38
2023-10-12T16:42:05.561386Z     debug   envoy http external/envoy/source/common/http/conn_manager_impl.cc:955   [C155][S12258433574594450538] request end stream        thread=38
s
Sorry, I should have been clearer. You should see the 
http request response headers (end_stream=true)
in the target pod's 
istio-proxy
logs, not Oathkeeper's 
istio-proxy
logs. If that's not present, I would confirm that Oathkeeper is seeing the request is it's container logs.
h
No worries! So, I have a gateway service that accepts HTTP/gRPC requests and routes them to gRPC endpoints. So, I found the 
http request response headers (end_stream=true)
in our api gateway 
istio-proxy
which is good. I also see the 
x-user
header there with 
my-special-user
. Seems like I am not allowed that header to propagate to the gRPC endpoints.
Okay interesting, so I added another field 
authorization
to the header mutation and I see that in my endpoint's metadata, but not the 
x-user
. So, it could be the envoy filter preventing or maybe not allowing through my api gateway
I think I found the root of the problem, but it still isn't fixing it. I see in some Virtual Services that allow and expose the 
Authorization
and 
Content Type
headers which could be blocking the headers. In the process of seeing if I can add other headers and resolve my problem.
Unfortunately, I can't get any other header through my pipeline. I narrowed down the 
authorization
header to only being needed in the envoy filter like your example. Adding any other heading there does not show up. I've caved and just decided to use the 
authorization
header for the 
identity.id
s
If there's nothing sensitive in it, I'd be happy to try and look at the envoy configuration from 
istioctl pc all <gateway-pod-name> -o json
. I'm still in the process of learning Envoy myself, but troubleshooting is always good practice!
9 Views
Open in Slack
PreviousNext
Powered by