#talk-keto
Title
h
happy-morning-85531
03/14/2022, 6:52 PMAnyone know why im getting 404 on the keto protobuf write API calls? Attaching logs in thread
Copy code
ime=2022-03-14T18:44:53Z level=info msg=started handling request http_request=map[headers:map[content-type:application/grpc grpc-accept-encoding:gzip te:trailers user-agent:grpc-java-netty/1.44.0 x-b3-parentspanid:52145d659ec72f10 x-b3-sampled:0 x-b3-spanid:3f331637fd61f3ce x-b3-traceid:fc7685fc00116c1c52145d659ec72f10 x-envoy-attempt-count:1 x-forwarded-client-cert:By=<spiffe://cluster.local/ns/ory/sa/ory-ksa;Hash=a814987f150b3d35acccfdee792e9b30c2ad4055e450b358402ed5d98f6a1714;Subject=>"";URI=<spiffe://cluster.local/ns/iam/sa/default> x-forwarded-proto:http x-request-id:c07a215e-7c89-4755-9b17-3ba7ab95deb7] host:keto-write.ory:80 method:POST path:/ory.keto.acl.v1alpha1.WriteService/TransactRelationTuples query:<nil> remote:127.0.0.6:51519 scheme:http]
time=2022-03-14T18:44:53Z level=info msg=completed handling request http_request=map[headers:map[content-type:application/grpc grpc-accept-encoding:gzip te:trailers user-agent:grpc-java-netty/1.44.0 x-b3-parentspanid:52145d659ec72f10 x-b3-sampled:0 x-b3-spanid:3f331637fd61f3ce x-b3-traceid:fc7685fc00116c1c52145d659ec72f10 x-envoy-attempt-count:1 x-forwarded-client-cert:By=<spiffe://cluster.local/ns/ory/sa/ory-ksa;Hash=a814987f150b3d35acccfdee792e9b30c2ad4055e450b358402ed5d98f6a1714;Subject=>"";URI=<spiffe://cluster.local/ns/iam/sa/default> x-forwarded-proto:http x-request-id:c07a215e-7c89-4755-9b17-3ba7ab95deb7] host:keto-write.ory:80 method:POST path:/ory.keto.acl.v1alpha1.WriteService/TransactRelationTuples query:<nil> remote:127.0.0.6:51519 scheme:http] http_response=map[headers:map[content-type:text/plain; charset=utf-8 x-content-type-options:nosniff] size:19 status:404 text_status:Not Found took:205.347µs]s
steep-lamp-91158
03/14/2022, 7:36 PM
We are multiplexing the REST and gRPC servers on the same port
That log statement looks like it comes from the REST handler 🤔 so the multiplexing is not working.... What client and ingress are you using?
This is the detection logic https://github.com/ory/keto/blob/ef103eb231c6ca466d7c928ca73b29bf3d4c23d1/internal/driver/daemon.go#L189
h
happy-morning-85531
03/14/2022, 7:45 PMI'm on kubernetes just using the plain service and it's a internal call between pods so no need for an ingress
s
steep-lamp-91158
03/14/2022, 7:56 PM
Was just wondering as there is envoy and forwarded headers and stuff
h
happy-morning-85531
03/14/2022, 7:57 PMIstio service mesh so it's being applied by sidecar/proxy
s
steep-lamp-91158
03/14/2022, 7:57 PM
Maybe the request is send over HTTP1?
h
happy-morning-85531
03/14/2022, 7:58 PMIt's a grpc client used in spring boot so sounds weird that it would be sending it using the wrong protocol (version)
But it's something happening in the mesh, integration tests works locally with docker compose works fine together er with this application
s
steep-lamp-91158
03/14/2022, 8:03 PM
Ok then I have no idea 😅
I can only tell by the logs that the request is wrongly handled by the REST server
h
happy-morning-85531
03/14/2022, 8:05 PMIts weird, since the criteria seems to be on the content-type header but from the logs the header is there
s
steep-lamp-91158
03/14/2022, 8:05 PM
It's the header and HTTP2
h
happy-morning-85531
03/14/2022, 8:06 PMah, illl snoop around and see if theres some rewrites happening
Istio is the culprit https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/
The ports in the helm chart are called
http-write|reador maybe not, creating another service and specifying grpc appProtocol didnt work either
s
steep-lamp-91158
03/14/2022, 8:57 PM
ah I remember some issues with naming the ports
h
happy-morning-85531
03/14/2022, 9:17 PMim debugging envoy and i can see the headers
Copy code
2022-03-14T21:12:30.340519Z debug envoy http [C4577][S10006260048823567630] request headers complete (end_stream=false):
':authority', 'keto-write-grpc.ory.svc.cluster.local:4467'
':path', '/ory.keto.acl.v1alpha1.WriteService/TransactRelationTuples'
':method', 'POST'
'content-type', 'application/grpc'
'te', 'trailers'
'user-agent', 'grpc-java-netty/1.44.0'
'grpc-accept-encoding', 'gzip'
'x-request-id', 'ef779e17-846f-4c8e-9d1c-90b8b6a68061'
'x-envoy-decorator-operation', 'keto-write.ory.svc.cluster.local:4467/*'
'x-envoy-peer-metadata', '<redacted>=='
'x-envoy-peer-metadata-id', 'sidecar~10.60.0.40~auth-provisioner-6c6bf845cd-rhv68.iam~iam.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-b3-traceid', '9901ce1f2b2bc3ada67eed3942baf2a4'
'x-b3-spanid', 'a67eed3942baf2a4'
'x-b3-sampled', '0'
'x-forwarded-proto', 'https'
'transfer-encoding', 'chunked'Just looking at the first headers prefixed with
:s
steep-lamp-91158
03/14/2022, 9:25 PM
at that point, yes
h
happy-morning-85531
03/14/2022, 9:46 PMStarting to think that this has something to with cmux together with istio
port-forwarding to the pod and using localhost together with a grpcui tool works fine
cmux does matching on connection level while istio can multiplex requests both http2 and grpc over the same connection
2 Views