https://www.ory.sh/ logo
h

happy-morning-85531

03/14/2022, 6:52 PM
Anyone know why im getting 404 on the keto protobuf write API calls? Attaching logs in thread
Copy code
ime=2022-03-14T18:44:53Z level=info msg=started handling request http_request=map[headers:map[content-type:application/grpc grpc-accept-encoding:gzip te:trailers user-agent:grpc-java-netty/1.44.0 x-b3-parentspanid:52145d659ec72f10 x-b3-sampled:0 x-b3-spanid:3f331637fd61f3ce x-b3-traceid:fc7685fc00116c1c52145d659ec72f10 x-envoy-attempt-count:1 x-forwarded-client-cert:By=<spiffe://cluster.local/ns/ory/sa/ory-ksa;Hash=a814987f150b3d35acccfdee792e9b30c2ad4055e450b358402ed5d98f6a1714;Subject=>"";URI=<spiffe://cluster.local/ns/iam/sa/default> x-forwarded-proto:http x-request-id:c07a215e-7c89-4755-9b17-3ba7ab95deb7] host:keto-write.ory:80 method:POST path:/ory.keto.acl.v1alpha1.WriteService/TransactRelationTuples query:<nil> remote:127.0.0.6:51519 scheme:http]
time=2022-03-14T18:44:53Z level=info msg=completed handling request http_request=map[headers:map[content-type:application/grpc grpc-accept-encoding:gzip te:trailers user-agent:grpc-java-netty/1.44.0 x-b3-parentspanid:52145d659ec72f10 x-b3-sampled:0 x-b3-spanid:3f331637fd61f3ce x-b3-traceid:fc7685fc00116c1c52145d659ec72f10 x-envoy-attempt-count:1 x-forwarded-client-cert:By=<spiffe://cluster.local/ns/ory/sa/ory-ksa;Hash=a814987f150b3d35acccfdee792e9b30c2ad4055e450b358402ed5d98f6a1714;Subject=>"";URI=<spiffe://cluster.local/ns/iam/sa/default> x-forwarded-proto:http x-request-id:c07a215e-7c89-4755-9b17-3ba7ab95deb7] host:keto-write.ory:80 method:POST path:/ory.keto.acl.v1alpha1.WriteService/TransactRelationTuples query:<nil> remote:127.0.0.6:51519 scheme:http] http_response=map[headers:map[content-type:text/plain; charset=utf-8 x-content-type-options:nosniff] size:19 status:404 text_status:Not Found took:205.347µs]
s

steep-lamp-91158

03/14/2022, 7:36 PM
We are multiplexing the REST and gRPC servers on the same port That log statement looks like it comes from the REST handler 🤔 so the multiplexing is not working.... What client and ingress are you using?
h

happy-morning-85531

03/14/2022, 7:45 PM
I'm on kubernetes just using the plain service and it's a internal call between pods so no need for an ingress
s

steep-lamp-91158

03/14/2022, 7:56 PM
Was just wondering as there is envoy and forwarded headers and stuff
h

happy-morning-85531

03/14/2022, 7:57 PM
Istio service mesh so it's being applied by sidecar/proxy
s

steep-lamp-91158

03/14/2022, 7:57 PM
Maybe the request is send over HTTP1?
h

happy-morning-85531

03/14/2022, 7:58 PM
It's a grpc client used in spring boot so sounds weird that it would be sending it using the wrong protocol (version)
But it's something happening in the mesh, integration tests works locally with docker compose works fine together er with this application
s

steep-lamp-91158

03/14/2022, 8:03 PM
Ok then I have no idea 😅 I can only tell by the logs that the request is wrongly handled by the REST server
h

happy-morning-85531

03/14/2022, 8:05 PM
Its weird, since the criteria seems to be on the content-type header but from the logs the header is there
s

steep-lamp-91158

03/14/2022, 8:05 PM
It's the header and HTTP2
h

happy-morning-85531

03/14/2022, 8:06 PM
ah, illl snoop around and see if theres some rewrites happening
Istio is the culprit https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/ The ports in the helm chart are called
http-write|read
or maybe not, creating another service and specifying grpc appProtocol didnt work either
s

steep-lamp-91158

03/14/2022, 8:57 PM
ah I remember some issues with naming the ports
h

happy-morning-85531

03/14/2022, 9:17 PM
im debugging envoy and i can see the headers
Copy code
2022-03-14T21:12:30.340519Z	debug	envoy http	[C4577][S10006260048823567630] request headers complete (end_stream=false):
':authority', 'keto-write-grpc.ory.svc.cluster.local:4467'
':path', '/ory.keto.acl.v1alpha1.WriteService/TransactRelationTuples'
':method', 'POST'
'content-type', 'application/grpc'
'te', 'trailers'
'user-agent', 'grpc-java-netty/1.44.0'
'grpc-accept-encoding', 'gzip'
'x-request-id', 'ef779e17-846f-4c8e-9d1c-90b8b6a68061'
'x-envoy-decorator-operation', 'keto-write.ory.svc.cluster.local:4467/*'
'x-envoy-peer-metadata', '<redacted>=='
'x-envoy-peer-metadata-id', 'sidecar~10.60.0.40~auth-provisioner-6c6bf845cd-rhv68.iam~iam.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-b3-traceid', '9901ce1f2b2bc3ada67eed3942baf2a4'
'x-b3-spanid', 'a67eed3942baf2a4'
'x-b3-sampled', '0'
'x-forwarded-proto', 'https'
'transfer-encoding', 'chunked'
Just looking at the first headers prefixed with
:
which seems to be http2 headers?
s

steep-lamp-91158

03/14/2022, 9:25 PM
at that point, yes
h

happy-morning-85531

03/14/2022, 9:46 PM
Starting to think that this has something to with cmux together with istio
port-forwarding to the pod and using localhost together with a grpcui tool works fine
cmux does matching on connection level while istio can multiplex requests both http2 and grpc over the same connection
2 Views