https://www.ory.sh/ logo
Join Slack
Powered by
Anyone know why im getting 404 on the keto protobu...
# talk-keto
h
Anyone know why im getting 404 on the keto protobuf write API calls? Attaching logs in thread
Copy code
ime=2022-03-14T18:44:53Z level=info msg=started handling request http_request=map[headers:map[content-type:application/grpc grpc-accept-encoding:gzip te:trailers user-agent:grpc-java-netty/1.44.0 x-b3-parentspanid:52145d659ec72f10 x-b3-sampled:0 x-b3-spanid:3f331637fd61f3ce x-b3-traceid:fc7685fc00116c1c52145d659ec72f10 x-envoy-attempt-count:1 x-forwarded-client-cert:By=<spiffe://cluster.local/ns/ory/sa/ory-ksa;Hash=a814987f150b3d35acccfdee792e9b30c2ad4055e450b358402ed5d98f6a1714;Subject=>"";URI=<spiffe://cluster.local/ns/iam/sa/default> x-forwarded-proto:http x-request-id:c07a215e-7c89-4755-9b17-3ba7ab95deb7] host:keto-write.ory:80 method:POST path:/ory.keto.acl.v1alpha1.WriteService/TransactRelationTuples query:<nil> remote:127.0.0.6:51519 scheme:http]
time=2022-03-14T18:44:53Z level=info msg=completed handling request http_request=map[headers:map[content-type:application/grpc grpc-accept-encoding:gzip te:trailers user-agent:grpc-java-netty/1.44.0 x-b3-parentspanid:52145d659ec72f10 x-b3-sampled:0 x-b3-spanid:3f331637fd61f3ce x-b3-traceid:fc7685fc00116c1c52145d659ec72f10 x-envoy-attempt-count:1 x-forwarded-client-cert:By=<spiffe://cluster.local/ns/ory/sa/ory-ksa;Hash=a814987f150b3d35acccfdee792e9b30c2ad4055e450b358402ed5d98f6a1714;Subject=>"";URI=<spiffe://cluster.local/ns/iam/sa/default> x-forwarded-proto:http x-request-id:c07a215e-7c89-4755-9b17-3ba7ab95deb7] host:keto-write.ory:80 method:POST path:/ory.keto.acl.v1alpha1.WriteService/TransactRelationTuples query:<nil> remote:127.0.0.6:51519 scheme:http] http_response=map[headers:map[content-type:text/plain; charset=utf-8 x-content-type-options:nosniff] size:19 status:404 text_status:Not Found took:205.347µs]
s
We are multiplexing the REST and gRPC servers on the same port That log statement looks like it comes from the REST handler 🤔 so the multiplexing is not working.... What client and ingress are you using?
h
I'm on kubernetes just using the plain service and it's a internal call between pods so no need for an ingress
s
Was just wondering as there is envoy and forwarded headers and stuff
h
Istio service mesh so it's being applied by sidecar/proxy
s
Maybe the request is send over HTTP1?
h
It's a grpc client used in spring boot so sounds weird that it would be sending it using the wrong protocol (version)
But it's something happening in the mesh, integration tests works locally with docker compose works fine together er with this application
s
Ok then I have no idea 😅 I can only tell by the logs that the request is wrongly handled by the REST server
h
Its weird, since the criteria seems to be on the content-type header but from the logs the header is there
s
It's the header and HTTP2
h
ah, illl snoop around and see if theres some rewrites happening
Istio is the culprit https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/ The ports in the helm chart are called 
http-write|read
or maybe not, creating another service and specifying grpc appProtocol didnt work either
s
ah I remember some issues with naming the ports
h
im debugging envoy and i can see the headers
Copy code
2022-03-14T21:12:30.340519Z	debug	envoy http	[C4577][S10006260048823567630] request headers complete (end_stream=false):
':authority', 'keto-write-grpc.ory.svc.cluster.local:4467'
':path', '/ory.keto.acl.v1alpha1.WriteService/TransactRelationTuples'
':method', 'POST'
'content-type', 'application/grpc'
'te', 'trailers'
'user-agent', 'grpc-java-netty/1.44.0'
'grpc-accept-encoding', 'gzip'
'x-request-id', 'ef779e17-846f-4c8e-9d1c-90b8b6a68061'
'x-envoy-decorator-operation', 'keto-write.ory.svc.cluster.local:4467/*'
'x-envoy-peer-metadata', '<redacted>=='
'x-envoy-peer-metadata-id', 'sidecar~10.60.0.40~auth-provisioner-6c6bf845cd-rhv68.iam~iam.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-b3-traceid', '9901ce1f2b2bc3ada67eed3942baf2a4'
'x-b3-spanid', 'a67eed3942baf2a4'
'x-b3-sampled', '0'
'x-forwarded-proto', 'https'
'transfer-encoding', 'chunked'
Just looking at the first headers prefixed with 
:
which seems to be http2 headers?
s
at that point, yes
h
Starting to think that this has something to with cmux together with istio
port-forwarding to the pod and using localhost together with a grpcui tool works fine
cmux does matching on connection level while istio can multiplex requests both http2 and grpc over the same connection
3 Views
Open in Slack
PreviousNext
Powered by