https://www.ory.sh/ logo
Join Slack
Powered by
Are there any issues with the keto cli tool connec...
# talk-keto
c
Are there any issues with the keto cli tool connecting to self-hosted instances behind an ingress? We have an instance deployed on k8s using your helm charts and external access with the cli tool fails with: 
Error doing the request: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/plain; charset=utf-8"
The command used is: 
keto --read-remote <http://keto.ory.orgname.com:443|keto.ory.orgname.com:443> --write-remote <http://keto-admin.ory.orgname.com:443|keto-admin.ory.orgname.com:443> relation-tuple create keto-data/test.json
, the contents of 
test.json
is:
Copy code
{
  "namespace": "groups",
  "object": "admin",
  "relation": "member",
  "subject_id": "test"
}
The only real change to the helm chart is we use an ingress (nginx-ingress) and we have tls enabled (with cert-manager annotations). No problems connecting if I port-forward into the node or if I send REST API calls. I suspect something with the nginx ingress is interfering with the grpc/rest content detection, any help appreciated even if it's just pointing me to an outstanding issue (not sure if https://github.com/ory/keto/issues/807 is related, closed but waiting on https://github.com/ory/keto/issues/1091).
The errors from the pod through the ingress are: 
time=2022-11-01T15:48:37Z level=info msg=started handling request http_request=map[headers:map[content-length:34 content-type:application/grpc user-agent:grpc-go/1.49.0 x-forwarded-for:93.89.134.132 x-forwarded-host:<http://keto-admin.ory.orgname.com:443|keto-admin.ory.orgname.com:443> x-forwarded-port:443 x-forwarded-proto:https x-forwarded-scheme:https x-real-ip:93.89.134.132 x-request-id:c721d69662e88f346cf98616a570c979 x-scheme:https] host:<http://keto-admin.ory.orgname.com:443|keto-admin.ory.orgname.com:443> method:POST path:/ory.keto.relation_tuples.v1alpha2.WriteService/TransactRelationTuples query:<nil> remote:10.42.2.2:39038 scheme:http]
time=2022-11-01T15:48:37Z level=info msg=completed handling request http_request=map[headers:map[content-length:34 content-type:application/grpc user-agent:grpc-go/1.49.0 x-forwarded-for:93.89.134.132 x-forwarded-host:<http://keto-admin.ory.orgname.com:443|keto-admin.ory.orgname.com:443> x-forwarded-port:443 x-forwarded-proto:https x-forwarded-scheme:https x-real-ip:93.89.134.132 x-request-id:c721d69662e88f346cf98616a570c979 x-scheme:https] host:<http://keto-admin.ory.orgname.com:443|keto-admin.ory.orgname.com:443> method:POST path:/ory.keto.relation_tuples.v1alpha2.WriteService/TransactRelationTuples query:<nil> remote:10.42.2.2:39038 scheme:http] http_response=map[headers:map[content-type:text/plain; charset=utf-8 x-content-type-options:nosniff] size:19 status:404 text_status:Not Found took:388.047µs]
I had a typo in my json causing the error with the forwarded connections. I've edited my post to remove that and the issues caused by it. The original problem still remains. No joy using the CLI through nginx-ingress but works fine over port-forward (and REST API calls work fine as well).
s
The multiplex first matches on http/2. Make sure nginx allows doing that.
c
nginx-ingress defaults to allow http2, but it does do TLS termination... and I know TLS is a requirement of http/2 so not sure how that plays with keto
s
Maybe try to verify what exactly nginx forwards
c
Thanks, I'll dig into it tomorrow, I suspect something is happening to the content-type header as it looks like that is what 
internal/driver/daemon.go
uses to identify gRPC vs REST
s
c
Thank-you @steep-lamp-91158, I now have gRPC working by adding this annotation to the ingress: 
<http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: 'GRPC'
however this then breaks REST I'll see if I can get both working as I have now proven both can be made to work independently
s
yeah there are some problems with this multiplexing.... see also https://github.com/ory/keto/issues/854
c
In that issue it mentions multiplexing will be dropped -- what's the plan to replace it, separate paths/ports for gRPC and REST?
s
we are investigating to use https://github.com/bufbuild/connect-go instead of grpc, which is grpc wire compatible, so all clients would still work the question is how rest would look like using that
c
If you're changing it might not be relevant, not sure on your timescales, but it looks like if we split into 2 ingresses based on path we could have both gRPC and REST on the same URL: https://github.com/kubernetes/ingress-nginx/issues/2492
s
that needs two ports right?
we want to have a POC this year, not sure when exactly it will land on master
c
I think it can work with one, I'll try and get a PoC up
Got it working
This is what the 
ingress
section of 
values.yaml
for the Helm chart looks like:
Copy code
ingress:
  read:
    annotations:
      <http://cert-manager.io/cluster-issuer|cert-manager.io/cluster-issuer>: letsencrypt-production
      <http://kubernetes.io/tls-acme|kubernetes.io/tls-acme>: 'true'
      <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
      <http://nginx.ingress.kubernetes.io/use-regex|nginx.ingress.kubernetes.io/use-regex>: 'true'
    enabled: true
    hosts:
      - host: <http://keto.ory.orgname.com|keto.ory.orgname.com>
        paths:
          - path: /ory.*/
            pathType: Prefix
    tls:
      - hosts:
          - <http://keto.ory.orgname.com|keto.ory.orgname.com>
        secretName: keto-ingress-cert
  write:
    annotations:
      <http://cert-manager.io/cluster-issuer|cert-manager.io/cluster-issuer>: letsencrypt-production
      <http://kubernetes.io/tls-acme|kubernetes.io/tls-acme>: 'true'
      <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: GRPC
      <http://nginx.ingress.kubernetes.io/use-regex|nginx.ingress.kubernetes.io/use-regex>: 'true'
    enabled: true
    hosts:
      - host: <http://keto-admin.ory.orgname.com|keto-admin.ory.orgname.com>
        paths:
          - path: /ory.*/
            pathType: Prefix
    tls:
      - hosts:
          - <http://keto-admin.ory.orgname.com|keto-admin.ory.orgname.com>
        secretName: keto-admin-ingress-cert
In addition 2 extra ingress were added that look like this: For the read:
Copy code
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  annotations:
    <http://cert-manager.io/cluster-issuer|cert-manager.io/cluster-issuer>: letsencrypt-production
    <http://kubernetes.io/tls-acme|kubernetes.io/tls-acme>: "true"
  name: keto-read-http
  namespace: ory
spec:
  ingressClassName: nginx
  rules:
  - host: <http://keto.ory.orgname.com|keto.ory.orgname.com>
    http:
      paths:
      - backend:
          service:
            name: keto-read
            port:
              name: grpc-read
        path: /relation-tuples/
        pathType: Prefix
  tls:
  - hosts:
    - <http://keto.ory.orgname.com|keto.ory.orgname.com>
    secretName: keto-ingress-cert
For the write:
Copy code
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  annotations:
    <http://cert-manager.io/cluster-issuer|cert-manager.io/cluster-issuer>: letsencrypt-production
    <http://kubernetes.io/tls-acme|kubernetes.io/tls-acme>: "true"
  name: keto-write-http
  namespace: ory
spec:
  ingressClassName: nginx
  rules:
  - host: <http://keto-admin.ory.orgname.com|keto-admin.ory.orgname.com>
    http:
      paths:
      - backend:
          service:
            name: keto-write
            port:
              name: grpc-write
        path: /admin/
        pathType: Prefix
  tls:
  - hosts:
    - <http://keto-admin.ory.orgname.com|keto-admin.ory.orgname.com>
    secretName: keto-ingress-cert
This sets up 2 ingress for each of read/write (because the 'backend-protocol: GRPC' flag can only be set for the whole ingress) and then handles each based on path. For gRPC any request for the path /ory.*/ goes to the respective read/write endpoint with the gRPC flag enabled. For read any request to /relation-tuples/ goes to the read endpoint and for write any request to /admin/ goes to the write endpoint; in both cases without the gRPC flag enabled.
This assumes an nginx ingress, there are other ingress controllers out there so ymmv
23 Views
Open in Slack
PreviousNext
Powered by