https://www.ory.sh/ logo
Powered by
h

happy-morning-85531

02/23/2022, 12:25 PM
Hi, anyone who have some time over and could help me debug why I cant get my deployment of kratos together with the selfserve-node-ui on kubernetes (exposed through istio) to work?
I currently get stuck in a endless loop of redirects
i

incalculable-city-41629

02/23/2022, 12:46 PM
Hi Maybe an issue within the configuration of Kratos Sometimes that happens when the front url part isn't correct
First what do you get without istio?
h

happy-morning-85531

02/23/2022, 12:48 PM
I havent tried without istio, its used to expose our public services together with cert-manager so im not sure i can cut that part out right now (even for testing)
i

incalculable-city-41629

02/23/2022, 1:59 PM
Ok
h

happy-morning-85531

02/23/2022, 2:00 PM
It seems to be related to the browser part only, as using the api of kratis works fine
i

incalculable-city-41629

02/23/2022, 2:00 PM
Can you share your Kratos config ?
h

happy-morning-85531

02/23/2022, 2:02 PM
Copy code
config:
      courier:
        smtp: {}
        #template_override_path: /conf/courier-templates

      identity:
        default_schema_url: file:///etc/config/identity.company_employees.schema.json

      serve:
        public:
          base_url: <http://kratos-public:80>
          port: 4433
        admin:
          base_url: <http://kratos-admin:80>
          port: 4434

      selfservice:
        default_browser_return_url: <https://dashboard.dev.company.com/>
        whitelisted_return_urls:
          - <https://dashboard.dev.company.com>

        methods:
          password:
            enabled: true

        flows:
          error:
            ui_url: <https://dashboard.dev.company.com/error>
          settings:
            ui_url: <https://dashboard.dev.company.com/settings>
            privileged_session_max_age: 15m
          recovery:
            enabled: true
            ui_url: <https://dashboard.dev.company.com/recovery>
          verification:
            enabled: true
            ui_url: <https://dashboard.dev.company.com/verification>
            after:
              default_browser_return_url: <https://dashboard.dev.company.com/>
          logout:
            after:
              default_browser_return_url: <https://dashboard.dev.company.com/login>
          login:
            ui_url: <https://dashboard.dev.company.com/login>
            lifespan: 10m
          registration:
            lifespan: 10m
            ui_url: <https://dashboard.dev.company.com/registration>
            after:
              password:
                hooks:
                  - hook: session

      secrets: {}
note that this is using the helm chart so its creating an identity schema using a config map, i used one from the quick starts
i

incalculable-city-41629

02/23/2022, 2:14 PM
do we agree that the dashboard url is redirect to self-ui ?
h

happy-morning-85531

02/23/2022, 2:15 PM
i

incalculable-city-41629

02/23/2022, 2:17 PM
ok I 👀
I did not see anything wrong
do you use istio also as sidecar ?
h

happy-morning-85531

02/23/2022, 3:02 PM
Yes
The issue seems only to be related with the browser "api" of kratos, as curling non browser endpoints work fine
Copy code
flowId=$(curl -s -X GET -H "Accept: application/json"  <https://accounts.dev.example.com/self-service/registration/api> | jq -r '.id')
and
Copy code
curl -s -X GET -H "Accept: application/json" <https://accounts.dev.example.com/self-service/registration/flows>\?id\=$flowId  | jq
but 
<https://accounts.dev.example.com/self-service/login/browser?aal=&refresh=&return_to=>
returns with endless redirect
although the curls are using the registration and not the login
but 
<https://accounts.dev.company.se/self-service/registration/browser?return_to=>
also results in endless redirect
But according to ory.sh/kratos-knative-demo
• ORY Kratos and the application need to share the same DNS-name in order to be able to communicate with cookies. That's not how things are working out-of-the-box in Knative where each service gets its own DNS-name.
Maybe this is the issue?
🤔 1
I do see an error in logs
Copy code
time=2022-02-23T15:12:00Z level=error msg=An error occurred while handling a request audience=application error=map[debug: details:map[docs:<https://www.ory.sh/kratos/docs/debug/csrf> hint:The anti-CSRF cookie was found but the CSRF token was not included in the HTTP request body (csrf_token) nor in the HTTP Header (X-CSRF-Token). reject_reason:The HTTP Cookie Header was set and a CSRF token was sent but they do not match. We recommend deleting all cookies for this domain and retrying the flow.] message:the request was rejected to protect you from Cross-Site-Request-Forgery reason:The request was rejected to protect you from Cross-Site-Request-Forgery (CSRF) which could cause account takeover, leaking personal information, and other serious security issues. status:Forbidden status_code:403] http_request=map[headers:map[accept:application/json, text/plain, */* user-agent:axios/0.21.4 x-b3-parentspanid:07a74f5993647eef x-b3-sampled:0 x-b3-spanid:58f6bfef170f377c x-b3-traceid:1689ef216248e17a07a74f5993647eef x-envoy-attempt-count:1 x-forwarded-client-cert:By=<spiffe://cluster.local/ns/ory/sa/kratos-ksa;Hash=5e07920ae90f22a5065e91e2f81ee4bc04cde28c0b8d753ed5f6669ec6571dd3;Subject=>"";URI=<spiffe://cluster.local/ns/ory/sa/default> x-forwarded-proto:http x-request-id:5dac89b8-6b41-4c62-8dd1-e4081fd5792c] host:kratos-public method:GET path:/self-service/registration/flows query:Value is sensitive and has been redacted. To see the value set config key "log.leak_sensitive_values = true" or environment variable "LOG_LEAK_SENSITIVE_VALUES=true". remote:127.0.0.6:60649 scheme:http] http_response=map[status_code:403] service_name=Ory Kratos service_version=v0.8.0-alpha.3
i

incalculable-city-41629

02/23/2022, 4:06 PM
weird
can you try to add this section in your config :
Copy code
session:
  cookie:
    domain: <your.domain>
    persistent: true
h

happy-morning-85531

02/23/2022, 4:08 PM
i added it already, but without the persistent flag, ill try with it
domain is the TLD right?
i

incalculable-city-41629

02/23/2022, 4:09 PM
yes
h

happy-morning-85531

02/23/2022, 4:10 PM
i just verified that istio is not stripping any headers on the public urls, so that shouldnt be the problem
i

incalculable-city-41629

02/23/2022, 4:10 PM
ok
h

happy-morning-85531

02/23/2022, 4:11 PM
still redirecting
i

incalculable-city-41629

02/23/2022, 4:11 PM
ok
what happens if you set that in the config
Copy code
serve:
    public:
      cors:
        enabled: false
btw I not sure about that :
Copy code
base_url: <http://kratos-public:80>
      port: 4433
the port part does not sound good to me can you try
Copy code
base_url: <http://kratos-public:4433>
h

happy-morning-85531

02/23/2022, 4:16 PM
it should use the kratos-public service which only exposes port 80
i

incalculable-city-41629

02/23/2022, 4:16 PM
oh
ok
h

happy-morning-85531

02/23/2022, 4:16 PM
Copy code
spec:

  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http-public
i

incalculable-city-41629

02/23/2022, 4:16 PM
so you can remove the port part in both
get it
h

happy-morning-85531

02/23/2022, 4:18 PM
to just
Copy code
serve:
        public:
          base_url: <http://kratos-public>
        admin:
          base_url: <http://kratos-admin>
?
i

incalculable-city-41629

02/23/2022, 4:18 PM
yes
h

happy-morning-85531

02/23/2022, 4:19 PM
still redirecting 😕
i

incalculable-city-41629

02/23/2022, 4:19 PM
ok
even with cors disable ?
h

happy-morning-85531

02/23/2022, 4:20 PM
ill apply it again
i

incalculable-city-41629

02/23/2022, 4:20 PM
ok can you get the tag of both images ui and kratos ?
h

happy-morning-85531

02/23/2022, 4:21 PM
sure
Copy code
image:
    repository: oryd/kratos
    tag: v0.8.0-alpha.3
and
Copy code
image: oryd/kratos-selfservice-ui-node:v0.8.0-alpha.3
i

incalculable-city-41629

02/23/2022, 4:22 PM
ok
in the service can you use the port number instead of the name
as istio do some stuff undergroud maybe
h

happy-morning-85531

02/23/2022, 4:23 PM
I guess ican create an additional service and point to that instead
i

incalculable-city-41629

02/23/2022, 4:23 PM
yes
h

happy-morning-85531

02/23/2022, 4:23 PM
since its using your helm charts im not able to change that unless i switch to kustomize but thats going to be a hassle
im just going to explicitly try to set port 80 in the serve config
since the default of the helm chart is 4433 and 4434
i

incalculable-city-41629

02/23/2022, 4:24 PM
ok
h

happy-morning-85531

02/23/2022, 4:26 PM
ok that doesnt work obv
Copy code
time=2022-02-23T16:25:37Z level=warning msg=The config has no version specified. Add the version to improve your development experience. audience=application service_name=Ory Kratos service_version=v0.8.0-alpha.3
time=2022-02-23T16:25:37Z level=info msg=Software quality assurance features are enabled. Learn more at: <https://www.ory.sh/docs/ecosystem/sqa> audience=application service_name=Ory Kratos service_version=v0.8.0-alpha.3
time=2022-02-23T16:25:37Z level=info msg=TLS has not been configured for admin, skipping audience=application service_name=Ory Kratos service_version=v0.8.0-alpha.3
time=2022-02-23T16:25:37Z level=info msg=Starting the admin httpd on: 0.0.0.0:80 audience=application service_name=Ory Kratos service_version=v0.8.0-alpha.3
time=2022-02-23T16:25:37Z level=info msg=TLS has not been configured for public, skipping audience=application service_name=Ory Kratos service_version=v0.8.0-alpha.3
time=2022-02-23T16:25:37Z level=info msg=Starting the public httpd on: 0.0.0.0:80 audience=application service_name=Ory Kratos service_version=v0.8.0-alpha.3
time=2022-02-23T16:25:37Z level=fatal msg=Failed to gracefully shutdown admin httpd: listen tcp 0.0.0.0:80: bind: permission denied audience=application service_name=Ory Kratos service_version=v0.8.0-alpha.3
its the same process, facepalm
i

incalculable-city-41629

02/23/2022, 4:27 PM
You have two kind of instance 1. istio (ingress) 2. istion (sidecar) Right ? I guess you can disable the 2nd one ?
h

happy-morning-85531

02/23/2022, 4:27 PM
so i should add seperate service for both admin and public
correct
i

incalculable-city-41629

02/23/2022, 4:28 PM
let’s try that first
h

happy-morning-85531

02/23/2022, 4:30 PM
applying
Copy code
apiVersion: v1
kind: Service
metadata:
  name: kratos-admin-test
  namespace: ory
  labels:
    <http://app.kubernetes.io/component|app.kubernetes.io/component>: admin
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: kratos
    <http://helm.sh/chart|helm.sh/chart>: kratos-0.21.8
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: ory
    <http://app.kubernetes.io/version|app.kubernetes.io/version>: "v0.8.0-alpha.3"
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 4434
      protocol: TCP
      name: http
  selector:
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: kratos
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: ory
---
# Source: ory/charts/kratos/templates/service-public.yaml
apiVersion: v1
kind: Service
metadata:
  name: kratos-public-test
  namespace: ory
  labels:
    <http://app.kubernetes.io/component|app.kubernetes.io/component>: public
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: kratos
    <http://helm.sh/chart|helm.sh/chart>: kratos-0.21.8
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: ory
    <http://app.kubernetes.io/version|app.kubernetes.io/version>: "v0.8.0-alpha.3"
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 4433
      protocol: TCP
      name: http
  selector:
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: kratos
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: ory
i

incalculable-city-41629

02/23/2022, 4:32 PM
not sure on target port according to the log it seems both are listening on 80
h

happy-morning-85531

02/23/2022, 4:32 PM
that was when i tried setting port to 80 explicitly
i

incalculable-city-41629

02/23/2022, 4:32 PM
ok
h

happy-morning-85531

02/23/2022, 4:32 PM
now its as default
Copy code
audience=application service_name=Ory Kratos service_version=v0.8.0-alpha.3
time=2022-02-23T16:32:30Z level=warning msg=The config has no version specified. Add the version to improve your development experience. audience=application service_name=Ory Kratos service_version=v0.8.0-alpha.3
time=2022-02-23T16:32:30Z level=info msg=Software quality assurance features are enabled. Learn more at: <https://www.ory.sh/docs/ecosystem/sqa> audience=application service_name=Ory Kratos service_version=v0.8.0-alpha.3
time=2022-02-23T16:32:30Z level=info msg=TLS has not been configured for admin, skipping audience=application service_name=Ory Kratos service_version=v0.8.0-alpha.3
time=2022-02-23T16:32:30Z level=info msg=Starting the admin httpd on: 0.0.0.0:4434 audience=application service_name=Ory Kratos service_version=v0.8.0-alpha.3
time=2022-02-23T16:32:30Z level=info msg=TLS has not been configured for public, skipping audience=application service_name=Ory Kratos service_version=v0.8.0-alpha.3
time=2022-02-23T16:32:30Z level=info msg=Starting the public httpd on: 0.0.0.0:4433 audience=application service_name=Ory Kratos service_version=v0.8.0-alpha.3
i

incalculable-city-41629

02/23/2022, 4:32 PM
👍
h

happy-morning-85531

02/23/2022, 4:33 PM
no dice with using other svcs
ill try to remove the sidecar
i

incalculable-city-41629

02/23/2022, 4:33 PM
ok
h

happy-morning-85531

02/23/2022, 4:34 PM
sidecar removed still redirecting 😢
i

incalculable-city-41629

02/23/2022, 4:35 PM
pfff
h

happy-morning-85531

02/23/2022, 4:36 PM
i very much appreciate the help though 🙂
i

incalculable-city-41629

02/23/2022, 4:37 PM
no problem
last chance
base_url:
set the external fqdn
h

happy-morning-85531

02/23/2022, 4:39 PM
for the public one i assume?
i

incalculable-city-41629

02/23/2022, 4:39 PM
yes
in the whitelist add internal address too
h

happy-morning-85531

02/23/2022, 4:41 PM
i just changed to publicc fqdn still redirecting
i

incalculable-city-41629

02/23/2022, 4:41 PM
ok
🤷🏽‍♂️
h

happy-morning-85531

02/23/2022, 4:42 PM
Ill add the the urls but ive seen errors when you try to redirect to a non whitelisted one
so doubt that that is the problem but i can try
i

incalculable-city-41629

02/23/2022, 4:43 PM
ok
h

happy-morning-85531

02/23/2022, 4:45 PM
oh well seems like we've tried most things 😕
just bouncing between 
<https://accounts.dev.example.com/self-service/login/browser?aal=&refresh=&return_to=>
and 
<https://dashboard.dev.example.com/login?flow=154b99ba-32ae-4fbc-9e96-5b40fe409bb2>
maybe the problem is with the selfserve ui which redirects back?
I got it to work... I just uncommented the "base_url" in values.yaml part of the selfservice-ui-node chart and the redirects disappeared ..
i

incalculable-city-41629

02/23/2022, 6:01 PM
lol
h

happy-morning-85531

02/23/2022, 6:02 PM
the base url was set to the selfservice url before ..
this is a real head in desk situation
2 Views
Powered by