https://www.ory.sh/ logo
Join SlackCommunities
Powered by
does anyone know of an online tool similar to rege...
# talk-oathkeeper
c
does anyone know of an online tool similar to regex101 for the glob package used in oathkeeper ? (is it github.com/gobwas/glob) ? I've tried https://www.digitalocean.com/community/tools/glob but a pattern than works there does not work in oathkeeper ({"error":{"code":404,"status":"Not Found","message":"Requested url does not match any rules"}})
e
That tool should work, just don't include 
<>
in the tool, but do include it in your config
c
wait .. what is <> in the config ? 😉
where should <> go ?
e
beginning and end of the glob i believe.
eg:
Copy code
<https://domain.com/><**.css>
Would match all css files etc
c
oh, ok. thanks. will give that a try
e
👍 Let me know how it goes!
c
doesn't work 😞
Copy code
"match": {
      "url": "https://<*>",
    
gives

{"error":{"code":404,"status":"Not Found","message":"Requested url does not match any rules"}}
trying with the first url
e
double **
c
that would match all 4
or do I have to double in oathkeeper
e
You want to match all 4?
c
no
I want to have a rule for the first two, and a rule each for the other 2
or a rule each for all 4 😉
e
Okay, so think for the first one: 
https://<*>
Second one: 
https://<*>/<**>
I think
c
https://<*>/<**>
seems to match the top 2
r
@curved-oxygen-35290 maybe you can retrace and tell people what you are trying to achieve? It seems the domain is the same?
e.g. then why bother matching on it
c
the domain is ephemeral - it will change every time I start a workspace. So I need to wildcard the domain. After that, I want "" and "/" to go to service A, and "/ui" to go to service B and "/everythingelse" to go to service C
r
ah, not familiar with gitpod
e
Move the second rule above the first
Then it'll only match the main domain if there is no /UI etc
r
ok, I would try this for service A:
Copy code
https://*.<http://gitpod.io|gitpod.io><,/>
service B:
Copy code
https://*.<http://gitpod.io/ui|gitpod.io/ui>
service C:
Copy code
https://*.<http://gitpod.io/<**|gitpod.io/<**>>
e
^ that works too
r
had a typo in A
hopefully not anymore 😄
also, if you run it with DEBUG or TRACE you should see what it's trying to match on. maybe DEBUG is enough, I don't remember. TRACE produces lots and lots of output which may be a bit hard to get through.
c
so, I have a match 
"url": "https://*.<http://gitpod.io|gitpod.io><,/>",
as the only rule but get 404 on either "" or "/"
r
What does the log say?
c
Copy code
ime=2023-02-19T14:56:01Z level=warning msg=Access request denied audience=application error=map[debug: message:Requested url does not match any rules reason: stack_trace:
<http://github.com/ory/oathkeeper/rule.(*RepositoryMemory).Match|github.com/ory/oathkeeper/rule.(*RepositoryMemory).Match>
        /project/rule/repository_memory.go:131
<http://github.com/ory/oathkeeper/proxy.(*Proxy).Director|github.com/ory/oathkeeper/proxy.(*Proxy).Director>
        /project/proxy/proxy.go:110
net/http/httputil.(*ReverseProxy).ServeHTTP
        /usr/local/go/src/net/http/httputil/reverseproxy.go:263
<http://github.com/urfave/negroni.Wrap.func1|github.com/urfave/negroni.Wrap.func1>
        /go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:46
<http://github.com/urfave/negroni.HandlerFunc.ServeHTTP|github.com/urfave/negroni.HandlerFunc.ServeHTTP>
        /go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:29
<http://github.com/urfave/negroni.middleware.ServeHTTP|github.com/urfave/negroni.middleware.ServeHTTP>
        /go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:38
<http://github.com/ory/x/corsx.ContextualizedMiddleware.func1|github.com/ory/x/corsx.ContextualizedMiddleware.func1>
        /go/pkg/mod/github.com/ory/x@v0.0.532/corsx/middleware.go:26
<http://github.com/urfave/negroni.HandlerFunc.ServeHTTP|github.com/urfave/negroni.HandlerFunc.ServeHTTP>
        /go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:29
<http://github.com/urfave/negroni.middleware.ServeHTTP|github.com/urfave/negroni.middleware.ServeHTTP>
        /go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:38
<http://github.com/ory/x/reqlog.(*Middleware).ServeHTTP|github.com/ory/x/reqlog.(*Middleware).ServeHTTP>
        /go/pkg/mod/github.com/ory/x@v0.0.532/reqlog/middleware.go:142
<http://github.com/urfave/negroni.middleware.ServeHTTP|github.com/urfave/negroni.middleware.ServeHTTP>
        /go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:38
<http://github.com/ory/oathkeeper/metrics.(*Middleware).ServeHTTP|github.com/ory/oathkeeper/metrics.(*Middleware).ServeHTTP>
I am using glob. Would it be better to use regex, or are there the same "special" patterns for oathkeeper ? 😉
e
What does your Oathkeeper config look like? Maybe it isn't actually loading the rules
c
Copy code
access_rules:
  matching_strategy: glob
  repositories:
    - file:///home/bbs/.config/rules.json
"url": "<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io><**>",
works
"url": "https://*.<http://gitpod.io|gitpod.io><**>",
does not
wth
e
The first * won't match
It's not in <>
c
was trying to follow this 😉
let me try with <>
e
Yeah didn't spot that
c
nope. 404 😞
e
Okay ummm
Try this
c
"url": "https://<*>.<http://gitpod.io|gitpod.io><**>",
btw - really appreciate the help 😉
e
https://<*.<http://gitpod.io/**|gitpod.io/**>>
Maybe it only evaluates one <>
c
"url": "https://<*.<http://gitpod.io/**|gitpod.io/**>>",
nope
e
What URL are you trying to match?
c
the same as above
"url": "https://<**.<http://gitpod.io/**|gitpod.io/**>>",
works 🤔
does the "." mess with the globbing ?
e
Nah shouldn't
Okay so: 
https://<*.<http://gitpod.io|gitpod.io>*>
Should match:
Copy code
<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io>
<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io/>
Then: 
https://<*.<http://gitpod.io/ui|gitpod.io/ui>>
Should match:
Copy code
<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io/ui>
c
in that order ?
both urls give 
Requested url does not match any rules
I'm starting to think something's borked with the matcher. It can't be this complicated 😉
e
Yeah, my yml file one works exactly as intended.
c
I'm going to switch to yml
will let you know 😉 thanks again for the help
e
So 
<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io>
doesn't match a rule?
c
nope
nor 
/
nor 
/ui
e
Does Oathkeeper error when loading the rules at all?
c
yeah it has that stack trace I showed earlier
e
No i mean, when you start oathkeeper up, is there any errors about parsing the config/rules?
c
sorry , no
e
All good. Okay, that's fine. It's super weird it's not matching.
Okay try: 
https://<**.<http://gitpod.io|gitpod.io>*>
Should match:
Copy code
<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io>
<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io/>
Then: 
https://<**.<http://gitpod.io/ui|gitpod.io/ui>>
Should match:
Copy code
<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io/ui>
Maybe it needs the double ** at the start
r
I would really like to have a rule validator in oathkeeper 😄
e
Okay, so i just tested with my dev setup & domain. I managed to get these to work:
c
now tearing my hair out 😢
Copy code
[
  
  {
    "id": "allow_all",
    "version": "v0.40.1",
    "upstream": {
      "url": "<http://umvikeli:4200>"
    },
    "match": {
      "url": "https://<**.gitpod.io*>",
      "methods": [
        "GET",
        "PUT",
        "POST",
        "DELETE",
        "PATCH"
      ]
    },
    "authenticators": [
      {
        "handler": "anonymous"
      }
    ],
    "authorizer": {
      "handler": "allow"
    },
    "mutators": [
      {
        "handler": "header",
        "config": {
          "headers": {
            "X-User": "{{ print .Subject }}"
          }
        }
      }
    ]
  }
,{
  "id": "allow_all",
  "version": "v0.40.1",
  "upstream": {
    "url": "<http://umvikeli:8081>",
    "strip_path": "ui"
  },
  "match": {
    "url": "https://<**.gitpod.io/ui>",
    "methods": [
      "GET",
      "PUT",
      "POST",
      "DELETE",
      "PATCH"
    ]
  },
  "authenticators": [
    {
      "handler": "anonymous"
    }
  ],
  "authorizer": {
    "handler": "allow"
  },
  "mutators": [
    {
      "handler": "header",
      "config": {
        "headers": {
          "X-User": "{{ print .Subject }}"
        }
      }
    }
  ]
}

]
no url matches. every one is a 404
no errors in the logs to start up with
Copy code
time=2023-02-19T16:24:48Z level=debug msg=Viper detected a configuration change, updating matching strategy audience=application event=matching_strategy_config_change service_name=ORY Oathkeeper service_version=v0.40.1 source=entrypoint
time=2023-02-19T16:24:48Z level=debug msg=One or more access rule repositories changed, reloading access rules. audience=application event=repository_change file=file:///home/bbs/.config/rules.json service_name=ORY Oathkeeper service_version=v0.40.1 source=config_update
time=2023-02-19T16:24:48Z level=debug msg=Fetching access rules from given location because something changed. audience=application location=file:///home/bbs/.config/rules.json service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-02-19T16:24:48Z level=info msg=Software quality assurance features are enabled. Learn more at: <https://www.ory.sh/docs/ecosystem/sqa> audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-02-19T16:24:48Z level=info msg=TLS has not been configured for api, skipping audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-02-19T16:24:48Z level=info msg=Listening on http://:3000 audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-02-19T16:24:48Z level=info msg=TLS has not been configured for proxy, skipping audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-02-19T16:24:48Z level=info msg=Listening on http://:8080 audience=application service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-02-19T16:24:48Z level=info msg=Listening on http://:9000 audience=application service_name=ORY Oathkeeper service_version=v0.40.1
last hope is to convert to yml 🤣
e
Okay, what URL are you visiting & what is the subsequent error message
c
ok. this config
Copy code
- id: "api:protected"
  upstream:
    preserve_host: false
    url: "<http://umvikeli:4200>"
  match:
    url: "https://<**.gitpod.io*>"
    methods:
      - GET
      - DELETE
      - PATCH
      - POST
      - PUT
  authenticators:
    - handler: anonymous
  mutators:
    - handler: header
      config:
        headers:
          X-User: "{{ print .Subject }}"
  authorizer:
    handler: allow
(yes, it's loaded ... 
time=2023-02-19T16:34:15Z level=debug msg=One or more access rule repositories changed, reloading access rules. audience=application event=repository_change file=file:///home/bbs/.config/access-rules.yml service_name=ORY Oathkeeper service_version=v0.40.1 source=config_update
this url 
<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io>
Copy code
{
  "error": {
    "code": 404,
    "status": "Not Found",
    "message": "Requested url does not match any rules"
  }
}
wait a moment - does this pattern handle query params ?
I've just seen the network tab in devtools has 
<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io/?vscodeBrowserReqId=xxxx>
translating the above .. should be 
https://<*.<http://gitpod.io/**|gitpod.io/**>>
?
e
Yeah, but you'll need your UI one above it in that case Before: 
https://*.<http://domain.com/ui/**|domain.com/ui/**>
After: 
https://*.<http://domain.com/**|domain.com/**>
c
I'm trying to get one to work first 🤣
e
Then yeah, try that one
c
just in case you don't beleive me 😉
e
Copy code
preserve_query: true
maybe?
c
nope
wait - I don't think the service restarted
nope. it had
Copy code
/usr/local/go/src/runtime/asm_amd64.s:1594 status:Not Found status_code:404] granted=false http_host=<http://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io|8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io> http_method=GET http_url=<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io/?vscodeBrowserReqId=1676825263176> http_user_agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 service_name=ORY Oathkeeper service_version=v0.40.1
time=2023-02-19T16:47:43Z level=info msg=An error occurred while handling a request audience=application error=map[debug: message:Requested url does not match any rules reason: stack_trace:
Copy code
access_rules:
  matching_strategy: glob
waaait a moment
Copy code
[{"id":"api:protected","version":"v0.40.1","description":"","match":{"methods":["GET","DELETE","PATCH","POST","PUT"],"url":"https://\u003c*.<http://gitpod.io/**\u003e%22},%22authenticators%22:[{%22handler%22:%22anonymous%22,%22config%22:null}],%22authorizer%22:{%22handler%22:%22allow%22,%22config%22:null},%22mutators%22:[{%22handler%22:%22header%22,%22config%22:{%22headers%22:{%22X-User%22:%22{{|gitpod.io/**\u003e"},"authenticators":[{"handler":"anonymous","config":null}],"authorizer":{"handler":"allow","config":null},"mutators":[{"handler":"header","config":{"headers":{"X-User":"{{> print .Subject }}"}}}],"errors":null,"upstream":{"preserve_host":false,"strip_path":"","url":"<http://umvikeli:4200>"}}]
the "list rules" api returns this
is 
\u003e
correct ?
e
That's <>
in unicode
c
yeah .. just checking that it's expected to be like that
e
**.gitpod.io double *
Other than that, i'm out of ideas
That looks fine to me.
c
ffs
yup that works for "" and "/"
e
😂
For the UI you're gunna want:
Copy code
https://**.<http://gitpod.io/ui/**|gitpod.io/ui/**>
c
and yet another problem. Using the two rules above 
<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io/ui/welcome>
gives 
{"error":{"code":500,"status":"Internal Server Error","message":"Expected exactly one rule but found multiple rules"}}
this is so much harder than it needs to be ...
e
What's the two rules you are using?
c
Copy code
- id: "ui:protected"
  upstream:
    preserve_host: false
    preserve_query: true
    strip_path: "/ui/"
    url: "<http://umvikeli:8081>"
  match:
    url: "https://<**.gitpod.io/ui/**>"
    methods:
      - GET
      - DELETE
      - PATCH
      - POST
      - PUT
  authenticators:
    - handler: anonymous
  mutators:
    - handler: header
      config:
        headers:
          X-User: "{{ print .Subject }}"
  authorizer:
    handler: allow

- id: "api:protected"
  upstream:
    preserve_host: false
    preserve_query: true
    url: "<http://umvikeli:4200>"
  match:
    url: "https://<**.gitpod.io/**>"
    methods:
      - GET
      - DELETE
      - PATCH
      - POST
      - PUT
  authenticators:
    - handler: anonymous
  mutators:
    - handler: header
      config:
        headers:
          X-User: "{{ print .Subject }}"
  authorizer:
    handler: allow
e
Your second rule should be 
https://**.<http://gitpod.io|gitpod.io>**
here is a better one for the ui: 
https://**.<http://gitpod.io/ui{,/**}|gitpod.io/ui{,/**}>
c
so that didn't work .. 404 (for ui)
e
https://<**.<http://gitpod.io/ui{,/**}|gitpod.io/ui{,/**}>>
Keep forgetting the <> xD
c
so 
"https://<**.<http://gitpod.io/ui{,/**}|gitpod.io/ui{,/**}>>"
then 
"https://<**.<http://gitpod.io|gitpod.io>**>"
?
e
Yeah
c
ui works, but "" or "/" gives
trying ``https://<**.gitpod.io{,/**}>``
e
That'll match UI too
https://**.<http://gitpod.io|gitpod.io>*
just try one *
c
nope. this is nuts !
e
https://<**.<http://gitpod.io/ui{,/**}|gitpod.io/ui{,/**}>>
== 
https://<**.<http://gitpod.io/{ui,ui/**}|gitpod.io/{ui,ui/**}>>
If you want the more readable one
Make sure the config reloaded
c
I stop and restart the container
e
Could try this: 
https://<**.<http://gitpod.io|gitpod.io>{,/}>
c
that worked - but onto problem #3 😉
e
Hit me xD
c
however, I've got a feeling I know what the base problem is
the rules are all checked - not the first one that matches
e
Yeah
c
so if I want a ruleset that has ui going to service A and everything else going to service B that's not going to work
{"error":{"code":500,"status":"Internal Server Error","message":"Expected exactly one rule but found multiple rules"}}
e
What is the URL for service B?
c
/
I'm going to have to mount it on a sub path
e
Nah i can exclude routes easy.
c
I suspect going to have to use regex
e
https://<**.<http://gitpod.io/!(ui){,/**}|gitpod.io/!(ui){,/**}>>
Think something like that would work
c
something is not quite right ..
this seems to match what I want
however, 
url: "https://<**.<http://gitpod.io/{ui,ui/**}|gitpod.io/{ui,ui/**}>>"
works for 
<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io/ui/>
but not 
<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io/ui>
(404)
e
Might be your "strip_path"
Is it doing it correctly?
c
the log seems to indicate a route matching issue
Copy code
level=warning msg=Access request denied audience=application error=map[debug: message:Requested url does not match any rules reason: stack_trace:
e
I'm lost on that one, that should work just fine
https://<**.<http://gitpod.io/ui{,/**}|gitpod.io/ui{,/**}>>
does this one work?
c
no - because I suspect the query is getting in the way
that explains why ui doesn't work and ui/ does (the /** pattern matches the ?)
e
Yeah
c
surprised the query is part of the url pattern matching
e
Think you just need 
{ui,ui**,ui/**}
Or just 
{ui**,ui/**}
actually
That should grab your query
c
nope 😞
me this morning, and me this afternoon ....
e
xD
c
I need to make dinner for the kids. Thanks so much for your help, appreciated .
e
All good, totally welcome. I'm helping others to learn it myself tbh 😛
r
wow
congrats to both of you 😄
e
Lots of messages xD
c
not what I'm seeing ... query parameters ...
e
Yeah i'm stumped here, 
{ui**,ui/**}
& 
{ui,ui/**}
should work
r
I am almost sure it's thrown off by the domain part
e
Could be but I've tested each glob separately and they are all isolated
r
or, maybe the go globs are not what you expect them to be
i know that there are a few things as to how globbing works vs other languages
e
True. I guess i could grab the GO package and write a few tests
r
here is an example of what we do (for local dev):
Copy code
- id: ory:runway:protected-api
  upstream:
    preserve_host: false
    url: <http://controller:8000>
  match:
    url: <http://api.runway.127.0.0.1.nip.io/><{,apps,apps/**,certs,certs/**,keys/**,users,admin/**,auth/whoami/}>
    methods:
      - GET
      - POST
      - DELETE
      - OPTIONS
so essentially this allows all the methods on the host (with trailing slash) and then for /apps*, /certs* etc.
but nowhere are we using wildcards for the domain
e
Yeah, not quite sure if the domain has been the issue so far, just the paths
Eg
I tested a similar glob in my local dev setup too and it worked fine.
r
the first one doesn't match, but that's expected
i mean, my rule doesn't satisfy that
e
Yeah
Figured
c
@red-machine-69654 what if you make 2 rules
Copy code
url: http://<*.<http://nip.io/<{,apps,apps/**,certs,certs/**}|nip.io/<{,apps,apps/**,certs,certs/**}>>
and
Copy code
url: http://<*.<http://nip.io/<{,keys/**,users,admin/**,auth/whoami/}|nip.io/<{,keys/**,users,admin/**,auth/whoami/}>>
r
what about it?
c
does it work ?
r
oh, that would not work
as they collide, the first 
,
at least i would see that causing issues
c
wou,d it be better to use regex ?
r
heh. i think the answer is almost always no 😄
more regex never helped anyone
if i was to split them:
Copy code
url: http://<*.<http://nip.io/<{,apps,apps/**,certs,certs/**}|nip.io/<{,apps,apps/**,certs,certs/**}>>

url: http://<*.<http://nip.io/<{keys/**,users,admin/**,auth/whoami/}|nip.io/<{keys/**,users,admin/**,auth/whoami/}>>
See the difference? The first will also match 
http://*.<http://nip.io/|nip.io/>
but the second one won't
i mean, in general, i think you are missing another 
>
but anyway
c
ok, so I think I've found the cause of one of the problems ... 
url: "https://<**.<http://gitpod.io/{,ui,ui/**}|gitpod.io/{,ui,ui/**}>>"
matches 
<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io/ui/>
and 
<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io/ui/welcome>
but doesn't match 
<https://8080-jmls-umvikeli-3iteye1cbgt.ws-eu87.gitpod.io/ui>
but not because of oathkeeper 😠 Turns out that the "kratos-selfservice-ui-node" app , even though 
BASE_PATH=ui
is defined, redirects "/ui" to "/welcome" .. but redirects "/ui/" to "/ui/welcome" I have strip_path set to "/ui" so need to figure out if it's a strip path problem or an issue with the selfservice app itself
so an 
{,
matches the root of the url ?
e
If you need paths for the quickstart, it's listed on the tutorial & github: https://github.com/ory/kratos/blob/master/contrib/quickstart/oathkeeper/access-rules.yml
c
thanks - the point I was trying to make is that "ui" gets redirected to "/welcome" but "ui/" gets redirected to "ui/welcome". doesn't seem to be strip_path related though 🤷‍♂️
e
Sounds like "ui" is getting stripped and maybe redirecting to your first match statement?
3 Views
Open in Slack
PreviousNext
Powered by