https://www.ory.sh/ logo
Join Slack
Powered by
Hello dear Ory'ers! Has anyone tried fetching pagi...
# talk-kratos
c
Hello dear Ory'ers! Has anyone tried fetching paginated kratos data from the Ory native (Go) SDK? It seems rather finnicky, maybe since I can't set the page_token or page_size params in the sdk requests. The result is I'm getting a lot of duplicate responses, and usually also miss the last page. My logic thus far is: Request a page (i tried starting on page 0, page 1 and just not using a page at all), then receive a response with a "next" page, which I then request - receive a response with a "next" page, ad nauseum. In my head, by the time I receive a response with no "next" page, I should have received all the identities, right? Well, wrong, I'm missing the last page, and I also have a bunch of duplicates 🙃
s
logic seems solid, which endpoint are you trying?
c
This one
Copy code
oc.IdentityApi.ListIdentities(oc.AuthCtx(ctx))
I'll dig into the code and see which url it resolves to
Copy code
localVarPath := localBasePath + "/admin/identities"
s
can you maybe post the snippet with the loop where you run the queries?
c
Of course 🙂
Copy code
req := oc.IdentityApi.ListIdentities(oc.AuthCtx(ctx)).PerPage(50)

	var identities []ory.Identity
	page := 0
	for {
		moreIdentities, res, err := req.Page(int64(page)).Execute()
		if err := oc.appendMessageIfError(res, err); err != nil {
			return nil, err
		}
		identities = append(identities, moreIdentities...)
		fmt.Println(res.Header.Get("link"))
		next, ok := link.ParseHeader(res.Header)["next"]
		if !ok {
			break
		}
		uri, err := url.ParseRequestURI(next.URI)
		if err != nil {
			return nil, err
		}
		fmt.Println("Looking up page", uri.Query().Get("page"))
		page, err = strconv.Atoi(uri.Query().Get("page"))
		if err != nil {
			return nil, err
		}
	}
I even left the Println debugging in there 🙂
So I read in a github issue that the pagination logic is 1-indexed, not 0, but since the "rel=first" page says page=0, I tried that too
Both render the same result though
s
Both render the same result though
yeah thats why you'd want to start at 1
c
Ah, yeah
Makes sense
s
but the code looks fine, can you maybe paste the link headers you got from the log?
just the page param, so we can make sure they are correct
c
Copy code
Looking up page 1
<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=0&page_size=50&page_token=eyJvZmZzZXQiOiIwIiwidiI6Mn0&per_page=50>>; rel="first",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=2&page_size=50&page_token=eyJvZmZzZXQiOiIxMDAiLCJ2IjoyfQ&per_page=50>>; rel="next",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=0&page_size=50&page_token=eyJvZmZzZXQiOiIwIiwidiI6Mn0&per_page=50>>; rel="prev",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=7&page_size=50&page_token=eyJvZmZzZXQiOiIzNTAiLCJ2IjoyfQ&per_page=50>>; rel="last"
Looking up page 2
<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=0&page_size=50&page_token=eyJvZmZzZXQiOiIwIiwidiI6Mn0&per_page=50>>; rel="first",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=3&page_size=50&page_token=eyJvZmZzZXQiOiIxNTAiLCJ2IjoyfQ&per_page=50>>; rel="next",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=1&page_size=50&page_token=eyJvZmZzZXQiOiI1MCIsInYiOjJ9&per_page=50>>; rel="prev",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=7&page_size=50&page_token=eyJvZmZzZXQiOiIzNTAiLCJ2IjoyfQ&per_page=50>>; rel="last"
Looking up page 3
<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=0&page_size=50&page_token=eyJvZmZzZXQiOiIwIiwidiI6Mn0&per_page=50>>; rel="first",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=4&page_size=50&page_token=eyJvZmZzZXQiOiIyMDAiLCJ2IjoyfQ&per_page=50>>; rel="next",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=2&page_size=50&page_token=eyJvZmZzZXQiOiIxMDAiLCJ2IjoyfQ&per_page=50>>; rel="prev",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=7&page_size=50&page_token=eyJvZmZzZXQiOiIzNTAiLCJ2IjoyfQ&per_page=50>>; rel="last"
Looking up page 4
<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=0&page_size=50&page_token=eyJvZmZzZXQiOiIwIiwidiI6Mn0&per_page=50>>; rel="first",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=5&page_size=50&page_token=eyJvZmZzZXQiOiIyNTAiLCJ2IjoyfQ&per_page=50>>; rel="next",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=3&page_size=50&page_token=eyJvZmZzZXQiOiIxNTAiLCJ2IjoyfQ&per_page=50>>; rel="prev",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=7&page_size=50&page_token=eyJvZmZzZXQiOiIzNTAiLCJ2IjoyfQ&per_page=50>>; rel="last"
Looking up page 5
<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=0&page_size=50&page_token=eyJvZmZzZXQiOiIwIiwidiI6Mn0&per_page=50>>; rel="first",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=6&page_size=50&page_token=eyJvZmZzZXQiOiIzMDAiLCJ2IjoyfQ&per_page=50>>; rel="next",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=4&page_size=50&page_token=eyJvZmZzZXQiOiIyMDAiLCJ2IjoyfQ&per_page=50>>; rel="prev",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=7&page_size=50&page_token=eyJvZmZzZXQiOiIzNTAiLCJ2IjoyfQ&per_page=50>>; rel="last"
Looking up page 6
<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=0&page_size=50&page_token=eyJvZmZzZXQiOiIwIiwidiI6Mn0&per_page=50>>; rel="first",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=7&page_size=50&page_token=eyJvZmZzZXQiOiIzNTAiLCJ2IjoyfQ&per_page=50>>; rel="next",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=5&page_size=50&page_token=eyJvZmZzZXQiOiIyNTAiLCJ2IjoyfQ&per_page=50>>; rel="prev",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=7&page_size=50&page_token=eyJvZmZzZXQiOiIzNTAiLCJ2IjoyfQ&per_page=50>>; rel="last"
Looking up page 7
<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=0&page_size=50&page_token=eyJvZmZzZXQiOiIwIiwidiI6Mn0&per_page=50>>; rel="first",<<https://nostalgic-agnesi-otu9x8e3js.projects.oryapis.com/identities?page=6&page_size=50&page_token=eyJvZmZzZXQiOiIzMDAiLCJ2IjoyfQ&per_page=50>>; rel="prev"
Looks kinda messy, but there they are 😄
(also, notice how "rel=first" actually gives me page 0, not 1. Minor detail, but, you know, would render duplicates if both page 0 and 1 is read)
s
hmm I'm actually confused why you get both 
page
and 
page_token
lemme investigate
c
Thanks!
I would actually prefer page_token, but the SDK does not allow me to set it
s
do you have the latest sdk?
c
v1.1.21
I'm using the big, aggregated one, github.com/ory/client-go
s
yeah that one is up to date with ory network
did this just start to happen or did you not try before?
c
I simply didn't try it before, we just did a migration that bumped us above the default page size 😛
s
hm so it looks like the page token takes precedence over the page, but it is missing in the sdk
c
And I was like "Heyo, we are missing some identities"
s
but in that case it should work with the page size
c
Yeah. So I just confirmed your suspicion that the duplicates are caused by reading both page 0 and page 1, so that makes total sense
However, I'm still not getting the last page
It literally says page 7 is the last one, but I'm missing like 4 identities that should be on page 8 in this case
Fun fact, if I manually read page 8 (without getting it from the link header), I do get my missing 5 identities
However, if I am to trust the notion that page numbers are not neccessarily in order, and that I should go to the page the header tells me to - well, I would feel a lot more comfortable being able to just trust the header at all times
I also don't know what happens if the next page is empty, maybe I'll just get an error like a 404 or something?
s
ok so it is rather a bug with the last page not being detected correctly
i was looking into whether the pagination is unstable, i.e. an identity can come up on multiple pages
but I guess that is not the case then
c
I mean, it's obviously not an atomic operation, so if someone makes changes to the data while I'm at page 5/8 or whatever, I could still be unlucky and get a duplicate I guess, but that's just something I gotta tolerate
s
yes that for sure
c
Like, if I read users 151-200, and someone deletes user number 199, that will move user 201 down to 200, so when I request 201-250, I'll miss that one user
But those things are pretty much impossible to mitigate
Anyways, it'd be real nice to get the missing page from the link header 😁
For now I can just work around it by reading one more page, but for the sake of robustness it's nicer to get it from the ory backend
s
well we can do time-travel queries in the db 🤔 so you always use a certain timestamp and every subsequent read would read as if it were at that same time as well
c
Really? That's pretty cool! I didn't know the db was like a ledger db (or whatever buzzword you wanna call it)
Would be a nice feature, but it's not more important than the missing page 😛
s
c
Very cool!
s
I think I found the issue, IMO one part of the code assumes 0-indexed pages, while some other assumes 1-indexed pages
c
Oh, that'll make for some interesting side-effects 😎
s
yeah.... let's call it interesting 😂
c
Aight, two more questions then: Will it end up being 0-indexed or 1-indexed in the end? Fix deployed when?
I'll work on replacing the keto sdk with grpc in the mean time, for that extra ns/op 🔥
s
I guess 1 will be less breaking right?
c
Yeah, probably if that's what most consumers are doing already
s
c
Nice 🍾
12 Views
Open in Slack
PreviousNext
Powered by