Hey everyone 👋. I’m evaluating Keto for a project and I’m wondering around how it works with large lists of objects and sorting. Let’s say I have three “channels” of objects, and I want some groups to have access to all of a channel’s objects, and some users to have access to specific objects, I may want to list a user’s accessible objects, in order of an attribute, regardless of channel. It’s not obvious to me what the correct modelling here would be for large datasets. AFAICT I either need to determine all of the objects a user has access to and then filter down the objects in the application and then sort them, or loop over all the objects and then check permissions in pages until I have a “page” of objects. Any other option I think of ends up moving some of the permissions data back into the application domain, which feels incorrect (e.g. having group requirements on the object table). This was discussed in this issue without any real answer so I’m wondering how this might be solved: https://github.com/ory/keto/issues/546#issuecomment-817761825

Hi Mal, you are right that currently there are two strategies when dealing with a list of items: • query Keto for a list of all item IDs that the user can view:

items:*#view@user

, then use this set of IDs to filter in the application DB, or • Query the application DB first, then (batch) query Keto. Both approaches work, and which one you choose should depend on if you expect Keto or your DB query to filter out more, and then do the stricter filter first. However, as you noted, this will not scale to very large lists, since we are always hitting two services. @steep-lamp-91158, @fast-lunch-54279 and I just talked about how to better solve this in the future, so we are still ideating, feel free to chime in. Some thoughts: • Directly translating a check request to a (partial) SQL query as mentioned in the OPA article is infeasible to do in a generic way in Keto, especially with the advent of user-set rewrites (which uses recursive queries that are not possible to represent in plain SQL) • We could expose "permission change stream" (including the "materialized" indirect tuples from subject set expansions and user-set rewrites) that would allow the application to mirror certain permission sets in their database to do an efficient

JOIN

. E.g., in the application DB, you would have a table

items_permissions

with columns "item_id", "user_id" and "relation", so that in order to get a list of items for a user you would do

select * from items i join items_permissions p where (i.id = p.items_id and p.user_id = ? and p.relation = 'view')

As I said, we are still in the brainstorming phase here. Do you have any other ideas or use cases we should consider? Happy to chat 🙂.

Thanks for the insight Henning 👋. Having the application have a copy / cache of the data seems inevitable at this point. The idea of a downstream consumer of changes / state with the application subscribing to those changes and updating its representation is somewhat appealing. I think I have a solution in mind which doesn’t push all of the permissions into Keto. We can simply store a group id against the domain objects in our application, and query Keto for a user’s groups then filter in SQL. While this means some application level permissions it feels like an acceptable trade off (e.g. the domain object attachment to the group is stored in the application not in keto).