things i've explored so far include: - adding writ...
# talk-keto
f
things i've explored so far include: • adding writes to keto in application code alongside writes to the database • using postgres triggers to push messages that describe changes onto a message queue, and handling the messages by writing to keto • writing a logical replication client for postgres that maps changes onto writes to keto all of these have undesirable characteristics. the first two cannot actually guarantee consistency in all reasonable cases. the third has stronger guarantees but it is both difficult to do and couples the synchronization use case to the storage technology. i feel like i'm chasing something unattainable and maybe i would be better off just seeing what other folks have had success with. thank you!
s
Maybe you don't want to replicate the data in the first place but have Keto as the single source of truth? Or are you doing some fancy join queries that need that
parent_id
?
f
no fancy joins, just regular joins.
File
and
Folder
are tables in a relational database. developers are not going to accept a solution where they have to make requests to a permissions service in order to perform ordinary relational joins
and in any case storing the relationship in keto wouldn't solve the root problem that there is data that must be stored in two data stores with a guarantee of consistency. even though the
parent_id
column is no longer a concern, we would still have to guarantee that any parent relationships in keto are synced with the actual existence of the referenced rows in postgres.
i think what we're gonna try is run a change data capture process against our database and sync relevant updates into keto in an eventually consistent manner. that's the only option i'm aware of that can guarantee the relationships in keto will not become corrupted
s
sorry, somehow overlooked your answers I guess the eventual consistent approach will be the most robust as well. Maybe you can detect edge cases where updates did not propagate yet and force the propagation, then those edge case requests would just take longer, instead of returning the wrong result.
f
oh that's interesting. good suggestion, we'll look into it. thank you!