Hi, I'm doing some updates to my Ory Network proj...
# ory-network
e
Hi, I'm doing some updates to my Ory Network project using the API using the PATCH method and have noticed that if the patches happen very close to one another then the last one in wins to some degree and some of the previous patches will be ignored. Is this a known issue and is there a way to stop this from happening? Ideally I'd like to be able to run several PATCH requests in parallel and not have to batch them all up into one PATCH so that I can separate the config updates as required
m
Hey @enough-truck-65683 yes I suppose if you have several path requests coming in at the same time there are some concurrency issues.
Ideally I'd like to be able to run several PATCH requests in parallel
can you explain a bit more what the use case is behind this? I would assume you don't have too many config update changes in a short time normally?
s
I was actually experimenting with working on a terraform provider that uses the API to patch the config for Ory Network. It's all going well except when I try to update multiple resources at once which I narrowed down to multiple resource updates calling patch operations at the same time. I did also try using the patch with revision endpoint but they'll both get the current revision at the same time and spawn their own new revisions with the last one seemingly winning. It also means each resource would be making 2 calls to the API to update one resource
s
First: please always use the endpoint with the revision ID, as that ensures your request either succeeds or fails, but not silently gets overridden. We will eventually remove support for PATCH without the revision.
On the other side, can't you merge the operations into one PATCH?
s
As far as I can see, different resources when creating a terraform provider all call their own respective create/read/update operations and never have the option to cross over and therefore have no way to merge the patches. What's the recommendation for getting the latest revision ID? Will I always need to make a call to get the project before doing the patch? Or is there a way to tell it to always just update the latest revision as I know my patch operations won't be interfering with one another in terms of what they update?
I did originally use the patch with revision but still ended up with a situation where they'd both read the latest revision at the same time and then both succeed in patching but the last one seemed to win. I can try to replicate it later today with a script calling the API if that's helpful?
s
I see, that makes sense. What exactly are the resources you want to manage? Our recommendation would be to follow what the API offers, i.e. having the full config as a resource.
s
Eventually I wanted to be able to manage all Ory Network config via terraform and was hoping to utilize patch operations to be able to treat separate configuration options as individual resources. Having one resource that controls the whole config seems to defeat that purpose. I noticed that there's no official Ory terraform provider so was hoping to provide one that people can use. You can see an example of what I was aiming for in my first attempt at a (still buggy) provider here https://registry.terraform.io/providers/kibblator/ory/latest/docs https://github.com/kibblator/terraform-provider-ory Forgive my code, this is my first time writing Go and also a terraform provider. Its a mixture of trying to learn a new language and provide something useful for the community
s
We do have plans to offer an official provider, but it is currently not being actively worked on. I do understand your point about partial updates, but there is really nothing we can do to support that, as we have to treat the config as a whole.
s
So I assume the Ory cli patch command would suffer with the same issue if called concurrently since it's utilizing the API calls under the hood?
s
exactly
btw, this is what chatgpt recommended, just as a starting point for options we have:
Yes, you can achieve fine-grained Terraform support for your API even if the API itself only deals with a huge JSON blob. This is a common pattern when providers need to manage a monolithic configuration model but expose more granular resources to users. ### Key Concept: Terraform Provider as the Translation Layer Your Terraform provider acts as the abstraction layer between: • Fine-grained Terraform resources (
server_config_logging
,
server_config_auth
, etc.) • The single blob API (
PUT /config { huge_blob }
) --- ### Strategy: Local Merge of Resources 1. Split State in Terraform Resources Define several fine-grained resources in your provider, like:
Copy code
hcl
   resource "myapi_logging_config" "main" {
     level = "info"
   }

   resource "myapi_auth_config" "main" {
     method = "oauth2"
   }
2. Provider Maintains a Shared State or Cache In your provider's code, for each resource's
Read
,
Create
,
Update
, and
Delete
, you: • Read the full config blob from the API • Modify just the relevant piece for that resource • Merge all resource-specific changes back into the blob • Push the updated blob back to the API 3. Synchronize Access Across Resources Since all resources operate on the same shared blob, concurrency is a problem. Solutions include: • Locking mechanisms (e.g., mutexes in Go) • Serialization of resource operations using Terraform SDK's
CustomizeDiff
or an umbrella "controller" resource --- ### Example Implementation Pattern (in Go):
Copy code
go
var configMutex sync.Mutex

func resourceLoggingConfigUpdate(d *schema.ResourceData, meta interface{}) error {
    configMutex.Lock()
    defer configMutex.Unlock()

    client := meta.(*APIClient)
    fullConfig, err := client.GetConfig()
    if err != nil {
        return err
    }

    // Modify the logging section
    fullConfig.Logging.Level = d.Get("level").(string)

    // Push entire blob
    return client.UpdateConfig(fullConfig)
}
--- ### Alternative: Umbrella Resource +
terraform-provider-helper
You can also create one top-level resource (e.g.
myapi_config
) and expose helper modules or internal subresources inside it, but that tends to be less ergonomic for users. --- ### TL;DR You can merge multiple Terraform resources into one JSON blob in the provider code Use internal locking and shared reads/writes Avoid API races or overwrites by reading-modifying-writing the blob carefully This is a common and accepted practice in custom Terraform providers --- Would you like a code example or scaffold of such a provider setup?
s
I'll give that a look thanks 👍 Appreciate the support 😁
s
np, if you get something working it should speed up things on our side and we could officially support your provider
s
Just an update to this. I think I've been able to work around this issue using a shared mutex lock. Thanks again!