Good morning Ory team! The Hydra 2.0 release looks really cool! Especially the DB changes part - we are eager to test it on our self-hosted version! Now we're wondering how can we do this upgrade with no downtime (at least minimal downtime). From our understanding, we need to upgrade 3 components: • Our service (which uses the Hydra Go SDK) • Hydra itself to 2.0 • Hydra database What's the best strategy here? 🙂 (some thoughts and questions in 🧵 )

Any updates here? I am also curious about the recommended path. It would be great to make this upgrade path a bit more explicit in the release notes if possible.

Any follow up on this ? We’re also interested in this. We have some ideas of how to do it but having some insights from the maintainers would help to make the process smoother.

Hey, the upgrade path is very dependent on your deployment and database environment and difficult for us to recommend. Generally, if you have lots of data and a lock-happy database like MySQL you will have to accept some downtime, I think. Generally: 1. Do a dry-run of the release: a. Test the SQL upgrade on a production database back up before and check for regressions with manual QA b. Check if your code integrations continue working with (they should as we did not add API changes except for the path change) c. See how long it takes to upgrade and identify strategies to reduce the upgrade time 2. If everything works: a. Prepare the configuration templates for v2 in your deployment environment b. Run SQL Migrations (might cause downtime) c. Once they are finished switch to v2 binary If you observe problems during SQL Migrations, there is an open issue that we are trying to find the root cause for. Please comment there if you have the same problem: https://github.com/ory/hydra/issues/3346

Thanks @high-optician-2097 for the feedback 🙏 Of course it's specific, but I was wondering if there was some tweaking that could be done in order to smooth the deployment 😉 We are using postgresql as a DB for Hydra. The strategy we came up with so far: • As you say, dry-run the migration ◦ See how long it takes on our prod DB ◦ If everything still works as expected ◦ But in an isolated env • Once we have this is done and we have a good understanding of the time taken and the specifics: ◦ Clone the DB in prod ◦ Spin up Hydra v2 servers pointing to the cloned DB + the migration job on the cloned DB ◦ Once the migration is done, switch routing from the v1 servers to the v2 servers ◦ And later, clean up v1 servers and the prior DB This should avoid downtime, with the trade-off that users that received Hydra OAuth2 tokens in mean time will have to reconnect. Therefore we are planning to do this when we expect low traffic, so minimize the degraded UX. Since the migration cannot be "undone", our fear is that the actual production env ends up in a state where we cannot come back to the prior one quickly enough (restoring a backup + rolling back instances to v1).

I understand. The plan sounds good! The risk of something going wrong is always there. This is a good case study into why we’re building out our service offering though too. Every adopter of Ory has to re-learn how to do these upgrades and there are multiple additional risk factors that we can not factor in during development (e.g. custom clean up jobs, custom deployment environments, custom database configurations, …). Our community spends a lot of time (and thus money) in getting all of this done. That’s we specialize in running our own software with the appropriate tooling and organizational structure 🙂 In the end to help the customer reduce the amount of work and money they have to put in when using Ory, and of course to further fund the development and innovation of Ory Open Source 🙂 If this is a good time to figure out how we can offload you with this work and get you on the Ory Network, we’re here!