TL;DR apparently the CI automation (flake/edge-cas...
# ory-network
h
TL;DR apparently the CI automation (flake/edge-case/…) removed three critical files which are responsible for synchronizing the application states across our Kubernetes clusters. Once these are gone, the system reconciles and thinks that all pods should be deleted, which was the case here. The post-mortem will most likely include the reason why that happened and what we’re doing to prevent gitops manifests incorrectly being deleted in the future (most likely locks/improved human review process)
1
t
Don't worry, I managed to do this same thing on one of our internal Platform K8 clusters recently, which caused Flux to action a cascading delete across the cluster - We've all been there 😉 🫂
❤️ 2
🙏 1
h
The bane of gitops …
Still better than manually running shell scripts but yeah .
c
Thanks!
g
The handling of the (brief) outage tonight was awesome. Rapid response and regular informative comms throughout kudos to the Ops Team
❤️ 2
🙏 3