We're having issues with Ory Network.
# ory-networka
acoustic-airline-26089
08/04/2023, 2:36 PMWe're having issues with Ory Network.
acoustic-airline-26089
08/04/2023, 2:37 PMcame back again almost immediately though....
acoustic-airline-26089
08/04/2023, 2:37 PMbut console wasn't loading and our app was getting CORS errors from whoami.
h
high-optician-2097
08/04/2023, 2:43 PM
We’re looking into it
✅ 1
high-optician-2097
08/04/2023, 2:45 PM
We received a few alerts, looks like some pods restarted. Everything is continuing to run though
a
acoustic-airline-26089
08/04/2023, 3:18 PMokay.
acoustic-airline-26089
08/04/2023, 3:18 PMthanks!
f
full-scientist-38268
08/04/2023, 6:05 PMI'm getting
security_csrf_violationfull-scientist-38268
08/04/2023, 6:05 PMCould this be related?
h
high-optician-2097
08/05/2023, 10:16 AM
No
high-optician-2097
08/05/2023, 10:18 AM
@acoustic-airline-26089 sorry for not following up yesterday. It looks like several Kubernetes VMs restarted due to an GKE upgrade and that caused a short increase in errors where network requests failed with EOF. The system recovered quickly and we could not see any other issues. We’ll still investigate what we can do to prevent these network errors.
a
acoustic-airline-26089
08/06/2023, 4:48 PM@high-optician-2097 - thanks for the update, sure, please keep us updated. Seems like lots of upgrades happening so trying to be patient and understanding 🙂
h
high-optician-2097
08/07/2023, 6:43 AM
Agree @acoustic-airline-26089 - this particular issue was a really short (less than a 10 or 15 seconds) elevation of error rates in one region of our system. We’ll be looking into what happened and what we can do to get this sorted out as well. Having said that, the system is now fully multi-region and for example other regions continued to operate - we also don’t have plans for major infrastructure changes such as the ones we performed recently in the near future 🙂
a
acoustic-airline-26089
08/10/2023, 2:05 PMdid this just happen again?
h
high-optician-2097
08/10/2023, 2:09 PM
I’m checking with SRE. What did you experience?
a
acoustic-airline-26089
08/10/2023, 2:10 PM Copy code
Access to XMLHttpRequest at '<https://auth.noah.com/self-service/login/browser>' from origin '<https://app.noah.com>' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
loggerBase.ts:109 ERROR: Error: Network Error
at createError.js:16:15
at XMLHttpRequest.<anonymous> (xhr.js:91:14)
instrumentMethod.js:37 GET <https://auth.noah.com/self-service/login/browser> net::ERR_FAILED 403acoustic-airline-26089
08/10/2023, 2:10 PMstill happening apparently.
acoustic-airline-26089
08/10/2023, 2:11 PMfor 'some' users.
h
high-optician-2097
08/10/2023, 2:12 PM
do you know the IPs of these users?
high-optician-2097
08/10/2023, 2:14 PM
because we see that some IPs have been blocked due to suspicious bot activity
high-optician-2097
08/10/2023, 2:16 PM
its actually only one ip in the last 30 minutes: 173.53.59.146
a
acoustic-airline-26089
08/10/2023, 2:24 PMyeah, that's the one: 173.53.59.146
h
high-optician-2097
08/10/2023, 2:27 PM
ok, is that a legitimate user?
a
acoustic-airline-26089
08/10/2023, 2:28 PMyes, our CSM.
h
high-optician-2097
08/10/2023, 2:28 PM
ok, let me check with SRE how to allow list this IP
high-optician-2097
08/10/2023, 4:04 PM
we explicitly allowed this IP now - but will keep the heightened bot detection online
✅ 1
2 Views