https://www.ory.sh/ logo
Join Slack
Powered by
Hey, we’re seeing some really strange behavior for...
# ory-network
k
Hey, we’re seeing some really strange behavior for users logging in with Facebook… Login succeeds but everyone is logged in as the same random account
✅ 1
🆘 1
@magnificent-energy-493 sorry for the ping but we really need help here
h
We have not changed anything. Do you know when the issue started?
It could be a facebook problem
@famous-art-85498
k
we received the first report yesterday at 11:20 pm UTC
h
thank you! we have not deployed any changes in the last days, it seems to be facebook related
k
it’s weird though because it logs into the same user that did exist
f
hey, I’m setting up fb app and to test on my end
h
another customer also reported problems
k
we’ve disabled facebook login in the meantime
h
but we did not deploy anything so the working theory is that the problem is on facebook side - can be anything from automated flagging because we have lots of return urls on the same domain to them having broken their api
yes
k
ah, thanks
h
good morning! the incident seems to be resolved (since saturday) according to the FB status page, however, we still get the same issue with being logged in as that same account.
t
it looks like due to this incident we have at least one identity with 
"facebook:"
as identifier (see below). I suspect that this leads to our current issue where every login with facebook results in the user being logged in with that identity. we therefore still have facebook login disabled and need to resolve this issue before we can re-enable. since many of our users can only login using facebook that's obviously a major issue.
Copy code
{
	"id": "xx",
	"credentials": {
		"oidc": {
			"type": "oidc",
			"identifiers": [
				"facebook:"
			],
			"version": 0,
			..
		},
..
}
is there a way to easily look up identities by that oidc identifier? I'm not sure if there are any other identies with this problem
do you have a recommendation how to fix this? if I update the identity and simply remove the facebook oidc credential, will the user be able to re-connect their facebook? or will the facebook login still be "taken" due to other internal data?
ok, next question: how do I remove that credential from the user? 🤔 I thought I can use https://www.ory.sh/docs/reference/api#tag/identity/operation/updateIdentity to set 
identifers
to an empty array, but looks like it just ignores the input
hey @magnificent-energy-493 sorry for the direct ping, but is there any chance somebody from the team can help us out here? it's relatively critical for us to resolve the issue
m
No worries Dan, feel free to ping me. We are actively looking into the facebook issue, will keep you posted.
t
thanks!
hi @magnificent-energy-493 do you have an update? our facebook login is still disabled
f
Hey Dan, I’m looking into this, will keep you updated
t
thank you. any ETA would help to since we need to communicate with our customers too...
h
We are still investigating why Facebook is returning incorrect data. The empty identifier for sure is an issue we will provide a fix for. I don’t have an exact ETA for you but the whole team is looking into it and we will have it fixed by tomorrow CET or at least an update
The empty identifier is not the root cause of the other users not being able to sign in though, at least in our analysis
t
thanks for the update!
h
We’re also in contact with people at Facebook but it’s proving non trivial to get information there :(
@thankful-dog-96817 have you by any chance already deleted the user with the empty subject ID? And if so, did that recover the login status?
t
I did not
I also can't delete the user, I only want to reset the facebook connection. but I couldn't figure out how to do that
h
You mean that you can’t delete the user in the UI / using the API / CLI?
What error are you getting?
t
no I mean it's a real customer with real data that I can't just delete
h
Oh, I see
Does that customer have another method of signing in?
Like a password, or you have their email, or something
t
I have their email. I'm not sure if they've set a password, but I think so
the user's identity id is 
1641282a-e4c7-4467-a542-77ca4e5a99c6
if that helps
h
Ok, and could you quickly describe what happens when another customer signs in using facebook? Are they signed into 
1641282a-e4c7-4467-a542-77ca4e5a99c6
?
t
correct
h
Oh, I see, very interesting
t
yeah, that's one way to put it 🙃
h
Ok, then it has to be the empty credential. One thing we can do is to remove the credential from the user, that would mean that they loose their facebook connection. But they can recover the account using a recovery email.
t
another way is that we have a bunch of unhappy customers who are rightfully worried and complaining how it was possible for them to suddenly be logged into a different account and see all the associated personal data :S
we have no test coverage for that particular edge case
h
And I’ll direct the team to look into if an empty credential can cause this issue. We’ll also add a safe-guard that no empty credential can be entered into the database
t
One thing we can do is to remove the credential from the user, that would mean that they loose their facebook connection.
yes, that would be my best guess as well. can you do that for us please?
h
If facebook instead of returning invalid userinfo data would have returned 502 gateway unavailable with no payload, all of this would not have been an issue. I understand the frustration that comes from a problem like this, but we too did not have a test case for such a failure scenario 😞 We will add the tests though to ensure this doesn’t happen again, and have also identified some areas for handling such errors with grace
Yes will do that, but first we want to confirm that this is indeed what’s happening. There’s still a point that doesn’t make sense, but we’re testing hypothesis
👍 1
t
sorry I didn't mean to come across as blaming you. shit happens, I totally understand
h
No problem 🙂 It’s eng #1 priority and we’ll get it sorted.
t
if anything I'm more happy now to have outsourced identity management. this sort of issue sounds like an absolute nightmare 🙃
❤️ 1
sometimes you're the bird and sometimes you're the worm
🐤 1
h
@thankful-dog-96817 since when did you have facebook enabled, and for how long did it work fine? do you have customers that signed in using facebook successfully?
t
enabled since around 23 Jun 2022 (+- a couple of days due to staged rollout)
customers who signed in with facebook sucessfully... I can't prove it right now, but I'd be surprised if it wasn't the case. it got through QA, so we'd at least have 1-2 test accounts that worked at some point
h
ok, thank you. we have identified all the problematic entries and will deploy a hotfix. facebook will be back to normal by the end of the day. we’ll let you know when exactly
t
thanks for the update!
h
@thankful-dog-96817 the hotfix is now live. Can you confirm that the issue is resolved on your end? Please note that the one user which had empty credentials will not be able to sign in using facebook right now. If they sign in using facebook, they will create a new account
t
looks like it working now. thanks! we'll run some more checks just to be sure
h
thank you - we also have some significant code changes and added test coverage for these use cases. They will land on production in the next couple of days but have to pass through QA and CI first. They will make sure that something like this never happens again
Open in Slack
PreviousNext
Powered by