Is it possible to have "Invite-only" and only allow logins with social sign in?

Sure, you can e.g. leave out the sign ups/registration forms in your frontend. I think there are other solutions also, will see if I can find something.

Will that prevent user from registering via SSO unless there is an outstanding invitation?