Finally figured it out I think, the PAT configured in the admin client (which I am also using for .toSession on the backend) seems to instead look for a session for the admin user, skipping the supplied session token/session cookie
It was late and Im pretty sure this was totally wrong, my issue was with custom domains, where I was checking for a session on the Ory Network domain, but signing in via the custom domain, which I suspect means that the cookie domain didnt match