Hello y'all I've found an error message that doesn't fit the problem that's happening. I thought of creating an issue on Hydra's Github page but I'll ask here first Basically, start Hydra using the default quickstart yaml files with only a slight update of the env variables (but the problem happens anyway without changing this env variables) So, in

quickstart.yaml

environment:                                        
      - DSN=sqlite:///var/lib/sqlite/db.sqlite?_fk=true 
      - LOG_LEAK_SENSITIVE_VALUES=true                  
      - TTL_ACCESS_TOKEN=15m                            
      - TTL_ID_TOKEN=15m                                
      - TTL_REFRESH_TOKEN=30m                           
      - LOG_LEVEL=trace                                 
      - SECRETS_SYSTEM=super_secure_system_believe_me

Then run

docker-compose -f quickstart.yml \                                                                                                                                                                                                    
    -f quickstart-postgres.yml \
    up --build

Whenever hydra server is ready, open the terminal and do

curl --connect-timeout 1 -m 0.01 localhost:4444/.well-known/openid-configuration

Then hydra will show the following error:

2023-08-11 10:06:51 time=2023-08-11T09:06:51Z level=info msg=An error occurred while handling a request func=<http://github.com/ory/x/logrusx.(*Logger).ReportError|github.com/ory/x/logrusx.(*Logger).ReportError> file=/go/pkg/mod/github.com/ory/x@v0.0.559/logrusx/logrus.go:230 audience=application error=map[debug: message:server_error reason:Could not ensure that signing keys for "hydra.openid.id-token" exists. If you are running against a persistent SQL database this is most likely because your "secrets.system" ("SECRETS_SYSTEM" environment variable) is not set or changed. When running with an SQL database backend you need to make sure that the secret is set and stays the same, unless when doing key rotation. This may also happen when you forget to run "hydra migrate sql.. stack_trace:
...
2023-08-11 10:06:51 <http://github.com/ory/x/prometheusx.(*MetricsManager).ServeHTTP|github.com/ory/x/prometheusx.(*MetricsManager).ServeHTTP>
2023-08-11 10:06:51     /go/pkg/mod/github.com/ory/x@v0.0.559/prometheusx/middleware.go:41 status:Internal Server Error status_code:500] http_request=map[headers:map[accept:*/* user-agent:curl/7.84.0] host:localhost:4444 method:GET path:/.well-known/openid-configuration query:<nil> remote:172.31.0.1:48368 scheme:http] http_response=map[status_code:499] service_name=Ory Hydra service_version=v2.2.0-rc.2

This error is not consistent with what's happening. The SECRET_SYSTEM is okay, everything is fine but Hydra receives a 499 (client ended the session early) but is saying the problem is internal. This is the piece of code that produces this error

func (j *DefaultJWTSigner) getKeys(ctx context.Context) (private *jose.JSONWebKey, err error) {
	private, err = GetOrGenerateKeys(ctx, j.r, j.r.KeyManager(), j.setID, uuid.Must(uuid.NewV4()).String(), string(jose.RS256))
	if err == nil {
		return private, nil
	}

	var netError net.Error
	if <http://errors.As|errors.As>(err, &netError) {
		return nil, errors.WithStack(fosite.ErrServerError.
			WithHintf(`Could not ensure that signing keys for "%s" exists. A network error occurred, see error for specific details.`, j.setID))
	}

	return nil, errors.WithStack(fosite.ErrServerError.
		WithWrap(err).
		WithHintf(`Could not ensure that signing keys for "%s" exists. If you are running against a persistent SQL database this is most likely because your "secrets.system" ("SECRETS_SYSTEM" environment variable) is not set or changed. When running with an SQL database backend you need to make sure that the secret is set and stays the same, unless when doing key rotation. This may also happen when you forget to run "hydra migrate sql..`, j.setID))
}

yes looks like the error handling can be improved there, not sure how to do it globally, but maybe @refined-kangaroo-48640 has a pointer?

Please open an issue in ory/hydra. curl

-m 0.01

means curl cancels the request after 10ms. Hydra detects that and should log status 499 and the string

context canceled

should appear somewhere in the error message.

curl

-m 0.01

means curl cancels the request after 10ms

yeah, @refined-kangaroo-48640 just did this to show the error since it's happening in my local environment with 3500ms (node-openid-client)