I've been having some issues with keto starting up in a broken state and was wondering if anyone had an advice or suggestions on what to do. To my knowledge I'm not doing anything strange (happy to share more configs and stuff), but sometimes when keto starts up in our CI environment (and it also happens on macos with the brewed formula) keto will return 404 for endpoints that exist and work normally other times. Here's an example of some curl output showing this condition:

#!/bin/bash -eo pipefail
for i in `seq 1 10`;
do
  curl -v -X GET '<http://localhost:4468/relation-tuples>' | grep '{"relation_tuples":\[\],"next_page_token":""}' && exit 0
  sleep 2
done
echo Failed waiting for keto && exit 1

Note: Unnecessary use of -X or --request, GET is already inferred.
* Uses proxy env variable NO_PROXY == '127.0.0.1,localhost,circleci-internal-outer-build-agent'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:4468...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 4468 (#0)
> GET /relation-tuples HTTP/1.1
> Host: localhost:4468
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Tue, 04 Oct 2022 19:41:32 GMT
< Content-Length: 19
< 
{ [19 bytes data]
100    19  100    19    0     0  19000      0 --:--:-- --:--:-- --:--:-- 19000
* Connection #0 to host localhost left intact

but other times the same check passes on the first try:

#!/bin/bash -eo pipefail
for i in `seq 1 10`;
do
  curl -v -X GET '<http://localhost:4468/relation-tuples>' | grep '{"relation_tuples":\[\],"next_page_token":""}' && exit 0
  sleep 2
done
echo Failed waiting for keto && exit 1

Note: Unnecessary use of -X or --request, GET is already inferred.
* Uses proxy env variable NO_PROXY == '127.0.0.1,localhost,circleci-internal-outer-build-agent'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:4468...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 4468 (#0)
> GET /relation-tuples HTTP/1.1
> Host: localhost:4468
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Date: Tue, 04 Oct 2022 18:43:05 GMT
< Content-Length: 44
< 
{ [44 bytes data]
100    44  100    44    0     0  22000      0 --:--:-- --:--:-- --:--:-- 22000
* Connection #0 to host localhost left intact
{"relation_tuples":[],"next_page_token":""}

(irrelevant if all your tests are running against a single instance) are you positive the server started without port conflicts? I generally have trouble with ports vs unit testing, where subsequent tests’ servers can’t start becasue the port is occupied by a previous run. this creates flaky races where test b hit test a’s port before a was closed, but test c errors because srv a is since closed, but srv c failed to start becasue srv a was running on port xxx at the time.

It's flaky enough on CI that it's hard to say there, but I've observed it locally (running from launchd on startup) where keto will be listening on that port ei

lsof

shows the following, but ill still get 404s from known good keto urls

> lsof -i :4468
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
keto    13377 nick   33u  IPv6 0x9d2250f89fcca365      0t0  TCP *:4468 (LISTEN)

is the 404 permanent or just for a short time after the start?

v0.10.0-alpha.0

Hm I'll try to reproduce, is there anything I should set specifically? What is the body you get? And since what version is it happening?

I think it's an empty body, the real way I've noticed the failure is lots of 404s from read endpoints, not so many from write endpoints this request returning 404 is indicative of the failures I've seen:

curl -v -X GET '<http://localhost:4468/relation-tuples>'

(used a different port for our test env but I've seen it on both)

I have an idea what it could be, I'll investigate tomorrow

@steep-lamp-91158 I was wondering if there were any updates or if you'd like me to file an issue on github or somewhere else?

I cannot reproduce unfortunately... Are you sure there is at any point only one server running? Could it be that there is two instances, both trying to bind to the same port?

I'm still seeing this happen, although rarely, so I ran our CI process on a 30 minute loop to get a better sense of the frequency. I realize that as a report from some random person this is not the best/most trustworthy data, so I just wanted to say thanks for taking a look at it so far. Are there any logs or things that would be helpful for diagnosing? I'm a bit concerned that this is happening in CI and locally as we're working on migrating our whole authentication system over to keto.

I'd really like to help but I cannot reproduce the issue... It would really help to get a reproducible case. I'm also wondering whether it is an actual Keto issue as I never saw it in our CI or production system. Could it be related to your CI setup? Eg when the previous run is not completely torn down? Maybe you can share how you start the server and log the PID of it, then you could verify in the failing run using

lsof -i :4466

whether it is the same PID on that port.

Thanks! I really appreciate it, Ill try and follow up with our CI provider. My understanding was that the instances run in docker so they're fresh every time. We run the database similarly and haven't seen similar issues with it. The thing that makes me think it's something in keto is that keto responds with a 404, which indicates to me that the server is running and even doing HTTP correctly (otherwise we'd get a timeout or connection failure)

maybe https://github.com/ory/keto/pull/1107 will solve this for you, we have only seen a flake in unit tests though