I spent about an hour this evening trying to debug why Bibliogram sometimes breaks nginx on my server. At random times, but somewhere around once per day, new connections to nginx's port hang. All connections to Bibliogram's port directly also hang. nginx apparently uses 100% CPU during this, but Bibliogram doesn't.
This can be fixed instantly by stopping and starting Bibliogram.
This doesn't happen for any other servers I run with nginx and pinski, so it's weird that it would happen for Bibliogram specifically.
Other people who run Bibliogram have not complained about this.
Stopping nginx while in this state makes only some of Bibliogram's endpoints available, which is very very weird. Investigating this further will probably bring me closer to the solution. It seems that endpoints like /api/user and /fragment work but endpoints like / and /u/* don't. Very very odd.
Attempting to access one of those weird endpoints with strace connected does successfully the response data because I can see it in writev, but writev fails with something like
= -1 EAGAIN and I don't understand what this means. After failing it immediately retries. I can see the timeout value in writev counting down with each attempt. This happens many many times per second. Bibliogram apparently uses 100% CPU during this.
Stopping and starting nginx only very temporarily alleviates the issue.
I don't know what a lot of this means, please send help.