I made good on my promise to show the stats over time for the various types of requests. It's not possible to glean too much from the data, but there is one really interesting thing that I'm still confused about.
Let's start with some notes:
The Y-axis is the number of incoming requests in a 10-minute interval, intervals starting from the time of the first request (UTC midnight). The entire X-axis duration is 24 hours, so 12 hours is the middle, 8 hours is one third along, and so on. I couldn't figure out how to put numbers according to the hour at the bottom.
All times in this post will be UTC. For readers unfamiliar with UTC, it's a made up time zone that's close to London's, but isn't affected by daylight saving time, and lots of computers (but not all!) like to use it as a standardised time zone for various reasons. (Here's the UTC time right now.) Of course, people and robots that are sending me requests will be operating in their own time zones, so even if something is in the middle of the night UTC it could be peak business hours elsewhere.
Static content has been removed from the graph because it was a distraction from the point that I actually want to talk about here.
So let's get to it:
The bright pink line is the number of requests for feeds (RSS or Atom). And the shape that this line takes is really, really weird. Yes, I've double and triple checked my code and I'm very confident that it gathered the values correctly.
From between about 1 am and 8 am UTC, only about one third of the robots are online, requesting at a fairly even volume throughout. At exactly 8 am, they all wake up in full force and send requests at a new, fairly constant volume, apart from 9 am and 10 am where there are massive spikes up to 600 requests at the start of both of those hours. There's another smaller spike at 1 pm or thereabouts. For the rest of the day it remains fairly constant, fluctuating a bit, but no other spikes or overall trend. And then finally, at 11 pm, most of the robots go back to sleep and it returns to the same volume as it was in the early morning. The spike at the very left edge of the graph is due to robots getting up for a midnight snack, I guess.
I have literally no idea how to explain this line. I figured it would either be roughly constant (they are robots that are online all the time), or peak at the start of every hour (that's when the robots decide to fetch from my site), or go in a wave up and down over the 24 hours (people turning their computers on and off in the peak time zone), but I definitely didn't expect this.
Does anyone have any idea what could have caused this trend? Robots don't have work days or work hours, robots don't go to sleep at night, and robots probably don't all operate in the same time zone.
Here's the script I used to process the logs if anyone wants to look at it or run it for themselves.