Additional Data Sources

[pgulley]

@rahulbot, you probably are the best person to chime in here- we have the deployment loop active, so we should start thinking about graphs we want to add to this.

In the original pitch document, we have:

• Total Feeds (just a count)
• Total Stories (just a count)
• Web Search Response Time (time histogram)
• Web Search Query volume (time histogram)

I could imagine also wanting to have some green lights that reflect if certain services are active and happy, just a bool for the rss-fetcher, the story-indexer, the web-search, etc.

[philbudne]

Here are a few of my centavos:

- Total Feeds (just a count)

Just the live ones (for some definition of live: has responded with a valid feed document, or has returned new stories in some window of time)? Or ALL (which could be deceptively large), tho I think something on the order of 90K (out of 150K) had returned a story in the past 12 months. Another angle: feed polls done in the last day/hour (could probably be summed in the graphite API from existing data for various choices of result/status).

- Total Stories (just a count)

DONE.

- Web Search Response Time (time histogram)

For ALL queries? That may have a large range/deviation. For some "representative" query? I'd ***LOVE*** to have one that we run every five minutes and is "JUST" hard/complex/big enough to be sensitive to system conditions, and representative of what an interactive user experiences, but not so hard as to cause appreciable load! The ones I currently run from ifill are VERY light weight (one word, one source, one week, and one word, one collection, four weeks)

- Web Search Query volume (time histogram)

ALL requests (including ones rejected by rate limiting), or just those we've actually executed?

I could imagine also wanting to have some green lights that reflect if certain services are active and happy, just a bool for the rss-fetcher, the story-indexer, the web-search, etc.

"colored circles" is pretty much all https://www.githubstatus.com/ shows! I suppose a good definition of "green" is "has no current alerts", but I'd have to investigate if it's possible to get that (it might be possible to have grafana report status changes via http "callouts" to the dashboard API, which saves the status in a disk file).

[rahulbot]

Some notes:

• Total feeds count: I like the idea of that being a "total live feeds"; something timely like "responds HTTP OK in last day" or something.
• web search time: I think I say a Grafana chart of this unit data , so could we average that over last hour or something?
• web query volume: I'd say all requests; showing heavy usage is in our favor for marketing
• Love the "colored circles status" idea, and for each service it would need a definition; put this right at the top

I think that'd be enough of a prototype to show the broader team and solicit a discussion of more ideas 👏🏽

[philbudne]

For @rahulbot 's definition of "total live feeds" I think looking at last_fetch_success and counting rows with system_status = 'Working' or system_status = 'parse error'

rss_fetcher=# select system_status, count(1) from feeds where last_fetch_success >= '2025-04-17' group by system_status order by count desc;
         system_status          | count  
--------------------------------+--------
 Working                        | 109175
 connect timeout                |    638
 read timeout                   |     45
 parse error                    |     29
 HTTP 500 Internal Server Error |     23
 HTTP 406 Not Acceptable        |     18
 connection error               |     11
 unknown hostname               |      9
 SSL error                      |      6
 HTTP 403 Forbidden             |      5
 HTTP 429 Too Many Requests     |      5
 HTTP 404 Not Found             |      4
 HTTP 502 Bad Gateway           |      4
 HTTP 522 <none>                |      3
 temporary DNS error            |      3
 HTTP 503 Service Unavailable   |      2
 fetch error                    |      1
 HTTP 409 Conflict              |      1
 HTTP 503 Backend fetch failed  |      1
 HTTP 504 Gateway Time-out      |      1
 HTTP 526 <none>                |      1
(21 rows)

[philbudne]

I'm not clear on what "web search time" means

[pgulley]

That approach to 'total live feeds' feels really reasonable to me!
"Web search time" isn't something we have a clear method for- so I agree, phil, that canonicalizing the api check process you have running on your crontab is a good idea. As far as representative queries go - I'm not sure if we have a good example middle ground between the naive queries you're running and a big complex research query. We can discuss Monday.

[pgulley]

I'll break out the remaining metrics into their own issues for more granular tracking.