What large-scale data or ML challenges are you facing?

[c-dickens]

Hi all 👋

It's been exciting to see the progress being made on the Rust library—great work to everyone involved!
We're always trying to learn where sketching might be a good fit, so we wanted to ask:

Problems or applications.
Are there large-scale data or ML challenges you're facing where you're not sure if sketching could help? We'd love to hear about them—even if you're unsure whether sketching applies, that's exactly what we want to explore.

New sketches.
Are there specific sketching algorithms you'd like to see implemented but haven't found good library support for?

This could be Rust-specific or something that might start in another language. No idea is too niche—sometimes the most interesting directions come from domain-specific problems.
Thanks!
Charlie

[PsiACE]

A common requirement in cache replacement algorithms is tracking element frequency and recency. Frequency can be directly computed using countmin, which has been proven effective in LFU-based algorithms, including variants with windows and decay. However, I'm uncertain about how to properly design and utilize a sketch to track recency. Previously, I calculated it by combining a countmin sketch with the length of a virtual queue.

[PavelVesely]

A common requirement in cache replacement algorithms is tracking element frequency and recency. Frequency can be directly computed using countmin, which has been proven effective in LFU-based algorithms, including variants with windows and decay. However, I'm uncertain about how to properly design and utilize a sketch to track recency. Previously, I calculated it by combining a countmin sketch with the length of a virtual queue.

Tracking recency for cache replacement policies sounds like an interesting problem but I'm not quite sure about the setting or which properties are desirable. If a cache just stores all the items, then it can also store a timestamp (to get recency for LRU) or frequency (for LFU) at the cost of using additional memory proportional to the cache size.

• Is it the goal to use additional memory sublinear in the cache size?
• What kind of error is tolerable for recency? For instance, is it fine to return an item accessed 1 s ago for the last, while there is an item without an access for 1.5 seconds?