Shape log size for high-churn tables

[JarvisH]

For tables which have a lot of activity with thousands of items added daily, and those items being subjected to n update operations, how does this affect shape log size? I understand compaction was already implemented and LRU GC is being considered, but my initial understanding is the shape logs will grow indefinitely. How to approach this issue for long-lived shapes?

[KyleAMathews]

Compaction was kinda implemented behind an experimental flag but it's proved a much harder challenge than we expected and it'll probably be removed.

Generally the solution we recommend is you add a cache buster key to your query array that updates every so often to force a new shape e.g. this returns a string which updates once a week:

function getUTCWeekAndYear() {
  const now = new Date();
  
  // Create UTC date at midnight
  const utc = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate()));
  
  // ISO week: week 1 contains the first Thursday of the year
  // Set to nearest Thursday: current date + 4 - current day (Mon=1, Sun=7)
  const dayNum = utc.getUTCDay() || 7; // Convert Sun=0 to Sun=7
  utc.setUTCDate(utc.getUTCDate() + 4 - dayNum);
  
  // Get first day of that year
  const yearStart = new Date(Date.UTC(utc.getUTCFullYear(), 0, 1));
  
  // Calculate week number: ceil(days since year start / 7)
  const weekNum = Math.ceil(((utc - yearStart) / 86400000 + 1) / 7);
  
  return {
    week: weekNum,
    year: utc.getUTCFullYear() // ISO week-year (may differ from calendar year at boundaries)
  };
}

[JarvisH]

Thank you for the quick reply. What exactly is meant by "query array"? Any previous shapes no longer in use would need to be manually removed I assume?

[JarvisH]

Trying to answer my own question: a cache busting param could be added like this:

   // Using tanstack/db:
    shapeOptions: {
      url: 'http://localhost:3000/v1/shape',
      params: {
        table: 'mytable',
        where: '...',
        cachebuster: getUTCWeekAndYear
      }
    }

And to clean up unused shapes, a backend job would simply delete any shapes older than x number of weeks?

As far as I could gather from the docs, even if an older shape was in use by a few clients, deleting it would not pose a problem as the tanstack/db client would simply refetch from scratch, is that correct?

[KyleAMathews]

yeah a background job could clean this up. We also have a way to set the max number of shapes on a server which will also cleanup the oldest shapes. And yeah, if a shape is deleted and someone asks for it, it'll just be recreated.