Using Tanstack DB without fetching the entire collection

[scottmessinger]

Hi! I really like the look of TanStack DB, but I'm pretty puzzled as to how you work with a collection where it's not feasible to fetch the entire collection at once. In working with an app like Notion, it isn't performant or feasible to sync all the data the user might access when the user loads the page. I can't figure out how that would work in TanStackDB, given the queryFn is designed to fetch the entire collection

How would I use TanStackDB in a Notion-like app where the data the user has access vastly exceeds what's possible to download into their browser?

[craigsmitham]

You would need apply filtering in the query function to ensure that the data that is fetched/synced from the server is only what is needed to display/work with for the use case. In some cases where data may only be hundreds of records, you can perhaps always fetch everything. In other cases, consider the logical boundaries the user will be working within for a particular use case. This could be the scope of a page, aggregate root, etc.

[scottmessinger]

Thanks! One thing I couldn't discern from the examples is when the initial fetch would happen. All the examples show using a livequery, so I imagined the fetch would happen on page load. Is the queryFn called on every query?

[KyleAMathews]

Right now it's as @craigsmitham says, you need your collections to fetch subsets of data — as large as makes sense. The plan is fairly soon to have APIs where this is fairly automatic — see #315 & #343

[KyleAMathews]

@craigsmitham yup that's a good pattern. Once we have a union operator, you'll be able to trivially merge all the collections together (partitioning needs that too) #58 vs. manually merging.

Am I misreading this?

yes... I'm not sure where you're getting the idea that DB can't load directly from IDs. You can load data however you want. I was just mentioning things we're planning to make lazy loading more ergonomic. But DB can load data from anywhere in any form so go crazy with whatever you want. DB enables some new patterns which we think are really powerful but you can do things the way you're used to with Query still.

[scottmessinger]

@KyleAMathews Thanks! I’m getting this from every single example on the docs site. In every one, the queryFn fetches the entire collection. Not a single example shows the queryFn fetching a document by id from the server. If there’s an example I’m missing, please let me know! It’s possible I’m just blind as a bat and am missing the page that shows how.

If the answer is I can fetch a single document from the server by id (and then fetch another later), how would that be done?

[KyleAMathews]

@craigsmitham explained one way above

[scottmessinger]

For instance:

const todoCollection = createCollection(
  queryCollectionOptions({
    queryKey: ['todos'],
    queryFn: async () => {
      const response = await fetch('/api/todos')
      return response.json()
    }

All the examples fetch the whole collection, hence my initial question and the examples in my examples of apps where this pattern wouldn’t fit.

I think TanStackDB is really neat, but it seems like it requires radically different way of fetching data, specifically over fetching data by default. I’m trying to find out if a more incremental way of fetching data from the server is possible and if so. how.

How would I incrementally fetch a collection by ID from the server? How would I build a blog site that doesn’t require fetching all the blogs on page load and instead fetches blog posts ID by ID as the user navigates pages? If this is possible, I can’t find an example on the docs (but that might be my own fault for not looking at the right place)

[scottmessinger]

I didn’t see your message when I posted that clarification. Sorry for the confusion!

I’ll reread the posts and think about it next week but none of what I’m hearing suggests the traditional data fetching case is straightforward (though it sounds possible).

If it’s easier than it appears, I’d love to see an example on the docs showing how you’d handle a blog site where a user navigates post by post and only bare minimum of blog posts are fetched. I think your point is “this is not as difficult as you’re making it out to be, Scott” which is probably fair. In which case, an example would really help me see it how you do.