Storing data to db and excess tokens loaded into context window

[c12i]

In an attempt to reduce the number of tokens loaded into the LLM's context window, it is becoming necessary to efficiently query the data from a database, instead of loading it on demand via web scraping

Take this tool call, for example, in response to this prompt: "Can we get Jalang'o's, Oscar Sudi's, Babu Owino, and Esther Passaris's profiles. Compare their contributions, bills, and activities." The following steps will be done by the LLM:

Fetch All Members (all houses or by house) -> Load the tokens into context -> Use the data from the loaded members to find the queried members -> Separately fetch each member's profile and load that into context

This simple query bloats the context window with:

• Full members list (very long, potentially over 3k lines of JSON)
• Separate member profiles (range between 60-200 lines of JSON, can rise to up to 5k lines of JSON if the full profile, including all_activity, and all_bills params are requested) (more tokens loaded to context) - LLM will usually optimize this and won't load all activity/ bills unless explicitly requested though

This simple conversation eats up so much context that attempting further queries is practically not possible, maybe with the odd chance that compaction is done successfully (with Claude), or if your LLM supports much larger context windows

Nevertheless, this is a problem that needs to be addressed. One way I see working around this is to read from a database rather than scraping on demand. At least this will shave off context consumed by loading the full list of members, and will provide an avenue for potential full-text search capabilities to allow users to search for specific terms across multiple sittings.

The choice of database is important, as it needs to be future-proof enough to allow adding further capabilities like full-text search as described before. How the database would also be updated with new data also needs to be determined, though this probably won't be as big a problem.

[c12i]

If a database is going to be used, it also makes sense to ensure that each stored sitting is also summarized (using a local LLM for cost savings). This should give the LLM the flexibility of only loading summaries when attempting to query from sittings across a wide date range, although important context might be lost in the process.

Note that the hansard's (from Mzalendo's democracy tools) do actually contain summaries, but they are too brief and don't really provide sufficient info on key details of the session, so these can't really be reliably used in their current state.