- Get crawling to work reliably and show url queue
- Hook up scrapers to graphile worker for schedule
- Add settings for adjusting html cleaning options for which tags, attributes, etc to remove
- Refine resultsGet llm tool to better work like api request and to limit sending large amounts of data through pagination
- Give llm option to work on raw html, etc
- Add html compaction for better html usage in context
- Add selection in any monaco view to be given to llm as part of message context (show in message ui selection)
- Remove db.select queries unless needed, use regex: db\s*\r?\n\s*.select
- Consolidate postgres collation version mismatch logic https://gist.github.com/troykelly/616df024050dd50744dde4a9579e152e