fix(llms): curated link list, llms-full.txt from source, fix install URL#471
fix(llms): curated link list, llms-full.txt from source, fix install URL#471billlevine wants to merge 14 commits intomainfrom
Conversation
github-actions
bot
added
the
documentation
This commit adds the generations of llms.txt (more robust than what was there) as well a single docs-content.txt that can be fed to an agent or answer engine to set context of how to work with Flox. This does add a Makefile (sorry) because I needed something to run multiple steps when building the documentaiton site. The README has been updated to explain Makefile usage (and it's completely optional). The tools directory is also new, with a small python file to generate the AI indexing files. The robots.txt that helps the crawlers know where to work is maintained in the flox/floxwebsite repo.
- Replace auto-generated 88-page sitemap with curated ~30 key links - Add llms-full.txt generated from source Markdown (not HTML parsing) - Rename docs-content.txt to llms-full.txt (standard convention) - Fix URL generation to strip /index suffixes - Explicitly correct install URL: flox.dev/download, not install.flox.dev Co-authored-by: Michael Stahnke <stahnma@gmail.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- Swap gnumake for just in the Flox environment manifest - Replace Makefile with a justfile (same recipes: dev, build, clean) - Update README to reference just commands Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
billlevine
force-pushed
the
ai-15-llms-txt-rework
branch
from
4924e3e to
c8c1572
Compare
cp -R ./site/* already covers llms.txt and llms-full.txt Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
These files were accidentally swept in via git add -A. They belong in a separate commit once lint issues are resolved. Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- Use `flox activate -c` in examples (preferred form per current docs) - Keep `flox activate --` in critical rule (both are valid non-interactive) - Fix `is_daemon` -> `is-daemon` (correct manifest TOML key) - Fix typo: "the you are" -> "you are" Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Adds an Installation section that: - Explicitly calls out that curl|bash and install.flox.dev do not exist - Covers macOS (Homebrew), Debian/Ubuntu (.deb), RPM (.rpm) - Points agents to flox.dev/download for package downloads Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Raw source markdown contains {{ FLOX_VERSION }} and {{ FLOX_PUBLIC_KEY }}
placeholders that MkDocs resolves during build. When concatenated
directly, these produce invalid URLs (e.g. download links truncated
at 'flox-') that fail link checking.
Read FLOX_VERSION from environment (set by flox activate hook) and
hardcode FLOX_PUBLIC_KEY (static value from mkdocs.yml) to produce
valid resolved content in llms-full.txt.
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
These are example URLs in code blocks (manifest env var examples, k8s config examples) — not real links. Add broad exclusions so they are skipped regardless of which file lychee scans. Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- auth.flox.dev/oauth/token: POST-only OAuth endpoint, GET returns 404 by design; appears in curl example in organizations.md - nix.dev/manual/nix/2.17/: versioned nix docs anchor referenced in manifest.toml docs; surfaced by llms-full.txt plain-text scanning These URLs are in code block examples and not meant to be navigated. Lychee skips them in HTML but treats them as links in plain text. Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- cache.flox.dev root: Nix binary cache CDN, root returns 404 but the service is valid; appears in nix.conf code block examples - downloads.flox.dev/by-env/stable/rpm: yum repo URL in installing-from-repo.md, not a web page; directory returns 404 Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
Note on approach: The lychee exclusions added in this PR are a side effect of generating The cleaner long-term fix would be to generate That said, this PR is intentionally pragmatic — we're likely moving to Mintlify, which has built-in Via Forge (interactive) • a3060b22 |
Both -c and -- are valid but behave differently: - -c: subshell with full hooks/profile (preferred for agents) - --: exec mode, no profile scripts Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
This looks like a pragmatic win to me! |
garbas
left a comment
garbas
left a comment
There was a problem hiding this comment.
Wouldn't it be easier to use the plugin for that?
1 - yes, if I had known about it :) I could use the plugin, and either ignore the guidance of having the curated instructions, or post-process to include it, but I'm not sure it's that much simpler without tradeoffs of the existing. |
4 participants
Supersedes #357 — built on Michael's branch with targeted fixes.
Thanks @stahnma for the original structure — kept the Python script approach,
CI integration, and the Key Terms / Quick Reference sections.
What changed
~31 key pages (all tutorials, all language guides, core concepts, getting
started). Less is more for llms.txt.
llms-full.txt: renamed fromdocs-content.txt(standard convention)and switched from fragile HTML regex extraction to concatenating source
Markdown files directly — simpler and more reliable.
flox.dev/download, notinstall.flox.dev(which does not exist).Addresses AI-13.
/indexsuffixes from generated links.## Optionalsection: K8s intro and Known Issues are marked optionalper the llms.txt spec (tools can drop these under context pressure).
justinstead ofmake: replacedgnumake/Makefilewithjust/justfileas the command runner. Same recipes:dev,build,clean.llms.txtandllms-full.txt(already covered bycp -R ./site/*).Why curated over auto-generated
The llms.txt spec intends a navigation index, not a sitemap dump.
Practitioners cite Firebase and MongoDB as cautionary tales of
over-generated files. The Stripe pattern (instructions first, curated
links) is what we're following here.
Closes #357
Co-authored-by: Michael Stahnke stahnma@gmail.com