A possible solution to the problem with llms.txt and the all-consuming data collection for gpts/llms

[VonderVuflya]

Hi, I’ve been watching the growing problem of GPT training data collection and how it affects companies. First it was Stack Overflow, now it’s Tailwind. I’m just one person and I’m unlikely to be able to influence this on my own, but it’s still worth trying. Maybe if @adamwathan brings this up, we’ll be able to make progress because you currently have enough media reach to raise this issue for the whole industry.

It seems to me the only fair way to resolve this conflict right now is for all GPT/LLM providers to pay at least some royalties for using the libraries that their models rely on when generating code. And it looks like the only practical way to achieve that is to create a new license—similar to MIT, but with an explicit clause for all LLMs stating that if they use the library, they must pay the author for each use or mention.

It might also be worth adding a license for websites alongside llms.txt something like a license.txt file that repeats the terms for using the site’s data (documentation).

[ProCode-Software]

I think a new license is a good idea. But, should open-source, local-hosted models be exempt from having to pay?

[VonderVuflya]

It is a complicated question: yes and no.
The reason is that, for example, Perplexity uses open models hosted on their own servers in the US (for security and other reasons). So how do you distinguish that from models that developers run locally via Ollama or LM Studio (not an ad)?

I just thought that another good solution, besides a license, would be to have something like llms.txt to help various GPTs use libraries more easily and efficiently. But implement it somehow as extensions. There could be a commercial license for GPT providers and a personal license for users.

As a user, I install LM Studio/Ollama, load any open model, and if I need support for Tailwind CSS or any other library, I install it as an extension.
Overall, an extension format like in VS Code, where there are paid and free extensions, would probably be the fairest approach for authors.