add rwkv world tokenizer

[LoganDark]

I had written multiple screens of text here but gpg-agent was unresponsive so .git deleted all my hard work when I tried to re-commit.

Gist is:

• Support world models with --rwkv-world flag.
• Load tokenizer before model to avoid wasting work if tokenizer fails to load. Rwkv python package is slow.

Tokenizer implementation used is my own, BlinkDL/ChatRWKV#137. It outperforms everything else, but hasn't been accepted upstream, so I just pasted it. It'll be fine.

Notes:

• Rwkv package not installed by windows auto installer, had to manually enter conda env and use pip.
• Can't load pth files by web interface due to UI can no longer load RWKV .pth models #2314, falsely closed by retarded stale bot, which is useless and counterproductive. But I digress. When it's possible to load from the web interface again, there should be a way to specify whether to use the world tokenizer there, so you can switch between raven and world models freely.
• Could autodetect the type of model perfectly by checking size of emb tensor, which represents vocab size, raven is 50277 and world is 65536, but can't do that without loading the whole thing into memory because pytorch is the world's worst library and can't read a single tensor from a pth file. Very bad idea as rwkv already loads the entire file due to pytorch's brain death, loading it a second time would be insane.

Sorry, I wrote 3-4x more than this last time, but I am not doing that again, because git overwrote it permanently, because it is a stupid piece of garbage. Hope this is an apt description.

Please enter the commit message for your changes. Lines starting

with '#' will be ignored, and an empty message aborts the commit.

On branch rwkv-world

Changes to be committed:

modified: README.md

modified: modules/RWKV.py

modified: modules/models.py

modified: modules/shared.py

[oobabooga]

RWKV can now be run through the transformers library. The RWKV code here is kept for legacy purposes and is no longer maintained.

[LoganDark]

The RWKV code here is kept for legacy purposes

ty. I use this code in projects that do not support transformers anyway, so this is reasonable.