precommit hook to validate + autogenerate index/index.json master index file

.husky/pre-commit

@@ -0,0 +1,8 @@
+# 1. Run your pnpm script
+pnpm run reindex
+
+# 2. If that script caused changes, fail the commit
+if ! git diff --quiet; then
+  echo "ERROR: Regenerated file changed. Please run 'pnpm reindex', add 'index/index.json' and commit again."
+  exit 1
+fi

README.md

@@ -1,31 +1,58 @@
 Sequence Token Directory
 ========================
 
-Token directory that contains a list of almost all ERC-20, ERC-721 and ERC-1155 tokens.
+Token directory that contains a comprehensive list of ERC-20, ERC-721, ERC-1155 and other contracts.
+
+**NOTES:**
+* The [./index/index.json](./index/index.json) is an auto-generated file that is a master index of all ./index/**/* contents
+including chain names, chain ids, file names, and sha256 hashes of the file contents. This file
+is perfect for using as the primary index of this repo, and when syncing contents you can traverse this
+index file and also compare the sha256 hash if the file has changed.
+* The [./index/deprecated.json](./index/deprecated.json) is a manually maintained file which lists all folders which are deprecated
+and as a result the files will be labelled as deprecated in the master index.json.
+* The [./index/external.json](./index/external.json) is a manually maintained file of external token list sources which are synced
+and downloaded to the [./index/_external](./index/_external) folder. We store the contents here to ensure data integrity,
+and we also compute and include these files in the master index.json.
+
+**REMINDERS:**
+* `pnpm reindex` is automatically called as a pre-commit hook anytime an entry it changed. You may also
+call it manually if you like.
+* `pnpm sync-external` must be called manually periodically to ensure we have the latest contents, this
+script is not run automatically.
+
+
+## Setup 
+
+* `pnpm install` will setup your local tools
+* `pnpm reindex` to reindex the token directory master index.json, but see notes above, as this
+is also automatically called as a pre-commit hook.
+* `pnpm sync-external` to sync ./index/external.json files to local ./index/_external/ folder.
 
 ## Token List Formats
 
-The ERC-20 token lists present in this repository follow the [Uniswap Token List Schema](https://github.com/Uniswap/token-lists). The original list was populated using [Coingecko](https://www.coingecko.com/en)'s erc20 token list [CoinGecko@95.1.0](https://tokens.coingecko.com/uniswap/all.json). Token description and links are taken from Coingecko's API.
+The ERC-20 token lists present in this repository follow the [Uniswap Token List Schema](https://github.com/Uniswap/token-lists). The original list was populated using [Coingecko](https://www.coingecko.com/en)'s erc20 token list [CoinGecko](https://tokens.coingecko.com/uniswap/all.json). Token description and links are taken from Coingecko's API.
+
+The ERC-721 and ERC-1155 token lists present in this repository follow the [Sequence Collectible List Schema](https://github.com/0xsequence/collectible-lists). 
 
-The ERC-721 and ERC-1155 token lists present in this repository follow the [Sequence Collectible List Schema](https://github.com/0xsequence/collectible-lists). The original list was populated using [Dune Analytics](https://www.duneanalytics.com/) via the query [#16838](https://explore.duneanalytics.com/queries/16838). Token description and links were taken from [OpenSea](https://opensea.io/)'s API.
 
-## How to Add or Update Your Token
+## How to Add or Update Your Token / Contract
 
 If a token is missing entirely, or contains incorrect or missing information, please stick to the following procedure;
 
-1. Fork the current Token Directory repository
-2. Add your token in the `tokens` array in the correct file in the [src/registry/](https://github.com/0xsequence/token-directory/tree/main/src/registry) folder
-   e.g. Registring a new ERC-721 token on Polygon should be done by adding entry [here](https://github.com/0xsequence/token-directory/blob/master/index/polygon/erc721.json).
-3. [Open a PR](https://github.com/0xsequence/token-directory/compare) comparing the main branch with your fork
-4. In the PR, add an explanation if this PR is for an existing token that needs to be updated
+1. Fork this repository
+2. git clone, then: `pnpm install` to setup local tools
+3. Add your entry directly inside of `./index/<chain>/<standard>.json`
+4. [Open a PR](https://github.com/0xsequence/token-directory/compare) comparing the master branch with your fork
+5. In the PR, add an explanation if this PR is for an existing token that needs to be updated
 
 
 ## Formats
+
 Depending on the standard, your token entries should respect the following format:
 
 ### ERC20
 
-See [here](https://github.com/0xsequence/token-directory/blob/main/index/mainnet/erc20.json) for examples.
+See [here](https://github.com/0xsequence/token-directory/blob/master/index/mainnet/erc20.json) for examples.
 
 ```typescript
 {
@@ -44,7 +71,7 @@ See [here](https://github.com/0xsequence/token-directory/blob/main/index/mainnet
 
 ### ERC721 and ERC1155
 
-See [here](https://github.com/0xsequence/token-directory/blob/main/index/mainnet/erc721.json) for erc721 and [here](https://github.com/0xsequence/token-directory/blob/main/index/mainnet/erc1155.json) for erc1155 examples.
+See [here](https://github.com/0xsequence/token-directory/blob/master/index/mainnet/erc721.json) for erc721 and [here](https://github.com/0xsequence/token-directory/blob/master/index/mainnet/erc1155.json) for erc1155 examples.
 
 ```typescript
 {