feat(commoncrawl-CC-MAIN-2025-43-draft/metadata.json): add remaining RecordSets

[handecelikkanat]

This PR add remaining 6 RecordSets:
- wat-records
- wet-records
- robotstxt-records
- non200responses-records
- cc-index-records
- cc-index-table-records

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[handecelikkanat]

@mqzhou-dev I added the remaining 6 RecordSets, do they look alright to you?

---

EDIT: Resolved - we dont have this feature yet.

Btw - did your previous PR (#1001) allow adding prefixes to the urls read in by read_lines? Like adding the fixed prefix "http://data.commoncrawl.org/" to the left of each read line? I am not sure if this is left for future or already added in :)
Asking in order to add to our metadata if it can already be represented.

[handecelikkanat]

@mqzhou-dev @benjelloun

Should this includes be of a url-like, rather than .*warc.gz?

Because in the associated record set (here), the type is url:

  "recordSet": [
    {
      "@id": "warc-records",
      "@type": "cr:RecordSet",
      "field": [
        {
          "@id": "warc-records/url",
          ...
          "dataType": "sc:URL",
...

So manifest FileSet including some urls - not the actual files.

Did I get it correct this time? 😆