Download Hugging Face datasets for text classification tutorials (#1288)

* use real dataset for domain tutorial

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* ruff

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* update content type classifier notebook

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* update fineweb notebook

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* update fineweb mixtral notebook

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* update fineweb nemotron notebook

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* update prompt task complexity tutorial

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* update multilingual domain tutorial

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* update content type with text_field=text_field

* update domain with text_field=text_field

* update fineweb with text_field=text_field

* update fineweb mixtral with text_field=text_field

* update fineweb nemotron with text_field=text_field

* update quality notebook

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

---------

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

Author: sarahyurick

tutorials/text/distributed-data-classification/aegis-classification.ipynb

@@ -263,7 +263,7 @@
     "\n",
     "```python\n",
     "pipeline = Pipeline(name=\"full_pipeline\")\n",
-    "pipeline.add_stage(read_stage)                # reader (JSONL/S3/etc.)\n",
+    "pipeline.add_stage(read_stage)                # reader (JSONL/Parquet)\n",
     "pipeline.add_stage(lang_id_stage)             # optional: language filter\n",
     "pipeline.add_stage(classifier_stage)          # classifier\n",
     "pipeline.add_stage(write_stage)               # writer (JSONL/Parquet)\n",
@@ -288,7 +288,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Since the pipeline ran to completion and the result was written to a JSONL file, we can shut down the Ray cluster with:"
+    "Since the pipeline ran to completion and the result was written to disk, we can shut down the Ray cluster with:"
    ]
   },
   {
@@ -388,7 +388,7 @@
    ],
    "source": [
     "# For simplicity, we take the first written file from the writer stage\n",
-    "# In real pipelines, the writer may return multiple files (shards) or objects\n",
+    "# In real pipelines, adjust as needed\n",
     "result_file = result[0].data[0]\n",
     "\n",
     "result_df = pd.read_json(result_file, lines=True)\n",

tutorials/text/distributed-data-classification/content-type-classification.ipynb

@@ -249,7 +249,7 @@
     "\n",
     "```python\n",
     "pipeline = Pipeline(name=\"full_pipeline\")\n",
-    "pipeline.add_stage(read_stage)                # reader (JSONL/S3/etc.)\n",
+    "pipeline.add_stage(read_stage)                # reader (JSONL/Parquet)\n",
     "pipeline.add_stage(lang_id_stage)             # optional: language filter\n",
     "pipeline.add_stage(classifier_stage)          # classifier\n",
     "pipeline.add_stage(write_stage)               # writer (JSONL/Parquet)\n",
@@ -274,7 +274,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Since the pipeline ran to completion and the result was written to a JSONL file, we can shut down the Ray cluster with:"
+    "Since the pipeline ran to completion and the result was written to disk, we can shut down the Ray cluster with:"
    ]
   },
   {
@@ -355,7 +355,7 @@
    ],
    "source": [
     "# For simplicity, we take the first written file from the writer stage\n",
-    "# In real pipelines, the writer may return multiple files (shards) or objects\n",
+    "# In real pipelines, adjust as needed\n",
     "result_file = result[0].data[0]\n",
     "\n",
     "result_df = pd.read_json(result_file, lines=True)\n",