Mirgate all leftover docks from experimental (#2243)

Migrate all remaining docs from experimental

Author: rhlbhatnagar

docs/concepts/datasets.md

@@ -0,0 +1,141 @@
+# Datasets and Experiment Results
+
+When we evaluate AI systems, we typically work with two main types of data:
+
+1. **Evaluation Datasets**: These are stored under the `datasets` directory.
+2. **Evaluation Results**: These are stored under the `experiments` directory.
+
+## Evaluation Datasets
+
+A dataset for evaluations contains:
+
+1. Inputs: a set of inputs that the system will process.
+2. Expected outputs (Optional): the expected outputs or responses from the system for the given inputs.
+3. Metadata (Optional): additional information that can be stored alongside the dataset.
+
+For example, in a Retrieval-Augmented Generation (RAG) system it might include query (input to the system), Grading notes (to grade the output from the system), and metadata like query complexity.
+
+Metadata is particularly useful for slicing and dicing the dataset, allowing you to analyze results across different facets. For instance, you might want to see how your system performs on complex queries versus simple ones, or how it handles different languages.
+
+## Experiment Results
+
+Experiment results include:
+
+1. All attributes from the dataset.
+2. The response from the evaluated system.
+3. Results of metrics.
+4. Optional metadata, such as a URI pointing to the system trace for a given input.
+
+For example, in a RAG system, the results might include Query, Grading notes, Response, Accuracy score (metric), link to the system trace, etc.
+
+## Working with Datasets in Ragas
+
+Ragas provides a `Dataset` class to work with evaluation datasets. Here's how you can use it:
+
+### Creating a Dataset
+
+```python
+from ragas import Dataset
+
+# Create a new dataset
+dataset = Dataset(name="my_evaluation", backend="local/csv", root_dir="./data")
+
+# Add a sample to the dataset
+dataset.append({
+    "id": "sample_1",
+    "query": "What is the capital of France?",
+    "expected_answer": "Paris",
+    "metadata": {"complexity": "simple", "language": "en"}
+})
+```
+
+### Loading an Existing Dataset
+
+```python
+# Load an existing dataset
+dataset = Dataset.load(
+    name="my_evaluation",
+    backend="local/csv",
+    root_dir="./data"
+)
+```
+
+### Dataset Structure
+
+Datasets in Ragas are flexible and can contain any fields you need for your evaluation. Common fields include:
+
+- `id`: Unique identifier for each sample
+- `query` or `input`: The input to your AI system
+- `expected_output` or `ground_truth`: The expected response (if available)
+- `metadata`: Additional information about the sample
+
+### Best Practices for Dataset Creation
+
+1. **Representative Samples**: Ensure your dataset represents the real-world scenarios your AI system will encounter.
+
+2. **Balanced Distribution**: Include samples across different difficulty levels, topics, and edge cases.
+
+3. **Quality Over Quantity**: It's better to have fewer high-quality, well-curated samples than many low-quality ones.
+
+4. **Metadata Rich**: Include relevant metadata that allows you to analyze performance across different dimensions.
+
+5. **Version Control**: Track changes to your datasets over time to ensure reproducibility.
+
+## Dataset Storage and Management
+
+### Local Storage
+
+For local development and small datasets, you can use CSV files:
+
+```python
+dataset = Dataset(name="my_eval", backend="local/csv", root_dir="./datasets")
+```
+
+### Cloud Storage
+
+For larger datasets or team collaboration, consider cloud backends:
+
+```python
+# Google Drive (experimental)
+dataset = Dataset(name="my_eval", backend="gdrive", root_dir="folder_id")
+
+# Other backends can be added as needed
+```
+
+### Dataset Versioning
+
+Keep track of dataset versions for reproducible experiments:
+
+```python
+# Include version in dataset name
+dataset = Dataset(name="my_eval_v1.2", backend="local/csv", root_dir="./datasets")
+```
+
+## Integration with Evaluation Workflows
+
+Datasets integrate seamlessly with Ragas evaluation workflows:
+
+```python
+from ragas import experiment, Dataset
+
+# Load your dataset
+dataset = Dataset.load(name="my_evaluation", backend="local/csv", root_dir="./data")
+
+# Define your experiment
+@experiment()
+async def my_experiment(row):
+    # Process the input through your AI system
+    response = await my_ai_system(row["query"])
+    
+    # Return results for metric evaluation
+    return {
+        **row,  # Include original data
+        "response": response,
+        "experiment_name": "baseline_v1"
+    }
+
+# Run evaluation on the dataset
+results = await my_experiment.arun(dataset)
+```
+
+This integration allows you to maintain a clear separation between your test data (datasets) and your evaluation results (experiments), making it easier to track progress and compare different approaches.