docs: fix outdated content in documentation (#1385)

* docs: fix outdated content in documentation

- Remove outdated etcd references (etcd backend was removed)
- Update version numbers from old versions to v51.0.0
- Fix executor-slots-policy to task-distribution with correct values
- Remove Sled database references from docker.md
- Update kubernetes.md docker tags and log output format
- Fix Python API: Ballista() -> BallistaBuilder()
- Fix scheduler-policy parameter name and default value

Co-Authored-By: Claude Opus 4.5 <[email protected]>

* chore: add CLAUDE.md to .gitignore

Co-Authored-By: Claude Opus 4.5 <[email protected]>

* style: format markdown with prettier

Co-Authored-By: Claude Opus 4.5 <[email protected]>

* docs: add benchmarking section to contributors guide

Link to benchmarks/README.md for TPC-H and performance testing instructions.

Co-Authored-By: Claude Opus 4.5 <[email protected]>

---------

Co-authored-by: Claude Opus 4.5 <[email protected]>

Author: andygrove

.gitignore

@@ -102,3 +102,6 @@ dev/dist
 
 # logs
 logs/
+
+# Claude Code guidance file (local only)
+CLAUDE.md

docs/developer/architecture.md

@@ -60,7 +60,7 @@ The scheduler process implements a gRPC interface (defined in
 | GetJobStatus         | Get the status of a submitted query                                  |
 | RegisterExecutor     | Executors call this method to register themselves with the scheduler |
 
-The scheduler can run in standalone mode, or can be run in clustered mode using etcd as backing store for state.
+The scheduler currently uses in-memory state storage.
 
 ## Executor Process
 

docs/source/contributors-guide/architecture.md

@@ -80,9 +80,6 @@ plan or a SQL string. The scheduler then creates an execution graph, which conta
 stages (pipelines) that can be scheduled independently. This process is explained in detail in the Distributed
 Query Scheduling section of this guide.
 
-It is possible to have multiple schedulers running with shared state in etcd, so that jobs can continue to run
-even if a scheduler process fails.
-
 ### Executor
 
 The executor processes connect to a scheduler and poll for tasks to perform. These tasks are physical plans in

docs/source/contributors-guide/development.md

@@ -65,3 +65,14 @@ cargo test
 cd examples
 cargo run --example standalone_sql --features=ballista/standalone
 ```
+
+## Benchmarking
+
+For performance testing and benchmarking with TPC-H and other datasets, see the [benchmarks README](../../../benchmarks/README.md).
+
+This includes instructions for:
+
+- Generating TPC-H test data
+- Running benchmarks against DataFusion and Ballista
+- Comparing performance with Apache Spark
+- Running load tests

docs/source/user-guide/cli.md

@@ -71,7 +71,7 @@ It is also possible to run the CLI in standalone mode, where it will create a sc
 ```bash
 $ ballista-cli
 
-Ballista CLI v8.0.0
+Ballista CLI v51.0.0
 
 > CREATE EXTERNAL TABLE foo (a INT, b INT) STORED AS CSV LOCATION 'data.csv';
 0 rows in set. Query took 0.001 seconds.

docs/source/user-guide/configs.md

@@ -96,14 +96,13 @@ manage the whole cluster are also needed to be taken care of.
 _Example: Specifying configuration options when starting the scheduler_
 
 ```shell
-./ballista-scheduler --scheduler-policy push-staged --event-loop-buffer-size 1000000 --executor-slots-policy
-round-robin-local
+./ballista-scheduler --scheduler-policy push-staged --event-loop-buffer-size 1000000 --task-distribution round-robin
 ```
 
-| key                                          | type   | default     | description                                                                                                                                                                     |
-| -------------------------------------------- | ------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| scheduler-policy                             | Utf8   | pull-staged | Sets the task scheduling policy for the scheduler, possible values: pull-staged, push-staged.                                                                                   |
-| event-loop-buffer-size                       | UInt32 | 10000       | Sets the event loop buffer size. for a system of high throughput, a larger value like 1000000 is recommended.                                                                   |
-| executor-slots-policy                        | Utf8   | bias        | Sets the executor slots policy for the scheduler, possible values: bias, round-robin, round-robin-local. For a cluster with single scheduler, round-robin-local is recommended. |
-| finished-job-data-clean-up-interval-seconds  | UInt64 | 300         | Sets the delayed interval for cleaning up finished job data, mainly the shuffle data, 0 means the cleaning up is disabled.                                                      |
-| finished-job-state-clean-up-interval-seconds | UInt64 | 3600        | Sets the delayed interval for cleaning up finished job state stored in the backend, 0 means the cleaning up is disabled.                                                        |
+| key                                          | type   | default     | description                                                                                                                |
+| -------------------------------------------- | ------ | ----------- | -------------------------------------------------------------------------------------------------------------------------- |
+| scheduler-policy                             | Utf8   | pull-staged | Sets the task scheduling policy for the scheduler, possible values: pull-staged, push-staged.                              |
+| event-loop-buffer-size                       | UInt32 | 10000       | Sets the event loop buffer size. for a system of high throughput, a larger value like 1000000 is recommended.              |
+| task-distribution                            | Utf8   | bias        | Sets the task distribution policy for the scheduler, possible values: bias, round-robin, consistent-hash.                  |
+| finished-job-data-clean-up-interval-seconds  | UInt64 | 300         | Sets the delayed interval for cleaning up finished job data, mainly the shuffle data, 0 means the cleaning up is disabled. |
+| finished-job-state-clean-up-interval-seconds | UInt64 | 3600        | Sets the delayed interval for cleaning up finished job state stored in the backend, 0 means the cleaning up is disabled.   |

docs/source/user-guide/deployment/docker-compose.md

@@ -39,15 +39,11 @@ This should show output similar to the following:
 ```bash
 $ docker-compose up
 Creating network "ballista-benchmarks_default" with the default driver
-Creating ballista-benchmarks_etcd_1 ... done
 Creating ballista-benchmarks_ballista-scheduler_1 ... done
 Creating ballista-benchmarks_ballista-executor_1  ... done
-Attaching to ballista-benchmarks_etcd_1, ballista-benchmarks_ballista-scheduler_1, ballista-benchmarks_ballista-executor_1
-ballista-executor_1   | [2021-08-28T15:55:22Z INFO  ballista_executor] Running with config:
-ballista-executor_1   | [2021-08-28T15:55:22Z INFO  ballista_executor] work_dir: /tmp/.tmpLVx39c
-ballista-executor_1   | [2021-08-28T15:55:22Z INFO  ballista_executor] concurrent_tasks: 4
-ballista-scheduler_1  | [2021-08-28T15:55:22Z INFO  ballista_scheduler] Ballista v0.12.0 Scheduler listening on 0.0.0.0:50050
-ballista-executor_1   | [2021-08-28T15:55:22Z INFO  ballista_executor] Ballista v0.12.0 Rust Executor listening on 0.0.0.0:50051
+Attaching to ballista-benchmarks_ballista-scheduler_1, ballista-benchmarks_ballista-executor_1
+ballista-scheduler_1  | INFO ballista_scheduler: Ballista v51.0.0 Scheduler listening on 0.0.0.0:50050
+ballista-executor_1   | INFO ballista_executor: Ballista v51.0.0 Rust Executor listening on 0.0.0.0:50051
 ```
 
 The scheduler listens on port 50050 and this is the port that clients will need to connect to.

docs/source/user-guide/deployment/docker.md

@@ -67,13 +67,10 @@ Run `docker logs CONTAINER_ID` to check the output from the process:
 
 ```
 $ docker logs a756055576f3
-2024-02-03T14:49:47.904571Z  INFO main ThreadId(01) ballista_scheduler::cluster: Initializing Sled database in temp directory
-
-2024-02-03T14:49:47.924679Z  INFO main ThreadId(01) ballista_scheduler::scheduler_process: Ballista v0.12.0 Scheduler listening on 0.0.0.0:50050
-2024-02-03T14:49:47.924709Z  INFO main ThreadId(01) ballista_scheduler::scheduler_process: Starting Scheduler grpc server with task scheduling policy of PullStaged
-2024-02-03T14:49:47.925261Z  INFO main ThreadId(01) ballista_scheduler::cluster::kv: Initializing heartbeat listener
-2024-02-03T14:49:47.925476Z  INFO main ThreadId(01) ballista_scheduler::scheduler_server::query_stage_scheduler: Starting QueryStageScheduler
-2024-02-03T14:49:47.925587Z  INFO tokio-runtime-worker ThreadId(47) ballista_core::event_loop: Starting the event loop query_stage
+INFO ballista_scheduler::scheduler_process: Ballista v51.0.0 Scheduler listening on 0.0.0.0:50050
+INFO ballista_scheduler::scheduler_process: Starting Scheduler grpc server with task scheduling policy of PullStaged
+INFO ballista_scheduler::scheduler_server::query_stage_scheduler: Starting QueryStageScheduler
+INFO ballista_core::event_loop: Starting the event loop query_stage
 ```
 
 ### Start Executors
@@ -99,11 +96,11 @@ Use `docker logs CONTAINER_ID` to check the output from the executor(s):
 
 ```
 $ docker logs fb8b530cee6d
-2024-02-03T14:50:24.061607Z  INFO main ThreadId(01) ballista_executor::executor_process: Running with config:
-2024-02-03T14:50:24.061649Z  INFO main ThreadId(01) ballista_executor::executor_process: work_dir: /tmp/.tmpAkP3pZ
-2024-02-03T14:50:24.061655Z  INFO main ThreadId(01) ballista_executor::executor_process: concurrent_tasks: 48
-2024-02-03T14:50:24.063256Z  INFO tokio-runtime-worker ThreadId(44) ballista_executor::executor_process: Ballista v0.12.0 Rust Executor Flight Server listening on 0.0.0.0:50051
-2024-02-03T14:50:24.063281Z  INFO tokio-runtime-worker ThreadId(47) ballista_executor::execution_loop: Starting poll work loop with scheduler
+INFO ballista_executor::executor_process: Running with config:
+INFO ballista_executor::executor_process: work_dir: /tmp/.tmpAkP3pZ
+INFO ballista_executor::executor_process: concurrent_tasks: 48
+INFO ballista_executor::executor_process: Ballista v51.0.0 Rust Executor Flight Server listening on 0.0.0.0:50051
+INFO ballista_executor::execution_loop: Starting poll work loop with scheduler
 ```
 
 ## Connect from the CLI

docs/source/user-guide/deployment/kubernetes.md

@@ -48,10 +48,10 @@ To create the required Docker images please refer to the [docker deployment page
 Once the images have been built, you can retag them and can push them to your favourite Docker registry.
 
 ```bash
-docker tag apache/datafusion-ballista-scheduler:0.12.0 <your-repo>/datafusion-ballista-scheduler:0.12.0
-docker tag apache/datafusion-ballista-executor:0.12.0 <your-repo>/datafusion-ballista-executor:0.12.0
-docker push <your-repo>/datafusion-ballista-scheduler:0.12.0
-docker push <your-repo>/datafusion-ballista-executor:0.12.0
+docker tag apache/datafusion-ballista-scheduler:latest <your-repo>/datafusion-ballista-scheduler:latest
+docker tag apache/datafusion-ballista-executor:latest <your-repo>/datafusion-ballista-executor:latest
+docker push <your-repo>/datafusion-ballista-scheduler:latest
+docker push <your-repo>/datafusion-ballista-executor:latest
 ```
 
 ## Create Persistent Volume and Persistent Volume Claim
@@ -139,7 +139,7 @@ spec:
     spec:
       containers:
         - name: ballista-scheduler
-          image: <your-repo>/datafusion-ballista-scheduler:0.12.0
+          image: <your-repo>/datafusion-ballista-scheduler:latest
           args: ["--bind-port=50050"]
           ports:
             - containerPort: 50050
@@ -169,7 +169,7 @@ spec:
     spec:
       containers:
         - name: ballista-executor
-          image: <your-repo>/datafusion-ballista-executor:0.12.0
+          image: <your-repo>/datafusion-ballista-executor:latest
           args:
             - "--bind-port=50051"
             - "--scheduler-host=ballista-scheduler"
@@ -208,13 +208,13 @@ ballista-executor-78cc5b6486-7crdm   0/1     Pending   0          42s
 ballista-scheduler-879f874c5-rnbd6   0/1     Pending   0          42s
 ```
 
-You can view the scheduler logs with `kubectl logs ballista-scheduler-0`:
+You can view the scheduler logs with `kubectl logs ballista-scheduler-<pod-id>`:
 
 ```
-$ kubectl logs ballista-scheduler-0
-[2021-02-19T00:24:01Z INFO  scheduler] Ballista v0.7.0 Scheduler listening on 0.0.0.0:50050
-[2021-02-19T00:24:16Z INFO  ballista::scheduler] Received register_executor request for ExecutorMetadata { id: "b5e81711-1c5c-46ec-8522-d8b359793188", host: "10.1.23.149", port: 50051 }
-[2021-02-19T00:24:17Z INFO  ballista::scheduler] Received register_executor request for ExecutorMetadata { id: "816e4502-a876-4ed8-b33f-86d243dcf63f", host: "10.1.23.150", port: 50051 }
+$ kubectl logs ballista-scheduler-<pod-id>
+INFO ballista_scheduler::scheduler_process: Ballista v51.0.0 Scheduler listening on 0.0.0.0:50050
+INFO ballista_scheduler::scheduler_server::grpc: Received register_executor request for ExecutorMetadata { id: "b5e81711-1c5c-46ec-8522-d8b359793188", host: "10.1.23.149", port: 50051 }
+INFO ballista_scheduler::scheduler_server::grpc: Received register_executor request for ExecutorMetadata { id: "816e4502-a876-4ed8-b33f-86d243dcf63f", host: "10.1.23.150", port: 50051 }
 ```
 
 ## Port Forwarding

docs/source/user-guide/python.md

@@ -117,12 +117,8 @@ The following example demonstrates creating arrays with PyArrow and then creatin
 from ballista import BallistaBuilder
 import pyarrow
 
-# an alias
-# TODO implement Functions
-f = ballista.functions
-
 # create a context
-ctx = Ballista().standalone()
+ctx = BallistaBuilder().standalone()
 
 # create a RecordBatch and a new DataFrame from it
 batch = pyarrow.RecordBatch.from_arrays(
@@ -132,9 +128,10 @@ batch = pyarrow.RecordBatch.from_arrays(
 df = ctx.create_dataframe([[batch]])
 
 # create a new statement
+from datafusion import col
 df = df.select(
-    f.col("a") + f.col("b"),
-    f.col("a") - f.col("b"),
+    col("a") + col("b"),
+    col("a") - col("b"),
 )
 
 # execute and collect the first (and only) batch