Remove sequential simulations and add continuous auth to Polaris benchmarks (#6)

* Remove sequential simulations and add continuous auth

* Remove sequential simulations now that the concurrent simulation
  throughput is configurable (and can be set to 1 until SQL
  implementation is able to keep up with throughput)
* Configure every workload using the configuration file
* Change authentication logic so that OAuth token is refreshed every minute.
  This makes it possible to run a benchmark for longer than the default
  OAuth validity period (1h).  It is useful for use cases like creating
  very large data sets or running longevity tests.

* Code review: use block instead of argument for Gatling actions

Author: pingtimeout

benchmarks/README.md

@@ -6,9 +6,9 @@
   to you under the Apache License, Version 2.0 (the
   "License"); you may not use this file except in compliance
   with the License.  You may obtain a copy of the License at
- 
+
    http://www.apache.org/licenses/LICENSE-2.0
- 
+
   Unless required by applicable law or agreed to in writing,
   software distributed under the License is distributed on an
   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -25,21 +25,28 @@ Benchmarks for the Polaris service using Gatling.
 
 ### Dataset Creation Benchmark
 
-The CreateTreeDataset benchmark creates a test dataset with a specific structure. It exists in two variants:
+The CreateTreeDataset benchmark creates a test dataset with a specific structure:
 
-- `org.apache.polaris.benchmarks.simulations.CreateTreeDatasetSequential`: Creates entities one at a time
-- `org.apache.polaris.benchmarks.simulations.CreateTreeDatasetConcurrent`: Creates up to 50 entities simultaneously
+- `org.apache.polaris.benchmarks.simulations.CreateTreeDataset`: Creates up to 50 entities simultaneously
 
-These are write-only workloads designed to populate the system for subsequent benchmarks.
+This is a write-only workload designed to populate the system for subsequent benchmarks.
 
 ### Read/Update Benchmark
 
-The ReadUpdateTreeDataset benchmark tests read and update operations on an existing dataset. It exists in two variants:
+The ReadUpdateTreeDataset benchmark tests read and update operations on an existing dataset:
+
+- `org.apache.polaris.benchmarks.simulations.ReadUpdateTreeDataset`: Performs up to 20 read/update operations simultaneously
+
+This benchmark can only be run after using CreateTreeDataset to populate the system.
+
+### Read-Only Benchmark
+
+The ReadTreeDataset benchmark is a 100% read workload that fetches a tree dataset in Polaris:
 
-- `org.apache.polaris.benchmarks.simulations.ReadUpdateTreeDatasetSequential`: Performs read/update operations one at a time
-- `org.apache.polaris.benchmarks.simulations.ReadUpdateTreeDatasetConcurrent`: Performs up to 20 read/update operations simultaneously
+- `org.apache.polaris.benchmarks.simulations.ReadTreeDataset`: Performs read-only operations to verify namespaces, tables, and views
+
+This benchmark is intended to be used against a Polaris instance with a pre-existing tree dataset. It has no side effects on the dataset and can be executed multiple times without any issues.
 
-These benchmarks can only be run after using CreateTreeDataset to populate the system.
 
 ## Parameters
 
@@ -117,13 +124,19 @@ workload {
 Run benchmarks with your configuration:
 
 ```bash
-# Sequential dataset creation
-./gradlew gatlingRun --simulation org.apache.polaris.benchmarks.simulations.CreateTreeDatasetSequential \
+# Dataset creation
+./gradlew gatlingRun --simulation org.apache.polaris.benchmarks.simulations.CreateTreeDataset \
+  -Dconfig.file=./application.conf
+
+# Read/Update operations
+./gradlew gatlingRun --simulation org.apache.polaris.benchmarks.simulations.ReadUpdateTreeDataset \
   -Dconfig.file=./application.conf
 
-# Concurrent dataset creation
-./gradlew gatlingRun --simulation org.apache.polaris.benchmarks.simulations.CreateTreeDatasetConcurrent \
+# Read-only operations
+./gradlew gatlingRun --simulation org.apache.polaris.benchmarks.simulations.ReadTreeDataset \
   -Dconfig.file=./application.conf
+
+
 ```
 
 A message will show the location of the Gatling report:

benchmarks/src/gatling/resources/benchmark-defaults.conf

@@ -140,4 +140,38 @@ workload {
   # Number of property updates to perform per individual view
   # Default: 10
   updates-per-view = 10
+
+
+  # Configuration for the ReadTreeDataset simulation
+  read-tree-dataset {
+    # Number of table operations to perform per second
+    # Default: 20
+    table-throughput = 20
+
+    # Number of view operations to perform per second
+    # Default: 10
+    view-throughput = 10
+  }
+
+  # Configuration for the CreateTreeDataset simulation
+  create-tree-dataset {
+    # Number of table operations to perform per second
+    # Default: 20
+    table-throughput = 20
+
+    # Number of view operations to perform per second
+    # Default: 10
+    view-throughput = 10
+  }
+
+  # Configuration for the ReadUpdateTreeDataset simulation
+  read-update-tree-dataset {
+    # Number of operations to perform per second
+    # Default: 100
+    throughput = 100
+
+    # Duration of the simulation in minutes
+    # Default: 5
+    duration-in-minutes = 5
+  }
 }

benchmarks/src/gatling/scala/org/apache/polaris/benchmarks/NAryTreeBuilder.scala

@@ -67,15 +67,14 @@ case class NAryTreeBuilder(nsWidth: Int, nsDepth: Int) {
    *
    * @return The total number of nodes in the tree.
    */
-  val numberOfNodes: Int = {
+  val numberOfNodes: Int =
     // The sum of nodes from level 0 to level d-1 is (n^(d+1) - 1) / (n - 1) if n > 1
     // Else, the sum of nodes from level 0 to level d-1 is d
     if (nsWidth == 1) {
       nsDepth
     } else {
       ((math.pow(nsWidth, nsDepth) - 1) / (nsWidth - 1)).toInt
     }
-  }
 
   /**
    * Returns a range of ordinals for the nodes on the last level of a complete n-ary tree.

benchmarks/src/gatling/scala/org/apache/polaris/benchmarks/parameters/BenchmarkConfig.scala

@@ -38,13 +38,31 @@ object BenchmarkConfig {
       http.getString("base-url")
     )
 
-    val workloadParams = WorkloadParameters(
-      workload.getDouble("read-write-ratio"),
-      workload.getInt("updates-per-namespace"),
-      workload.getInt("updates-per-table"),
-      workload.getInt("updates-per-view"),
-      workload.getLong("seed")
-    )
+    val workloadParams = {
+      val rtdConfig = workload.getConfig("read-tree-dataset")
+      val ctdConfig = workload.getConfig("create-tree-dataset")
+      val rutdConfig = workload.getConfig("read-update-tree-dataset")
+
+      WorkloadParameters(
+        workload.getDouble("read-write-ratio"),
+        workload.getInt("updates-per-namespace"),
+        workload.getInt("updates-per-table"),
+        workload.getInt("updates-per-view"),
+        workload.getLong("seed"),
+        ReadTreeDatasetParameters(
+          rtdConfig.getInt("table-throughput"),
+          rtdConfig.getInt("view-throughput")
+        ),
+        CreateTreeDatasetParameters(
+          ctdConfig.getInt("table-throughput"),
+          ctdConfig.getInt("view-throughput")
+        ),
+        ReadUpdateTreeDatasetParameters(
+          rutdConfig.getInt("throughput"),
+          rutdConfig.getInt("duration-in-minutes")
+        )
+      )
+    }
 
     val datasetParams = DatasetParameters(
       dataset.getInt("num-catalogs"),

benchmarks/src/gatling/scala/org/apache/polaris/benchmarks/simulations/ReadUpdateTreeDatasetSequential.scala renamed to benchmarks/src/gatling/scala/org/apache/polaris/benchmarks/parameters/CreateTreeDatasetParameters.scala

@@ -17,25 +17,18 @@
  * under the License.
  */
 
-package org.apache.polaris.benchmarks.simulations
+package org.apache.polaris.benchmarks.parameters
 
-import io.gatling.core.Predef._
-import io.gatling.http.Predef._
-
-import scala.concurrent.duration.DurationInt
-
-class ReadUpdateTreeDatasetSequential extends ReadUpdateTreeDataset {
-  // --------------------------------------------------------------------------------
-  // Build up the HTTP protocol configuration and set up the simulation
-  // --------------------------------------------------------------------------------
-  private val httpProtocol = http
-    .baseUrl(cp.baseUrl)
-    .acceptHeader("application/json")
-    .contentTypeHeader("application/json")
-
-  setUp(
-    authenticate
-      .inject(atOnceUsers(1))
-      .andThen(readWriteScenario.inject(constantUsersPerSec(1).during(5.minutes)))
-  ).protocols(httpProtocol)
+/**
+ * Case class to hold the parameters for the CreateTreeDataset simulation.
+ *
+ * @param tableThroughput The number of table operations to perform per second.
+ * @param viewThroughput The number of view operations to perform per second.
+ */
+case class CreateTreeDatasetParameters(
+    tableThroughput: Int,
+    viewThroughput: Int
+) {
+  require(tableThroughput >= 0, "Table throughput cannot be negative")
+  require(viewThroughput >= 0, "View throughput cannot be negative")
 }

benchmarks/src/gatling/scala/org/apache/polaris/benchmarks/simulations/ReadUpdateTreeDatasetConcurrent.scala renamed to benchmarks/src/gatling/scala/org/apache/polaris/benchmarks/parameters/ReadTreeDatasetParameters.scala

@@ -17,25 +17,18 @@
  * under the License.
  */
 
-package org.apache.polaris.benchmarks.simulations
+package org.apache.polaris.benchmarks.parameters
 
-import io.gatling.core.Predef._
-import io.gatling.http.Predef._
-
-import scala.concurrent.duration.DurationInt
-
-class ReadUpdateTreeDatasetConcurrent extends ReadUpdateTreeDataset {
-  // --------------------------------------------------------------------------------
-  // Build up the HTTP protocol configuration and set up the simulation
-  // --------------------------------------------------------------------------------
-  private val httpProtocol = http
-    .baseUrl(cp.baseUrl)
-    .acceptHeader("application/json")
-    .contentTypeHeader("application/json")
-
-  setUp(
-    authenticate
-      .inject(atOnceUsers(1))
-      .andThen(readWriteScenario.inject(constantUsersPerSec(100).during(5.minutes).randomized))
-  ).protocols(httpProtocol)
+/**
+ * Case class to hold the parameters for the ReadTreeDataset simulation.
+ *
+ * @param tableThroughput The number of table operations to perform per second.
+ * @param viewThroughput The number of view operations to perform per second.
+ */
+case class ReadTreeDatasetParameters(
+    tableThroughput: Int,
+    viewThroughput: Int
+) {
+  require(tableThroughput >= 0, "Table throughput cannot be negative")
+  require(viewThroughput >= 0, "View throughput cannot be negative")
 }

benchmarks/src/gatling/scala/org/apache/polaris/benchmarks/parameters/ReadUpdateTreeDatasetParameters.scala

@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.benchmarks.parameters
+
+/**
+ * Case class to hold the parameters for the ReadUpdateTreeDataset simulation.
+ *
+ * @param throughput The number of operations to perform per second.
+ * @param durationInMinutes The duration of the simulation in minutes.
+ */
+case class ReadUpdateTreeDatasetParameters(
+    throughput: Int,
+    durationInMinutes: Int
+) {
+  require(throughput >= 0, "Throughput cannot be negative")
+  require(durationInMinutes > 0, "Duration in minutes must be positive")
+}

benchmarks/src/gatling/scala/org/apache/polaris/benchmarks/parameters/WorkloadParameters.scala

@@ -24,7 +24,10 @@ case class WorkloadParameters(
     updatesPerNamespace: Int,
     updatesPerTable: Int,
     updatesPerView: Int,
-    seed: Long
+    seed: Long,
+    readTreeDataset: ReadTreeDatasetParameters,
+    createTreeDataset: CreateTreeDatasetParameters,
+    readUpdateTreeDataset: ReadUpdateTreeDatasetParameters
 ) {
   require(
     readWriteRatio >= 0.0 && readWriteRatio <= 1.0,

benchmarks/src/gatling/scala/org/apache/polaris/benchmarks/simulations/CreateTreeDataset.scala

@@ -21,6 +21,7 @@ package org.apache.polaris.benchmarks.simulations
 
 import io.gatling.core.Predef._
 import io.gatling.core.structure.ScenarioBuilder
+import io.gatling.http.Predef._
 import org.apache.polaris.benchmarks.actions._
 import org.apache.polaris.benchmarks.parameters.BenchmarkConfig.config
 import org.apache.polaris.benchmarks.parameters.{
@@ -30,7 +31,8 @@ import org.apache.polaris.benchmarks.parameters.{
 }
 import org.slf4j.LoggerFactory
 
-import java.util.concurrent.atomic.{AtomicInteger, AtomicReference}
+import java.util.concurrent.atomic.{AtomicBoolean, AtomicInteger, AtomicReference}
+import scala.concurrent.duration._
 
 /**
  * This simulation is a 100% write workload that creates a tree dataset in Polaris. It is intended
@@ -51,6 +53,7 @@ class CreateTreeDataset extends Simulation {
   // --------------------------------------------------------------------------------
   private val numNamespaces: Int = dp.nAryTree.numberOfNodes
   private val accessToken: AtomicReference[String] = new AtomicReference()
+  private val shouldRefreshToken: AtomicBoolean = new AtomicBoolean(true)
 
   private val authenticationActions = AuthenticationActions(cp, accessToken, 5, Set(500))
   private val catalogActions = CatalogActions(dp, accessToken, 0, Set())
@@ -64,11 +67,31 @@ class CreateTreeDataset extends Simulation {
   private val createdViews = new AtomicInteger()
 
   // --------------------------------------------------------------------------------
-  // Workload: Authenticate and store the access token for later use
+  // Authentication related workloads:
+  // * Authenticate and store the access token for later use every minute
+  // * Wait for an OAuth token to be available
+  // * Stop the token refresh loop
   // --------------------------------------------------------------------------------
-  val authenticate: ScenarioBuilder = scenario("Authenticate using the OAuth2 REST API endpoint")
-    .feed(authenticationActions.feeder())
-    .exec(authenticationActions.authenticateAndSaveAccessToken)
+  val continuouslyRefreshOauthToken: ScenarioBuilder =
+    scenario("Authenticate every minute using the Iceberg REST API")
+      .asLongAs(_ => shouldRefreshToken.get()) {
+        feed(authenticationActions.feeder())
+          .exec(authenticationActions.authenticateAndSaveAccessToken)
+          .pause(1.minute)
+      }
+
+  val waitForAuthentication: ScenarioBuilder =
+    scenario("Wait for the authentication token to be available")
+      .asLongAs(_ => accessToken.get() == null) {
+        pause(1.second)
+      }
+
+  val stopRefreshingToken: ScenarioBuilder =
+    scenario("Stop refreshing the authentication token")
+      .exec { session =>
+        shouldRefreshToken.set(false)
+        session
+      }
 
   // --------------------------------------------------------------------------------
   // Workload: Create catalogs
@@ -118,4 +141,34 @@ class CreateTreeDataset extends Simulation {
       feed(viewActions.viewCreationFeeder())
         .exec(viewActions.createView)
     )
+
+  // --------------------------------------------------------------------------------
+  // Build up the HTTP protocol configuration and set up the simulation
+  // --------------------------------------------------------------------------------
+  private val httpProtocol = http
+    .baseUrl(cp.baseUrl)
+    .acceptHeader("application/json")
+    .contentTypeHeader("application/json")
+
+  // Get the configured throughput for tables and views
+  private val tableThroughput = wp.createTreeDataset.tableThroughput
+  private val viewThroughput = wp.createTreeDataset.viewThroughput
+
+  setUp(
+    continuouslyRefreshOauthToken.inject(atOnceUsers(1)).protocols(httpProtocol),
+    waitForAuthentication
+      .inject(atOnceUsers(1))
+      .andThen(createCatalogs.inject(atOnceUsers(1)).protocols(httpProtocol))
+      .andThen(
+        createNamespaces
+          .inject(
+            constantUsersPerSec(1).during(1.seconds),
+            constantUsersPerSec(dp.nsWidth - 1).during(dp.nsDepth.seconds)
+          )
+          .protocols(httpProtocol)
+      )
+      .andThen(createTables.inject(atOnceUsers(tableThroughput)).protocols(httpProtocol))
+      .andThen(createViews.inject(atOnceUsers(viewThroughput)).protocols(httpProtocol))
+      .andThen(stopRefreshingToken.inject(atOnceUsers(1)).protocols(httpProtocol))
+  )
 }