Skip to content

Commit d7aad66

Browse files
maobaolongmaobaolong
authored
[#2315] feat(test): Introduce a client simulator tool to simulate send shuffle data request for testing server performance (#2316)
### What changes were proposed in this pull request? Introduce a client simulator tool to simulate send shuffle data request for testing server performance. ### Why are the changes needed? Fix: #2315 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? This has been used on our company.
1 parent c90d463 commit d7aad66

File tree

10 files changed

+1229
-0
lines changed

10 files changed

+1229
-0
lines changed

‎pom.xml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,7 @@
144144
<module>integration-test/common</module>
145145
<module>cli</module>
146146
<module>server-common</module>
147+
<module>tools/client-simulation-yarn</module>
147148
</modules>
148149

149150
<dependencies>

‎tools/client-simulation-yarn/README.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
<!--
2+
~ Licensed to the Apache Software Foundation (ASF) under one or more
3+
~ contributor license agreements. See the NOTICE file distributed with
4+
~ this work for additional information regarding copyright ownership.
5+
~ The ASF licenses this file to You under the Apache License, Version 2.0
6+
~ (the "License"); you may not use this file except in compliance with
7+
~ the License. You may obtain a copy of the License at
8+
~
9+
~ http://www.apache.org/licenses/LICENSE-2.0
10+
~
11+
~ Unless required by applicable law or agreed to in writing, software
12+
~ distributed under the License is distributed on an "AS IS" BASIS,
13+
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
~ See the License for the specific language governing permissions and
15+
~ limitations under the License.
16+
-->
17+
18+
# Uniffle Client Simulation On Yarn - Usage Guide
19+
20+
Currently, we have evaluated the performance of the flush operation using the Uniffle server's flush event recording and flush benchmark feature.
21+
This allows us to assess the server's maximum capability to handle flush block requests for small blocks (e.g., 1 KiB) and the write throughput limit for large blocks (e.g., 1 MiB).
22+
23+
However, there may also be performance bottlenecks between the server receiving requests and the actual flush operation. Therefore, we need a simulated client that continuously sends data to the server.
24+
25+
## Parameter Description
26+
27+
| Parameter Name | Default Value | Description |
28+
|----------------------------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
29+
| `uniffle.client.sim.serverId` | None | Uniffle server ID |
30+
| `uniffle.client.sim.container.num` | 3 | Number of containers to start in the Yarn application, which corresponds to the number of concurrent client processes |
31+
| `uniffle.client.sim.threadCount` | 1 | Number of concurrent threads running in each container process. The actual number of working threads is `threadCount + 1` when each thread is concurrent. |
32+
| `uniffle.client.sim.queueName` | default | Yarn resource queue name |
33+
| `uniffle.client.sim.jarPath.list` | None | HDFS addresses of additional JARs or other resources to download to AM or Task local, separated by commas (e.g., HDFS address of RSS shaded JAR) |
34+
| `uniffle.client.sim.tmp.hdfs.path` | None | A writable HDFS address for uploading temporary application resources |
35+
| `uniffle.client.sim.shuffleCount` | 1 | Number of shuffles included in a single `sendShuffleData` request |
36+
| `uniffle.client.sim.partitionCount` | 1 | Number of partitions included in each shuffle of a single `sendShuffleData` request |
37+
| `uniffle.client.sim.blockCount` | 1 | Number of blocks included in each partition of a single `sendShuffleData` request |
38+
| `uniffle.client.sim.blockSize` | 1024 | Size of each block in a single `sendShuffleData` request |
39+
| `uniffle.client.sim.am.vCores` | 8 | Number of virtual cores specified when requesting the Application Master (AM) |
40+
| `uniffle.client.sim.am.memory` | 4096 | Memory size (in MB) specified when requesting the AM |
41+
| `uniffle.client.sim.container.vCores` | 2 | Number of virtual cores specified when requesting task containers |
42+
| `uniffle.client.sim.container.memory` | 2048 | Memory size (in MB) specified when requesting task containers |
43+
| `uniffle.client.sim.am.jvm.opts` | None | Additional JVM options for debugging when the execution result is abnormal |
44+
| `uniffle.client.sim.container.jvm.opts` | None | Additional JVM options for debugging when the execution result is abnormal |
45+
46+
## Running Example
47+
48+
1. Change to the Hadoop directory and execute the test on Yarn program:
49+
50+
```bash
51+
cd $HADOOP_HOME
52+
```
53+
54+
2. Execute the example command:
55+
56+
```bash
57+
$ bin/yarn jar rss-client-simulation-yarn-0.11.0-SNAPSHOT.jar \
58+
-Duniffle.client.sim.serverId=<UNIFFLE_SERVER_ID> \
59+
-Duniffle.client.sim.container.num=1000 \
60+
-Duniffle.client.sim.queueName=<YOUR_QUEUE_NAME> \
61+
-Duniffle.client.sim.jarPath.list=hdfs://ns1/tmp/rss-client-spark3-shaded.jar \
62+
-Duniffle.client.sim.tmp.hdfs.path=hdfs://ns1/user/xx/tmp/uniffle-client-sim/ \
63+
-Duniffle.client.sim.shuffleCount=5 \
64+
-Duniffle.client.sim.partitionCount=50 \
65+
-Duniffle.client.sim.blockCount=100 \
66+
-Duniffle.client.sim.blockSize=10240 \
67+
-Duniffle.client.sim.threadCount=10
68+
```
69+
70+
3. Example Output:
71+
72+
```plaintext
73+
24/12/30 15:03:47 INFO simulator.UniffleClientSimOnYarnClient: appId: application_1729845342052_5295913
74+
...
75+
Application killed: application_1729845342052_5295913
76+
Application status: KILLED
77+
```
78+
79+
This guide provides a comprehensive overview of how to run the Uniffle Client simulator on Yarn, including parameter descriptions and a step-by-step example. Adjust the parameters as needed for your specific use case.

‎tools/client-simulation-yarn/pom.xml‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
~ Licensed to the Apache Software Foundation (ASF) under one or more
4+
~ contributor license agreements. See the NOTICE file distributed with
5+
~ this work for additional information regarding copyright ownership.
6+
~ The ASF licenses this file to You under the Apache License, Version 2.0
7+
~ (the "License"); you may not use this file except in compliance with
8+
~ the License. You may obtain a copy of the License at
9+
~
10+
~ http://www.apache.org/licenses/LICENSE-2.0
11+
~
12+
~ Unless required by applicable law or agreed to in writing, software
13+
~ distributed under the License is distributed on an "AS IS" BASIS,
14+
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
~ See the License for the specific language governing permissions and
16+
~ limitations under the License.
17+
-->
18+
19+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
20+
<modelVersion>4.0.0</modelVersion>
21+
<parent>
22+
<groupId>org.apache.uniffle</groupId>
23+
<artifactId>uniffle-parent</artifactId>
24+
<version>0.11.0-SNAPSHOT</version>
25+
<relativePath>../../pom.xml</relativePath>
26+
</parent>
27+
28+
<artifactId>rss-client-simulation-yarn</artifactId>
29+
<packaging>jar</packaging>
30+
<name>Apache Uniffle Client Simulation On Yarn</name>
31+
<properties>
32+
<uniffle.version>0.9.1</uniffle.version>
33+
</properties>
34+
35+
<dependencies>
36+
<dependency>
37+
<groupId>org.apache.uniffle</groupId>
38+
<artifactId>rss-client-spark3-shaded</artifactId>
39+
<version>${uniffle.version}</version>
40+
<exclusions>
41+
<exclusion>
42+
<artifactId>rss-client-spark3</artifactId>
43+
<groupId>org.apache.uniffle</groupId>
44+
</exclusion>
45+
</exclusions>
46+
</dependency>
47+
48+
<dependency>
49+
<groupId>org.apache.hadoop</groupId>
50+
<artifactId>hadoop-yarn-client</artifactId>
51+
<version>${hadoop.version}</version>
52+
<scope>provided</scope>
53+
<exclusions>
54+
<exclusion>
55+
<artifactId>protobuf-java</artifactId>
56+
<groupId>com.google.protobuf</groupId>
57+
</exclusion>
58+
</exclusions>
59+
</dependency>
60+
<dependency>
61+
<groupId>org.apache.hadoop</groupId>
62+
<artifactId>hadoop-common</artifactId>
63+
<scope>provided</scope>
64+
</dependency>
65+
</dependencies>
66+
<build>
67+
<plugins>
68+
<plugin>
69+
<groupId>org.apache.maven.plugins</groupId>
70+
<artifactId>maven-jar-plugin</artifactId>
71+
<configuration>
72+
<archive>
73+
<manifest>
74+
<mainClass>org.apache.uniffle.client.simulator.UniffleClientSimOnYarnClient</mainClass>
75+
</manifest>
76+
</archive>
77+
</configuration>
78+
</plugin>
79+
</plugins>
80+
</build>
81+
</project>

‎tools/client-simulation-yarn/src/main/java/org/apache/uniffle/client/simulator/Constants.java‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
package org.apache.uniffle.client.simulator;
19+
20+
public class Constants {
21+
public static final String KEY_PREFIX = "uniffle.client.sim.";
22+
public static final String KEY_AM_MEMORY = KEY_PREFIX + "am.memory";
23+
public static final String KEY_AM_VCORES = KEY_PREFIX + "am.vCores";
24+
public static final String KEY_CONTAINER_MEMORY = KEY_PREFIX + "container.memory";
25+
public static final String KEY_CONTAINER_VCORES = KEY_PREFIX + "container.vCores";
26+
public static final String KEY_QUEUE_NAME = KEY_PREFIX + "queueName";
27+
public static final String KEY_CONTAINER_NUM = KEY_PREFIX + "container.num";
28+
public static final int CONTAINER_NUM_DEFAULT = 3;
29+
public static final String KEY_SERVER_ID = KEY_PREFIX + "serverId";
30+
public static final String KEY_SHUFFLE_COUNT = KEY_PREFIX + "shuffleCount";
31+
public static final String KEY_PARTITION_COUNT = KEY_PREFIX + "partitionCount";
32+
public static final String KEY_BLOCK_COUNT = KEY_PREFIX + "blockCount";
33+
public static final String KEY_BLOCK_SIZE = KEY_PREFIX + "blockSize";
34+
public static final String KEY_EXTRA_JAR_PATH_LIST = KEY_PREFIX + "jarPath.list";
35+
public static final String KEY_YARN_APP_ID = KEY_PREFIX + "appId";
36+
public static final String KEY_CONTAINER_INDEX = KEY_PREFIX + "containerIndex";
37+
public static final String KEY_AM_EXTRA_JVM_OPTS = KEY_PREFIX + "am.jvm.opts";
38+
public static final String KEY_CONTAINER_EXTRA_JVM_OPTS = KEY_PREFIX + "container.jvm.opts";
39+
public static final String KEY_TMP_HDFS_PATH = KEY_PREFIX + "tmp.hdfs.path";
40+
public static final String TMP_HDFS_PATH_DEFAULT = "./tmp/uniffle-client-sim/";
41+
public static final String JOB_CONF_NAME = "job.conf";
42+
public static final String KEY_THREAD_COUNT = KEY_PREFIX + ".threadCount";
43+
}

‎tools/client-simulation-yarn/src/main/java/org/apache/uniffle/client/simulator/HadoopConfigApp.java‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
package org.apache.uniffle.client.simulator;
19+
20+
import java.io.BufferedReader;
21+
import java.io.FileInputStream;
22+
import java.io.IOException;
23+
import java.io.InputStreamReader;
24+
import java.nio.charset.StandardCharsets;
25+
import java.util.HashMap;
26+
import java.util.Map;
27+
import java.util.Properties;
28+
29+
import org.apache.commons.cli.CommandLine;
30+
import org.apache.commons.cli.CommandLineParser;
31+
import org.apache.commons.cli.GnuParser;
32+
import org.apache.commons.cli.Option;
33+
import org.apache.commons.cli.OptionBuilder;
34+
import org.apache.commons.cli.Options;
35+
import org.apache.commons.cli.ParseException;
36+
import org.apache.hadoop.conf.Configuration;
37+
38+
public class HadoopConfigApp {
39+
40+
private final Configuration conf;
41+
private Map<String, String> localConf = new HashMap<>();
42+
43+
public HadoopConfigApp(Configuration conf) {
44+
this.conf = conf;
45+
}
46+
47+
public int run(String[] args) {
48+
Options options = new Options();
49+
50+
Option confOption =
51+
OptionBuilder.withArgName("conf")
52+
.withLongOpt("conf")
53+
.hasArg()
54+
.withDescription("Path to the configuration file with key=value pairs")
55+
.create("c");
56+
options.addOption(confOption);
57+
58+
Option defineOption =
59+
OptionBuilder.withArgName("key=value")
60+
.withLongOpt("define")
61+
.hasArgs(2)
62+
.withValueSeparator()
63+
.withDescription("Define a key-value pair configuration")
64+
.create("D");
65+
options.addOption(defineOption);
66+
67+
CommandLineParser parser = new GnuParser();
68+
CommandLine cmd;
69+
70+
try {
71+
cmd = parser.parse(options, args);
72+
} catch (ParseException e) {
73+
System.err.println("Failed to parse command line arguments: " + e.getMessage());
74+
return 1;
75+
}
76+
77+
// handle --conf option
78+
if (cmd.hasOption("conf")) {
79+
String confFile = cmd.getOptionValue("conf");
80+
try {
81+
loadConfigurationFromFile(localConf, confFile);
82+
} catch (IOException e) {
83+
System.err.println("Error loading configuration from file: " + e.getMessage());
84+
return 1;
85+
}
86+
}
87+
88+
// handle -D option
89+
if (cmd.hasOption("define")) {
90+
Properties properties = cmd.getOptionProperties("D");
91+
for (Map.Entry<Object, Object> entry : properties.entrySet()) {
92+
localConf.put(entry.getKey().toString(), entry.getValue().toString());
93+
}
94+
}
95+
96+
for (Map.Entry<String, String> entry : localConf.entrySet()) {
97+
conf.set(entry.getKey(), entry.getValue());
98+
}
99+
return 0;
100+
}
101+
102+
protected void loadConfigurationFromFile(Map<String, String> map, String filePath)
103+
throws IOException {
104+
try (BufferedReader reader =
105+
new BufferedReader(
106+
new InputStreamReader(new FileInputStream(filePath), StandardCharsets.UTF_8))) {
107+
String line;
108+
while ((line = reader.readLine()) != null) {
109+
line = line.trim();
110+
if (line.isEmpty() || line.startsWith("#")) {
111+
continue; // Skip empty lines and comments
112+
}
113+
int eq = line.indexOf('=');
114+
if (eq > 0) {
115+
String key = line.substring(0, eq).trim();
116+
String value = line.substring(eq + 1).trim();
117+
map.put(key, value);
118+
} else {
119+
System.err.println("Invalid configuration line: " + line);
120+
}
121+
}
122+
}
123+
}
124+
125+
public Map<String, String> getLocalConf() {
126+
return localConf;
127+
}
128+
129+
protected static String writeConfigurationToString(Map<String, String> map) {
130+
StringBuilder stringBuilder = new StringBuilder();
131+
for (Map.Entry<String, String> entry : map.entrySet()) {
132+
stringBuilder.append(entry.getKey()).append("=").append(entry.getValue()).append("\n");
133+
}
134+
return stringBuilder.toString().trim();
135+
}
136+
137+
public static void main(String[] args) {
138+
HadoopConfigApp app = new HadoopConfigApp(new Configuration());
139+
int exitCode = app.run(args);
140+
System.exit(exitCode);
141+
}
142+
}

0 commit comments

Comments
 (0)