Skip to content

Commit 6befe01

Browse files
committed
Improved documentation
Signed-off-by: aryans1204 <arshar1204@gmail.com> Fixed spellcheck in documentation Signed-off-by: aryans1204 <arshar1204@gmail.com>
1 parent 601e0a6 commit 6befe01

File tree

3 files changed

+34
-20
lines changed

3 files changed

+34
-20
lines changed

docs/generate_deploy_info_docs.md

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,10 @@
11
# Deploy Info JSON
22

3-
## The `deploy_info.json` file in the `vSwarm_deploy_metadata.tar.gz` is used to identify the relative file paths for the Knative YAML manifests for deploying vSwarm functions. It is generated using `workloads/container/generate_deploy_info.py` Python script, which outputs a JSON that embeds the yaml-location and pre-deployment commands for every vSwarm function. It also contains the path of YAML files needed as part of the pre-deployment commands to run certain vSwarm benchmarks, for example the `online-shop-database` which requires to be deployed before running `cartservice` benchmark.
3+
The `deploy_info.json` file is used to identify deployment information for services and their pre-dependencies. It is pre-generated, and stored inside the `vSwarm_deploy_metadata.tar.gz` to contain deployment information for vSwarm functions.
44

5-
While the `deploy_info.json` file ships with the `vSwarm_deploy_metadata.tar.gz`, In order to regenerate the `deploy_info.json` run from the root of this repository:
6-
```console
7-
tar -xzvf workloads/container/vSwarm_deploy_metadata.tar.gz -C workloads/container
8-
cd workloads/container/
9-
python3 generate_deploy_info.py
10-
```
5+
## Schema and Usage
6+
7+
The `deploy_info.json` file in the `vSwarm_deploy_metadata.tar.gz` is used to identify the relative file paths for the Knative YAML manifests for deploying vSwarm functions. It is generated using `workloads/container/generate_deploy_info.py` Python script, which outputs a JSON that embeds the yaml-location and pre-deployment commands for every vSwarm function. It also contains the path of YAML files needed as part of the pre-deployment commands to run certain vSwarm benchmarks, for example the `online-shop-database` which requires to be deployed before running `cartservice` benchmark.
118

129
The `deploy_info.json` has the following schema:
1310
```console
@@ -20,9 +17,19 @@ The `deploy_info.json` has the following schema:
2017
}
2118
```
2219

23-
## The `PredeploymentPath` is the path to the YAML file, which is applied via `kubectl apply -f`, before creating the service under `YamlLocation`. This pre-deployment step is required in some vSwarm benchmarks, like `cartservice` which depends on a separate service `online-shop-database` before it can be started.
20+
The `PredeploymentPath` is the path to the YAML file, which is applied via `kubectl apply -f`, before creating the service under `YamlLocation`. This pre-deployment step is required in some vSwarm benchmarks, like `cartservice` which depends on a separate service `online-shop-database` before it can be started.
21+
## Deployment File Generation
22+
23+
While the `deploy_info.json` file ships with the `vSwarm_deploy_metadata.tar.gz`, In order to regenerate the `deploy_info.json` run from the root of this repository:
24+
```console
25+
tar -xzvf workloads/container/vSwarm_deploy_metadata.tar.gz -C workloads/container
26+
cd workloads/container/
27+
python3 generate_deploy_info.py
28+
```
29+
30+
## YAML Generation
2431

25-
Similarly, the `workloads/container/generate_all_yamls.py` is a wrapper script that calls the `generate-yamls.py` script for each vSwarm benchmark in the `vSwarm_deploy_metadata.tar.gz`. The `generate-yamls.py` Python script parameterize the YAML script for the benchmark for different configurations, and creates YAML files accordingly.
32+
The `workloads/container/generate_all_yamls.py` is a wrapper script that calls the `generate-yamls.py` script for each vSwarm benchmark in the `vSwarm_deploy_metadata.tar.gz`. The `generate-yamls.py` Python script generates YAML files for vSwarm benchmarks with different workload parameters to create a variety of durations and memory consumption.
2633

2734
While the YAMLs are pre-generated inside the `vSwarm_deploy_metadata.tar.gz` tarball, to regenerate the YAMLs, run this from the root of this repository:
2835
```console

docs/loader.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,7 @@ To account for difference in CPU performance set `ITERATIONS_MULTIPLIER=102` if
110110
Cloudlab `xl170` or `d430` machines. (Date of measurement: 18-Oct-2022)
111111

112112
## Executing vSwarm functions
113-
If the input trace directory has a `mapper_output.json` file, which you would like to use as profiles for benchmark execution in the loader, run the following from the root of this directory:
113+
If you would like to use vSwarm benchmarks as profile functions to execute, first you need to generate a `mapper_output.json` using the `mapper` tool. Please refer to `mapper.md` docs for usage of the mapper tool to generate an output file. Once the `mapper_output.json` has been generated in the input trace directory, next run the following from the root of this repository:
114114

115115
```console
116116
# install pre-requisites
@@ -120,11 +120,12 @@ git lfs install
120120
git lfs fetch
121121
git lfs checkout
122122
```
123-
This retrieves the `yamls.tar.gz` from Git LFS. Then, untar this tarball by running the following command from the root of this directory:
123+
This retrieves the `vSwarm_deploy_metadata.tar.gz` from Git LFS. Then, untar this tarball by running the following command from the root of this repository:
124124

125125
```bash
126126
$ tar -xzvf workloads/container/vSwarm_deploy_metadata.tar.gz -C workloads/container/
127127
```
128+
This untar the tarball into the `workloads/container/yamls` directory, which contains deployment information and deployment YAML files for all vSwarm benchmarks. If you would like to change some of these deployment files of regenerate them, refer to `generate_deploy_info_docs.md` for an outline of what these deployment files are, and how you can regenerate them.
128129

129130
## Single execution
130131

@@ -138,12 +139,12 @@ To run load generator to use vSwarm functions based on `mapper_output.json` run
138139
```bash
139140
$ go run cmd/loader.go --config cmd/config_vswarm_trace.json
140141
```
142+
The difference between instructing the loader to run vSwarm functions is by setting the `VSwarm` flag in the loader configuration to `true`. For information on how to configure the workload for load generator, please refer to `docs/configuration.md`.
143+
141144
Additionally, one can specify log verbosity argument as `--verbosity [info, debug, trace]`. The default value is `info`.
142145

143146
To execute in a dry run mode without generating any load, set the `--dry-run` flag to `true`. This is useful for testing and validating configurations without executing actual requests.
144147

145-
For to configure the workload for load generator, please refer to `docs/configuration.md`.
146-
147148
There are a couple of constants that should not be exposed to the users. They can be examined and changed
148149
in `pkg/common/constants.go`.
149150

pkg/trace/mapper_trace_parser.go

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,13 @@ type DeploymentInfo struct {
2222
PredeploymentPath []string
2323
}
2424

25-
type JSONParser map[string]DeploymentInfo
25+
type MapperOutput struct {
26+
ProxyFunction string `json:"proxy-function"`
27+
}
28+
29+
type functionToDeploymentInfo map[string]DeploymentInfo
30+
31+
type functionToProxy map[string]MapperOutput
2632

2733
func NewMapperParser(directoryPath string, totalDuration int) *MapperTraceParser {
2834
return &MapperTraceParser{
@@ -33,7 +39,7 @@ func NewMapperParser(directoryPath string, totalDuration int) *MapperTraceParser
3339
}
3440
}
3541

36-
func (p *MapperTraceParser) extractFunctions(mapperOutput map[string]map[string]string, deploymentInfo JSONParser, dirPath string) []*common.Function {
42+
func (p *MapperTraceParser) extractFunctions(mapperOutput functionToProxy, deploymentInfo functionToDeploymentInfo, dirPath string) []*common.Function {
3743
var result []*common.Function
3844

3945
invocations := parseInvocationTrace(dirPath+"/invocations.csv", p.duration)
@@ -48,17 +54,17 @@ func (p *MapperTraceParser) extractFunctions(mapperOutput map[string]map[string]
4854
hashFunction := invocationStats.HashFunction
4955
hashApp := invocationStats.HashApp
5056
hashOwner := invocationStats.HashOwner
51-
proxyFunction := mapperOutput[hashFunction+hashOwner+hashApp]["proxy-function"]
57+
proxyFunction := mapperOutput[hashFunction+hashOwner+hashApp].ProxyFunction
5258
yamlPath := deploymentInfo[proxyFunction].YamlLocation
53-
predeploymentPaths := deploymentInfo[proxyFunction].PredeploymentPath
59+
predeploymentPath := deploymentInfo[proxyFunction].PredeploymentPath
5460
function := &common.Function{
5561
Name: fmt.Sprintf("%s-%d-%d", proxyFunction, i, p.functionNameGenerator.Uint64()),
5662

5763
InvocationStats: &invocationStats,
5864
RuntimeStats: runtimeByHashFunction[hashFunction],
5965
MemoryStats: memoryByHashFunction[hashFunction],
6066
YAMLPath: yamlPath,
61-
PredeploymentPath: predeploymentPaths,
67+
PredeploymentPath: predeploymentPath,
6268
}
6369

6470
result = append(result, function)
@@ -69,8 +75,8 @@ func (p *MapperTraceParser) extractFunctions(mapperOutput map[string]map[string]
6975

7076
func (p *MapperTraceParser) Parse() []*common.Function {
7177
var functions []*common.Function
72-
var mapperOutput map[string]map[string]string // HashFunction mapped to vSwarm function yaml.
73-
var deploymentInfo JSONParser
78+
var mapperOutput functionToProxy
79+
var deploymentInfo functionToDeploymentInfo
7480
// Read the deployment info file for yaml locations and predeployment commands if any
7581
deploymentInfoFile, err := os.ReadFile("test_data/test_deploy_info.json")
7682
if err != nil {

0 commit comments

Comments
 (0)