QuickStartupGuide

Startup

Quick startup guide

To start benchmarking, it is required to create two new directories on the FTP-server, the first one for the benchmark configuration files, and the second one for the files of benchmarking results. Further, please, follow instructions.

1. Prepare configuration files in accordance with src/configs/benchmark_configuration_file_template.xml and src/configs/remote_configuration_file_template.xml. Please, use GUI application (src/config_maker).
2. Copy the benchmark configuration files to the corresponding directory on the FTP-server.
3. Execute the src/remote_control/remote_start.py script. Please, follow src/remote_control/README.md.
4. Wait for completing the benchmark.
5. Copy benchmarking results from the FTP-server to the local machine for the further analysis.

Startup example

1. Fill out the configuration file for the benchmarking script. It is required to describe tests to be performed, you can find the template in the src/config/benchmark_configuration_file_template.xml. Fill the configuration file and save it to the benchmark_config/bench_config.xml on the FTP-server. Please, use the developed GUI application (config maker).

   <Tests>
     <Test>
       <Model>
           <Task>Classification</Task>
           <Name>densenet-121</Name>
           <Precision>FP32</Precision>
           <SourceFramework>Caffe</SourceFramework>
           <Path>/opt/intel/openvino/deployment_tools/tools/model_downloader/public/densenet-121/FP32</Path>
       </Model>
       <Dataset>
           <Name>ImageNet</Name>
           <Path>/tmp/data/</Path>
       </Dataset>
       <FrameworkIndependent>
           <InferenceFramework>OpenVINO_DLDT</InferenceFramework>
           <BatchSize>2</BatchSize>
           <Device>CPU</Device>
           <IterationCount>10</IterationCount>
           <TestTimeLimit>1000</TestTimeLimit>
       </FrameworkIndependent>
       <FrameworkDependent>
           <Mode>Sync</Mode>
           <Extension></Extension>
           <AsyncRequestCount></AsyncRequestCount>
           <ThreadCount></ThreadCount>
           <StreamCount></StreamCount>
       </FrameworkDependent>
     </Test>
   </Tests>

2. Fill out the configuration file for the remote start script, you can find the template in the src/config/remote_configuration_file_template.xml. Fill it and save to the /tmp/dl-benchmark/src/remote_start/remote_config.xml. Please, use the developed GUI application (config maker).

   <Computers>
     <Computer>
       <IP>4.4.4.4</IP>
       <Login>user</Login>
       <Password>user</Password>
       <OS>Linux</OS>
       <FTPClientPath>/tmp/dl-benchmark/src/remote_start/ftp_client.py</FTPClientPath>
       <BenchmarkConfig>/benchmark_config/bench_config.xml</BenchmarkConfig>
       <BenchmarkExecutor>docker_container</BenchmarkExecutor>
       <LogFile>/tmp/dl-benchmark/src/remote_start/log.txt</LogFile>
       <ResultFile>/tmp/dl-benchmark/src/remote_start/result.csv</ResultFile>
     </Computer>
   </Computers>

3. Execute the remote start script using the following command:

   python3 remote_start.py \
   -c /tmp/dl-benchmark/src/remote_start/remote_config.xml \
   -s 2.2.2.2 -l admin -p admin -r all_results.csv \
   --ftp_dir table_folder

4. [TBD] Wait for completing the benchmark. After completion, the table_folder directory will contain a table with the combined results named all_results.csv.

5. [TBD] Copy benchmarking results from the FTP-server to the local machine.

   scp [email protected]:/table_folder/all_results.csv /tmp/

6. [TBD] Convert csv to html using the following command:

   cd /tmp/dl-benchmark/csv2html
   python3 converter.py -t /tmp/all_results.csv -r /tmp/formatted_results.html