Skip to content

Commit 3a41c74

Browse files
matth-xmatth-x
committed
fix action
1 parent bbaf23f commit 3a41c74

File tree

2 files changed

+119
-14
lines changed

2 files changed

+119
-14
lines changed

‎docs/benchmarks.md‎

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -34,18 +34,26 @@ The following table shows the cumulated size of the objects files per module. Th
3434

3535
{{ read_csv('modules_v201.csv') }}
3636

37+
## Memory usage
38+
39+
MicroOCPP uses the heap memory to process incoming messages, maintain the device model and create outgoing OCPP messages. The total heap usage should remain low enough to not risk a heap depletion which would not only affect the OCPP module, but the whole controller, because heap memory is typically shared on microcontrollers. To assess the heap usage of MicroOCPP, a test suite runs a variety of simulated charger use cases and measures the maximum occupied memory. Then, the maximum observed value is considered as the memory requirement of MicroOCPP.
40+
41+
Another important figure is the base level which is much closer to the average heap usage. The total heap usage consists of a base level and a dynamic part. Some memory objects are only initialized once during startup or as the device model is populated (e.g. Charging Schedules) and therefore belong to the base which changes only slowly over time. In contrast, objects for the JSON parsing and serialization and the internal execution of the operations are highly dynamic as they are instantiated for one operation and freed again after completion of the action. If the firmware contains multiple components besides MicroOCPP with this usage pattern, then the average total memory occupation of the device RAM is even closer to the base levels of the individual components.
42+
43+
The following table shows the dynamic heap usage for a variety of test cases, followed by the base level and resulting maximum memory occupation of MicroOCPP. At the time being, the measurements are limited to only OCPP 2.0.1 and a narrow set of test cases. They will be gradually extended over time.
44+
45+
**Table 3: Memory usage per use case and total**
46+
47+
{{ read_csv('heap_v201.csv') }}
48+
3749
## Full data sets
3850

3951
This section contains the raw data which is the basis for the evaluations above.
4052

41-
**Table 3: All compilation units for OCPP 1.6 firmware**
53+
**Table 4: All compilation units for OCPP 1.6 firmware**
4254

4355
{{ read_csv('compile_units_v16.csv') }}
4456

45-
**Table 4: All compilation units for OCPP 2.0.1 firmware**
57+
**Table 5: All compilation units for OCPP 2.0.1 firmware**
4658

4759
{{ read_csv('compile_units_v201.csv') }}
48-
49-
## Heap memory (preview)
50-
51-
{{ read_csv('heap_v201.csv') }}

‎tests/benchmarks/scripts/measure_heap.py‎

Lines changed: 105 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -10,15 +10,102 @@
1010
import pandas as pd
1111

1212

13-
# Test case selection
14-
#testcase_name_list = ['TC_B_06_CS', 'TC_B_07_CS', 'TC_C_02_CS']
15-
testcase_name_list = ['TC_B_06_CS']
13+
requests.packages.urllib3.disable_warnings() # avoid the URL to be printed to console
14+
15+
# Test case selection (commented out a few to simplify testing for now)
16+
testcase_name_list = [
17+
'TC_B_06_CS',
18+
#'TC_B_07_CS',
19+
#'TC_B_09_CS',
20+
'TC_B_10_CS',
21+
#'TC_B_11_CS',
22+
#'TC_B_12_CS',
23+
'TC_B_13_CS',
24+
#'TC_B_32_CS',
25+
#'TC_B_34_CS',
26+
'TC_B_35_CS',
27+
#'TC_B_36_CS',
28+
#'TC_B_37_CS',
29+
'TC_B_39_CS',
30+
#'TC_C_02_CS',
31+
#'TC_C_04_CS',
32+
'TC_C_06_CS',
33+
#'TC_C_07_CS',
34+
#'TC_C_49_CS',
35+
'TC_E_01_CS',
36+
#'TC_E_02_CS',
37+
#'TC_E_03_CS',
38+
'TC_E_04_CS',
39+
#'TC_E_05_CS',
40+
#'TC_E_06_CS',
41+
'TC_E_07_CS',
42+
#'TC_E_09_CS',
43+
#'TC_E_13_CS',
44+
'TC_E_15_CS',
45+
#'TC_E_17_CS',
46+
#'TC_E_20_CS',
47+
'TC_E_21_CS',
48+
#'TC_E_24_CS',
49+
#'TC_E_25_CS',
50+
'TC_E_28_CS',
51+
#'TC_E_29_CS',
52+
#'TC_E_30_CS',
53+
'TC_E_31_CS',
54+
#'TC_E_32_CS',
55+
#'TC_E_33_CS',
56+
'TC_E_34_CS',
57+
#'TC_E_35_CS',
58+
#'TC_E_39_CS',
59+
'TC_F_01_CS',
60+
#'TC_F_02_CS',
61+
#'TC_F_03_CS',
62+
'TC_F_04_CS',
63+
#'TC_F_05_CS',
64+
#'TC_F_06_CS',
65+
'TC_F_07_CS',
66+
#'TC_F_08_CS',
67+
#'TC_F_09_CS',
68+
'TC_F_10_CS',
69+
#'TC_F_11_CS',
70+
#'TC_F_12_CS',
71+
'TC_F_13_CS',
72+
#'TC_F_14_CS',
73+
#'TC_F_20_CS',
74+
'TC_F_23_CS',
75+
#'TC_F_24_CS',
76+
#'TC_F_26_CS',
77+
'TC_F_27_CS',
78+
#'TC_G_01_CS',
79+
#'TC_G_02_CS',
80+
'TC_G_03_CS',
81+
#'TC_G_04_CS',
82+
#'TC_G_05_CS',
83+
'TC_G_06_CS',
84+
#'TC_G_07_CS',
85+
#'TC_G_08_CS',
86+
'TC_G_09_CS',
87+
#'TC_G_10_CS',
88+
#'TC_G_11_CS',
89+
'TC_G_12_CS',
90+
#'TC_G_13_CS',
91+
#'TC_G_14_CS',
92+
'TC_G_15_CS',
93+
#'TC_G_16_CS',
94+
#'TC_G_17_CS',
95+
'TC_J_07_CS',
96+
#'TC_J_08_CS',
97+
#'TC_J_09_CS',
98+
'TC_J_10_CS',
99+
]
100+
101+
testcase_name_list = ['TC_B_06_CS', 'TC_E_05_CS']
16102

17103
# Result data set
18104
df = pd.DataFrame(columns=['FN_BLOCK', 'Testcase', 'Pass', 'Heap usage (Bytes)'])
19105
df.index.names = ['TC_ID']
20106

21-
requests.packages.urllib3.disable_warnings() # avoid the URL to be printed to console
107+
max_memory_total = 0
108+
min_memory_base = 1000 * 1000 * 1000
22109

23110
def connect_ssh():
24111

@@ -111,6 +198,9 @@ def setup_simulator():
111198

112199
def run_measurements():
113200

201+
global max_memory_total
202+
global min_memory_base
203+
114204
print("Fetch TCs from Test Driver")
115205

116206
response = requests.get(os.environ['TEST_DRIVER_URL'] + '/ocpp2.0.1/CS/testcases/' + os.environ['TEST_DRIVER_CONFIG'],
@@ -154,6 +244,7 @@ def run_measurements():
154244
json.loads(os.environ['MO_SIM_API_CONFIG'])['pass']))
155245
print(f'Simulator API /memory/info:\n > {response.status_code}, current heap={response.json()["total_current"]}, max heap={response.json()["total_max"]}')
156246
base_memory_level = response.json()["total_max"]
247+
min_memory_base = min(min_memory_base, response.json()["total_max"])
157248

158249
print("Start Test Driver")
159250

@@ -201,14 +292,20 @@ def run_measurements():
201292
headers={'Authorization': 'Bearer ' + os.environ['TEST_DRIVER_KEY']},
202293
verify=False)
203294
print(f'Test Driver /testcases/{testcase["testcase_name"]}/execute:\n > {test_response.status_code}')
204-
#print(json.dumps(test_response.json(), indent=4))
295+
#try:
296+
# print(json.dumps(test_response.json(), indent=4))
297+
#except:
298+
# print(' > No JSON')
205299

206300
sim_response = requests.get('https://cicd.micro-ocpp.com:8443/api/memory/info',
207301
auth=(json.loads(os.environ['MO_SIM_API_CONFIG'])['user'],
208302
json.loads(os.environ['MO_SIM_API_CONFIG'])['pass']))
209303
print(f'Simulator API /memory/info:\n > {sim_response.status_code}, current heap={sim_response.json()["total_current"]}, max heap={sim_response.json()["total_max"]}')
210304

211-
df.loc[testcase['testcase_name']] = [testcase['functional_block'], testcase['description'], 'x' if test_response.json()['data'][0]['verdict'] == "pass" else '-', str(sim_response.json()["total_max"] - base_memory_level)]
305+
df.loc[testcase['testcase_name']] = [testcase['functional_block'], testcase['description'], 'x' if test_response.status_code == 200 and test_response.json()['data'][0]['verdict'] == "pass" else '-', str(sim_response.json()["total_max"] - min(base_memory_level, sim_response.json()["total_current"]))]
306+
307+
max_memory_total = max(max_memory_total, sim_response.json()["total_max"])
308+
min_memory_base = min(min_memory_base, sim_response.json()["total_current"])
212309

213310
print("Stop Test Driver")
214311

@@ -239,9 +336,9 @@ def run_measurements():
239336
df.loc[f'TC_{i[0]}'] = [i, f'**{i}**', ' ', ' ']
240337

241338
df.loc['|MO_SIM_000'] = ['-', '**Simulator stats**', ' ', ' ']
242-
df.loc['|MO_SIM_010'] = ['-', 'Base memory occupation', ' ', str(base_memory_level)]
339+
df.loc['|MO_SIM_010'] = ['-', 'Base memory occupation', ' ', str(min_memory_base)]
243340
df.loc['|MO_SIM_020'] = ['-', 'Test case maximum', ' ', str(max_memory)]
244-
df.loc['|MO_SIM_030'] = ['-', 'Total memory maximum', ' ', str(base_memory_level + max_memory)]
341+
df.loc['|MO_SIM_030'] = ['-', 'Total memory maximum', ' ', str(max_memory_total)]
245342

246343
df.sort_index(inplace=True)
247344

0 commit comments

Comments
 (0)