Skip to content

Commit 883d078

Browse files
jasonrandrewsjasonrandrews
authored
Merge pull request #611 from boltliu85/hugepage
add Learn how to enable libhugetlbfs to increase performance
2 parents f4141bb + 7968dea commit 883d078

File tree

6 files changed

+607
-0
lines changed

6 files changed

+607
-0
lines changed

‎content/learning-paths/servers-and-cloud-computing/libhugetlbfs/_index.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
---
2+
title: Learn how to enable libhugetlbfs to increase performance on Arm Server
3+
4+
minutes_to_complete: 60
5+
6+
who_is_this_for: This is an advanced topic for performance engineers who wants to tune performance in arm server.
7+
8+
learning_objectives:
9+
- enable libhugetlbfs to increase performance
10+
- see how much performance improved on workloads like MySQL, Redis.
11+
12+
prerequisites:
13+
- system with ubuntu 20 installed
14+
- knowledge to build MySQL server, and run sysbench benchmark test
15+
- knowledge to build Redis server, and run memtier benchmark test
16+
17+
author_primary: Bolt Liu
18+
19+
skilllevels: Advanced
20+
subjects: Databases
21+
armips:
22+
- Neoverse
23+
operatingsystems:
24+
- Linux
25+
tools_software_languages:
26+
- C
27+
- C++
28+
29+
test_images:
30+
- ubuntu:latest
31+
test_link: null
32+
test_maintenance: true
33+
test_status:
34+
- passed
35+
36+
weight: 1
37+
layout: learningpathall
38+
learning_path_main_page: 'yes'
39+
---

‎content/learning-paths/servers-and-cloud-computing/libhugetlbfs/_next-steps.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
---
2+
# ================================================================================
3+
# Edit
4+
# ================================================================================
5+
6+
next_step_guidance: >
7+
You can continue learning about migrating applications to Arm.
8+
# 1-3 sentence recommendation outlining how the reader can generally keep learning about these topics, and a specific explanation of why the next step is being recommended.
9+
10+
recommended_path: "/learning-paths/servers-and-cloud-computing/mysql_tune/"
11+
# Link to the next learning path being recommended(For example this could be /learning-paths/servers-and-cloud-computing/mongodb).
12+
13+
# further_reading links to references related to this path. Can be:
14+
# Manuals for a tool / software mentioned (type: documentation)
15+
# Blog about related topics (type: blog)
16+
# General online references (type: website)
17+
18+
further_reading:
19+
- resource:
20+
title: libhugetlbfs manpage
21+
link: https://linux.die.net/man/7/libhugetlbfs
22+
type: documentation
23+
24+
# ================================================================================
25+
# FIXED, DO NOT MODIFY
26+
# ================================================================================
27+
weight: 21 # set to always be larger than the content in this path, and one more than 'review'
28+
title: "Next Steps" # Always the same
29+
layout: "learningpathall" # All files under learning paths have this same wrapper
30+
---

‎content/learning-paths/servers-and-cloud-computing/libhugetlbfs/_review.md‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
---
2+
# ================================================================================
3+
# Edit
4+
# ================================================================================
5+
6+
# Always 3 questions. Should try to test the reader's knowledge, and reinforce the key points you want them to remember.
7+
# question: A one sentence question
8+
# answers: The correct answers (from 2-4 answer options only). Should be surrounded by quotes.
9+
# correct_answer: An integer indicating what answer is correct (index starts from 0)
10+
# explanation: A short (1-3 sentence) explanation of why the correct answer is correct. Can add additional context if desired
11+
12+
13+
review:
14+
- questions:
15+
question: >
16+
In which build stage libhugetlbfs will take effect?
17+
answers:
18+
- preprocessing
19+
- compilation
20+
- assembly
21+
- linking
22+
correct_answer: 4
23+
explanation: >
24+
libhugetlbfs takes effect during linking stage to place program sections into hugepage.
25+
26+
- questions:
27+
question: >
28+
libhugetlbfs could only eanble code section of a program, is it true?
29+
answers:
30+
- Yes
31+
- No
32+
correct_answer: 2
33+
explanation: >
34+
Though code sectition is the typical section to be placed in hugepage, other sections like data can also be placed in hugepage.
35+
36+
- questions:
37+
question: >
38+
After enabling libhugetlbfs on MySQL, which perf event would be decresed dramatically?
39+
answers:
40+
- l1d_tlb_refill
41+
- l1i_tlb_refill
42+
- l2d_tlb_refill
43+
correct_answer: 2
44+
explanation: >
45+
After enabling libhugetlbfs on MySQL, we could see l1i_tlb_refill decreases dramatically from 490,265,467 to 70,741,621.
46+
47+
48+
49+
# ================================================================================
50+
# FIXED, DO NOT MODIFY
51+
# ================================================================================
52+
title: "Review" # Always the same title
53+
weight: 20 # Set to always be larger than the content in this path
54+
layout: "learningpathall" # All files under learning paths have this same wrapper
55+
---

‎content/learning-paths/servers-and-cloud-computing/libhugetlbfs/libhugetlbfs_general.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
---
2+
title: "General enablement method of libhugetlbfs"
3+
weight: 2
4+
layout: "learningpathall"
5+
---
6+
7+
## Introduction of libhugetlbfs
8+
libhugetlbfs is a library that can back application text, data, malloc() and shared memory with hugepages. This is of benefit to applications that use large amounts of address space and suffer a performance hit due to TLB misses. Hence, by enabling libhugetlbfs, workloads with large code/data/heap sections would see significant performance improvement.
9+
10+
11+
## Install necessary packages
12+
On ubuntu 20, install necessary package and create symbolic link:
13+
```
14+
$ sudo apt-get install libhugetlbfs-dev libhugetlbfs-bin
15+
$ sudo ln -s /usr/bin/ld.hugetlbfs /usr/share/libhugetlbfs/ld
16+
```
17+
## Add compile option to enable libhugetlbfs
18+
add the following build option to build script (gcc option), and rebuild workload, from build option we could learn libhugetlbfs would be enabled in linking stage:
19+
```
20+
-B /usr/share/libhugetlbfs -Wl,--hugetlbfs-align -no-pie -Wl,--no-as-needed
21+
22+
```
23+
24+
## Enable system hugepage
25+
26+
enable Linux system hugepage, for example, setting 1000 huge pages, for 2M huge pages, that's 2G:
27+
```
28+
# echo 1000 > /proc/sys/vm/nr_hugepages
29+
# cat /proc/meminfo |grep HugePages_Total
30+
HugePages_Total: 1000
31+
32+
```
33+
34+
35+
## Add HUGETLB_ELFMAP=RW before starting workload
36+
add HUGETLB_ELFMAP=RW prefix before starting the workload, which places both READ (such as code) and WRITE (such as data) in hugepage, such as:
37+
```
38+
$ HUGETLB_ELFMAP=RW [workload]
39+
```
40+
41+
42+
## Check hugepage is used
43+
44+
make sure hugepage is used by checking meminfo:
45+
46+
```
47+
cat /proc/meminfo
48+
49+
HugePages_Total: 1000
50+
HugePages_Free: 994
51+
52+
```
53+
54+
Also check if the process is having huge page mapped:
55+
```
56+
$ cat /proc/<pid>/smaps | less
57+
58+
00200000-00400000 r-xp 00000000 00:25 48337 /dev/hugepages/libhugetlbfs.tmp.0D0D7x (deleted)
59+
Size: 2048 kB
60+
KernelPageSize: 2048 kB
61+
MMUPageSize: 2048 kB
62+
Rss: 0 kB
63+
Pss: 0 kB
64+
Shared_Clean: 0 kB
65+
Shared_Dirty: 0 kB
66+
Private_Clean: 0 kB
67+
Private_Dirty: 0 kB
68+
Referenced: 0 kB
69+
Anonymous: 0 kB
70+
LazyFree: 0 kB
71+
AnonHugePages: 0 kB
72+
ShmemPmdMapped: 0 kB
73+
FilePmdMapped: 0 kB
74+
Shared_Hugetlb: 0 kB
75+
Private_Hugetlb: 2048 kB
76+
Swap: 0 kB
77+
SwapPss: 0 kB
78+
Locked: 0 kB
79+
THPeligible: 0
80+
```
81+
82+
83+
84+
85+
86+
87+
88+

‎content/learning-paths/servers-and-cloud-computing/libhugetlbfs/mysql_huge.md‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
---
2+
title: "Enable libhugetlbfs on MySQL"
3+
weight: 3
4+
layout: "learningpathall"
5+
---
6+
## Overview
7+
This page illustrates the steps to enable libhugetlbfs on MySQL and test results after enabling it.
8+
9+
10+
## Commands to build
11+
In order to build libhugetlbfs on MySQL, please add the following options to both -DCMAKE_C_FLAGS and -DCMAKE_CXX_FLAGS:
12+
```
13+
-B /usr/share/libhugetlbfs -Wl,--hugetlbfs-align -no-pie -Wl --no-as-needed
14+
```
15+
16+
for example:
17+
```
18+
$ cmake -DCMAKE_C_FLAGS="-g -mcpu=native -B /usr/share/libhugetlbfs -Wl,--hugetlbfs-align -no-pie -Wl,--no-as-needed" -DCMAKE_CXX_FLAGS="-g -mcpu=native -B /usr/share/libhugetlbfs -Wl,--hugetlbfs-align -no-pie -Wl,--no-as-needed" -DCMAKE_INSTALL_PREFIX=/home/mysql/mysql_install/1-install_8.0.33_huge -DWITH_BOOST=/home/mysql/boost_1_77_0/ ..
19+
$ make -j $(nproc)
20+
$ make install
21+
```
22+
23+
after build, check if the program has linked to libhugetlbfs.so, for example:
24+
```
25+
root@bolt-ecs:/data# ldd mysqld |grep huge
26+
libhugetlbfs.so.0 => /lib/aarch64-linux-gnu/libhugetlbfs.so.0 (0x0000ffffac690000)
27+
```
28+
29+
## Commands to run
30+
After rebuilding MySQL with libhugetlbfs, add HUGETLB_ELFMAP=RW at the beginning of the command to start MySQL. for example:
31+
```
32+
$ HUGETLB_ELFMAP=RW /home/mysql/mysql_install/1-install_8.0.33_huge/bin/mysqld ...
33+
```
34+
35+
please note don't export HUGETLB_ELFMAP=RW as an environment varible, it has to be specified right before the mysqld exectuable.
36+
37+
38+
## Test Results
39+
40+
By testing MySQL without/with libhugetlbfs, it shows performance increased by 11.9%～12.9%.
41+
42+
### Without libhugetlbfs
43+
Reboot server and do 2 round tests
44+
45+
#### First round
46+
47+
```
48+
Throughput:
49+
50+
events/s (eps): 8604.3221
51+
52+
time elapsed: 300.0912s
53+
54+
total number of events: 2582080
55+
56+
```
57+
58+
#### Second round
59+
TPS is 8524, also get the perf stat during the run:
60+
61+
```
62+
root@bolt-ecs:~# perf stat -e l1d_tlb_refill,l1i_tlb_refill,l2d_tlb_refill -a -- sleep 10
63+
64+
65+
66+
Performance counter stats for 'system wide':
67+
68+
69+
70+
815,254,864 l1d_tlb_refill
71+
72+
490,265,467 l1i_tlb_refill
73+
74+
422,887,362 l2d_tlb_refill
75+
76+
77+
78+
10.003289183 seconds time elapsed
79+
80+
Throughput:
81+
82+
events/s (eps): 8524.8307
83+
84+
time elapsed: 300.0878s
85+
86+
total number of events: 2558197
87+
88+
```
89+
90+
### With libhugetlbfs
91+
92+
Reboot and do 2 round test, in order to enable hugepage in server, need to do the following things:
93+
94+
```
95+
# chown -R mysql.mysql /dev/hugepages
96+
97+
# echo 40 > /proc/sys/vm/nr_hugepages
98+
99+
# cat /proc/meminfo
100+
101+
HugePages_Total: 40
102+
103+
HugePages_Free: 4
104+
105+
HugePages_Rsvd: 1
106+
107+
HugePages_Surp: 0
108+
109+
Hugepagesize: 2048 kB
110+
111+
Hugetlb: 81920 kB
112+
113+
```
114+
mysql used 36 huge pages (36*2M=72M) in the case.
115+
116+
117+
#### First round
118+
TPS is 9627, this is 12.9% increased compared to the 1st round without enabling hugepage:
119+
```
120+
Throughput:
121+
122+
events/s (eps): 9627.5017
123+
124+
time elapsed: 300.0855s
125+
126+
total number of events: 2889073
127+
128+
```
129+
#### Second round
130+
TPS is 9538, this is 11.9% increased compared to the 2nd round of without enabling hugepage, perf stat shows TLB misses are signifcantly reduced:
131+
132+
```
133+
root@bolt-ecs:~# perf stat -e l1d_tlb_refill,l1i_tlb_refill,l2d_tlb_refill -a -- sleep 10
134+
135+
136+
137+
Performance counter stats for 'system wide':
138+
139+
140+
141+
688,157,786 l1d_tlb_refill
142+
143+
70,741,621 l1i_tlb_refill
144+
145+
254,054,393 l2d_tlb_refill
146+
147+
148+
149+
10.002128509 seconds time elapsed
150+
151+
Throughput:
152+
153+
events/s (eps): 9538.9346
154+
155+
time elapsed: 300.0847s
156+
157+
total number of events: 2862487
158+
159+
```

0 commit comments

Comments
 (0)