Skip to content

Commit c43cad8

Browse files
bsdimpmchoo7
andcommitted
jemalloc: Merge from jemalloc 5.3.0 vendor branch
This brings in a merge from jemalloc 5.3.0. It's almost fully scripted, except: Three conflicts resolved by hand: include/jemalloc/internal/test_hooks.h Use the new name src/extent.c Use the new code src/jemalloc.c Use the new code since je_realloc has moved The script is recorded in FREEBSD-upgrade. The old script did svn commands that were basically a rebase of our changes. This update has a series of diff reduction changes before this. Note: I'd planned on fixing the above three conflicts with commits, but ran out of time when I did this work in January. I got discouraged when jemalloc was EOL'd and didn't pick this back up. I did the above by hand to get this into FreeBSD 15.0 This work is a repeat of the work by Minsoo Choo who did all these changes and created a pull request. Given the importance of jemalloc, I audited these changes by redoing them in this series of commits (and with the script that was checked in). I did this to confince myself and anybody else in doubt that there was no supply chain attack. The diffs between this series of commits and Minsoo's work are minor (though the version skew makes adds some noise). Interested parties can independent audit each step, I hope. I've listed Minsoo as a co-author since without his pull request to test again, this wouldn't have been possible. Thanks to brooks@ for help with getting the jemalloc 3 ABI compat symbols right. Co-authored-by: Minsoo Choo <[email protected]> Pull Request: #1337 Sponsored by: Netflix
2 parents da260ab + 48ec896 commit c43cad8

File tree

218 files changed

+47956
-12362
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

218 files changed

+47956
-12362
lines changed

contrib/jemalloc/ChangeLog

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,106 @@ brevity. Much more detail can be found in the git revision history:
44

55
https://github.com/jemalloc/jemalloc
66

7+
* 5.3.0 (May 6, 2022)
8+
9+
This release contains many speed and space optimizations, from micro
10+
optimizations on common paths to rework of internal data structures and
11+
locking schemes, and many more too detailed to list below. Multiple percent
12+
of system level metric improvements were measured in tested production
13+
workloads. The release has gone through large-scale production testing.
14+
15+
New features:
16+
- Add the thread.idle mallctl which hints that the calling thread will be
17+
idle for a nontrivial period of time. (@davidtgoldblatt)
18+
- Allow small size classes to be the maximum size class to cache in the
19+
thread-specific cache, through the opt.[lg_]tcache_max option. (@interwq,
20+
@jordalgo)
21+
- Make the behavior of realloc(ptr, 0) configurable with opt.zero_realloc.
22+
(@davidtgoldblatt)
23+
- Add 'make uninstall' support. (@sangshuduo, @Lapenkov)
24+
- Support C++17 over-aligned allocation. (@marksantaniello)
25+
- Add the thread.peak mallctl for approximate per-thread peak memory tracking.
26+
(@davidtgoldblatt)
27+
- Add interval-based stats output opt.stats_interval. (@interwq)
28+
- Add prof.prefix to override filename prefixes for dumps. (@zhxchen17)
29+
- Add high resolution timestamp support for profiling. (@tyroguru)
30+
- Add the --collapsed flag to jeprof for flamegraph generation.
31+
(@igorwwwwwwwwwwwwwwwwwwww)
32+
- Add the --debug-syms-by-id option to jeprof for debug symbols discovery.
33+
(@DeannaGelbart)
34+
- Add the opt.prof_leak_error option to exit with error code when leak is
35+
detected using opt.prof_final. (@yunxuo)
36+
- Add opt.cache_oblivious as an runtime alternative to config.cache_oblivious.
37+
(@interwq)
38+
- Add mallctl interfaces:
39+
+ opt.zero_realloc (@davidtgoldblatt)
40+
+ opt.cache_oblivious (@interwq)
41+
+ opt.prof_leak_error (@yunxuo)
42+
+ opt.stats_interval (@interwq)
43+
+ opt.stats_interval_opts (@interwq)
44+
+ opt.tcache_max (@interwq)
45+
+ opt.trust_madvise (@azat)
46+
+ prof.prefix (@zhxchen17)
47+
+ stats.zero_reallocs (@davidtgoldblatt)
48+
+ thread.idle (@davidtgoldblatt)
49+
+ thread.peak.{read,reset} (@davidtgoldblatt)
50+
51+
Bug fixes:
52+
- Fix the synchronization around explicit tcache creation which could cause
53+
invalid tcache identifiers. This regression was first released in 5.0.0.
54+
(@yoshinorim, @davidtgoldblatt)
55+
- Fix a profiling biasing issue which could cause incorrect heap usage and
56+
object counts. This issue existed in all previous releases with the heap
57+
profiling feature. (@davidtgoldblatt)
58+
- Fix the order of stats counter updating on large realloc which could cause
59+
failed assertions. This regression was first released in 5.0.0. (@azat)
60+
- Fix the locking on the arena destroy mallctl, which could cause concurrent
61+
arena creations to fail. This functionality was first introduced in 5.0.0.
62+
(@interwq)
63+
64+
Portability improvements:
65+
- Remove nothrow from system function declarations on macOS and FreeBSD.
66+
(@davidtgoldblatt, @fredemmott, @leres)
67+
- Improve overcommit and page alignment settings on NetBSD. (@zoulasc)
68+
- Improve CPU affinity support on BSD platforms. (@devnexen)
69+
- Improve utrace detection and support. (@devnexen)
70+
- Improve QEMU support with MADV_DONTNEED zeroed pages detection. (@azat)
71+
- Add memcntl support on Solaris / illumos. (@devnexen)
72+
- Improve CPU_SPINWAIT on ARM. (@AWSjswinney)
73+
- Improve TSD cleanup on FreeBSD. (@Lapenkov)
74+
- Disable percpu_arena if the CPU count cannot be reliably detected. (@azat)
75+
- Add malloc_size(3) override support. (@devnexen)
76+
- Add mmap VM_MAKE_TAG support. (@devnexen)
77+
- Add support for MADV_[NO]CORE. (@devnexen)
78+
- Add support for DragonFlyBSD. (@devnexen)
79+
- Fix the QUANTUM setting on MIPS64. (@brooksdavis)
80+
- Add the QUANTUM setting for ARC. (@vineetgarc)
81+
- Add the QUANTUM setting for LoongArch. (@wangjl-uos)
82+
- Add QNX support. (@jqian-aurora)
83+
- Avoid atexit(3) calls unless the relevant profiling features are enabled.
84+
(@BusyJay, @laiwei-rice, @interwq)
85+
- Fix unknown option detection when using Clang. (@Lapenkov)
86+
- Fix symbol conflict with musl libc. (@georgthegreat)
87+
- Add -Wimplicit-fallthrough checks. (@nickdesaulniers)
88+
- Add __forceinline support on MSVC. (@santagada)
89+
- Improve FreeBSD and Windows CI support. (@Lapenkov)
90+
- Add CI support for PPC64LE architecture. (@ezeeyahoo)
91+
92+
Incompatible changes:
93+
- Maximum size class allowed in tcache (opt.[lg_]tcache_max) now has an upper
94+
bound of 8MiB. (@interwq)
95+
96+
Optimizations and refactors (@davidtgoldblatt, @Lapenkov, @interwq):
97+
- Optimize the common cases of the thread cache operations.
98+
- Optimize internal data structures, including RB tree and pairing heap.
99+
- Optimize the internal locking on extent management.
100+
- Extract and refactor the internal page allocator and interface modules.
101+
102+
Documentation:
103+
- Fix doc build with --with-install-suffix. (@lawmurray, @interwq)
104+
- Add PROFILING_INTERNALS.md. (@davidtgoldblatt)
105+
- Ensure the proper order of doc building and installation. (@Mingli-Yu)
106+
7107
* 5.2.1 (August 5, 2019)
8108

9109
This release is primarily about Windows. A critical virtual memory leak is

contrib/jemalloc/FREEBSD-upgrade

Lines changed: 78 additions & 176 deletions
Original file line numberDiff line numberDiff line change
@@ -1,189 +1,91 @@
11
#!/bin/sh
2-
#
3-
# Usage: cd /usr/src/contrib/jemalloc
4-
# ./FREEBSD-upgrade <command> [args]
5-
#
6-
# At least the following ports are required when importing jemalloc:
7-
# - devel/autoconf
8-
# - devel/git
9-
# - devel/gmake
10-
# - textproc/docbook-xsl
11-
# - textproc/libxslt
12-
#
13-
# The normal workflow for importing a new release is:
14-
#
15-
# cd /usr/src/contrib/jemalloc
16-
#
17-
# Merge local changes that were made since the previous import:
18-
#
19-
# ./FREEBSD-upgrade merge-changes
20-
# ./FREEBSD-upgrade rediff
21-
#
22-
# Extract latest jemalloc release.
23-
#
24-
# ./FREEBSD-upgrade extract <rev>
25-
#
26-
# Fix patch conflicts as necessary, then regenerate diffs to update line
27-
# offsets:
28-
#
29-
# ./FREEBSD-upgrade rediff
30-
# ./FREEBSD-upgrade extract <rev>
31-
#
32-
# Do multiple buildworld/installworld rounds. If problems arise and patches
33-
# are needed, edit the code in ${work} as necessary, then:
34-
#
35-
# ./FREEBSD-upgrade rediff
36-
# ./FREEBSD-upgrade extract <rev>
37-
#
38-
# The rediff/extract order is important because rediff saves the local
39-
# changes, then extract blows away the work tree and re-creates it with the
40-
# diffs applied.
41-
#
42-
# Finally, to clean up:
43-
#
44-
# ./FREEBSD-upgrade clean
452

46-
set -e
47-
set -x
3+
# Note: you need docbook installed, as well as gmake (we need it to
4+
# make private_namespace.h)
485

49-
if [ ! -x "FREEBSD-upgrade" ] ; then
50-
echo "Run from within src/contrib/jemalloc/" >&2
51-
exit 1
52-
fi
6+
# git subtree merge -- not committed at this time.
7+
git subtree merge -P contrib/jemalloc vendor/jemalloc
8+
cd contrib/jemalloc
539

54-
if [ "x${JEMALLOC_REPO}" = "x" ] ; then
55-
JEMALLOC_REPO=https://github.com/jemalloc/jemalloc.git
56-
fi
10+
# Gut the tests, since they take up too much space.
11+
# Everything else can stay, but if not, add more to trim (there's
12+
# always a trade off between time and saved size.
13+
git rm -rf test msvc
14+
git commit --amend
5715

58-
src=`pwd`
16+
# kill the tests with empty files so we don't have to hack configure.ac
17+
mkdir -p test/include/test
18+
touch test/include/test/jemalloc_test_defs.h.in
19+
touch test/include/test/jemalloc_test.h.in
20+
echo 'exit 0' > test/test.sh.in
5921

60-
jemalloc_tmp="jemalloc.tmp"
61-
tmpdir="${src}/../${jemalloc_tmp}"
62-
bare_repo="${tmpdir}/jemalloc_bare.git"
63-
work="jemalloc_work.git"
64-
work_repo="${tmpdir}/${work}"
65-
namespace_repo="${tmpdir}/jemalloc_namespace.git"
66-
changes="${src}/FREEBSD-changes"
67-
68-
do_fetch() {
69-
local rev=$1
70-
if [ ! -d "${bare_repo}" ] ; then
71-
mkdir -p "${bare_repo}"
72-
git clone --bare ${JEMALLOC_REPO} ${bare_repo}
73-
fi
74-
(
75-
cd ${bare_repo}
76-
git fetch origin ${rev}
77-
)
78-
}
22+
# Reconfigure -- needed only to regenerate the .h files... We don't
23+
# use all the files generated.
24+
#
25+
# Also note: 5.2 lacks --with-lg-page-sizes, but 5.3 has it.
26+
# Also, there's got to be a way to not hard-wire version / hash.
27+
./autogen.sh --enable-xmalloc --enable-fill --enable-lazy-lock --enable-stats \
28+
--enable-utrace --with-malloc-conf=abort_conf:false \
29+
--with-xslroot=/usr/local/share/xsl/docbook --with-private-namespace=__ \
30+
--with-lg-page-sizes=12,13,14,15,16 \
31+
--with-version=5.3.0-0-g54eaed1d8b56b1aa528be3bdd1877e59c56fa90c
7932

80-
do_extract_helper() {
81-
local rev=$1
82-
local repo=$2
83-
do_fetch ${rev}
84-
rm -rf ${repo}
85-
git clone ${bare_repo} ${repo}
86-
(
87-
cd ${repo}
88-
if [ "x${rev}" != "x" ] ; then
89-
# Use optional rev argument to check out a revision other than HEAD on
90-
# master.
91-
git checkout ${rev}
92-
fi
93-
)
94-
}
33+
# Copy over the important generated .h files in configure
34+
cp ./include/jemalloc/jemalloc.h ../../lib/libc/stdlib/malloc/jemalloc/include/jemalloc
35+
git add ../../lib/libc/stdlib/malloc/jemalloc/include/jemalloc/jemalloc.h
36+
cp ./include/jemalloc/jemalloc_defs.h ../../lib/libc/stdlib/malloc/jemalloc/include/jemalloc
37+
git add ../../lib/libc/stdlib/malloc/jemalloc/include/jemalloc/jemalloc_defs.h
9538

96-
do_autogen() {
97-
./autogen.sh --enable-xmalloc --enable-utrace \
98-
--with-malloc-conf=abort_conf:false \
99-
--with-xslroot=/usr/local/share/xsl/docbook --with-private-namespace=__ \
100-
--with-lg-page-sizes=12,13,14,16
101-
}
39+
# need to make the namespace .h files, and copy a small subset into the tree
40+
# These are super-awkward to generate at buildworld time. Also, we assume we
41+
# only have to make one of these (currently true due to current unlikely to
42+
# change dependencies.
43+
gmake include/jemalloc/internal/private_namespace.h
44+
for i in private_namespace.h jemalloc_internal_defs.h public_namespace.h jemalloc_preamble.h; do
45+
cp include/jemalloc/internal/$i ../../lib/libc/stdlib/malloc/jemalloc/include/jemalloc/internal/
46+
git add ../../lib/libc/stdlib/malloc/jemalloc/include/jemalloc/internal/$i
47+
rm include/jemalloc/internal/$i
48+
done
49+
# OK, commit all the generated files
50+
git add VERSION
51+
git commit --amend
10252

103-
do_extract_diff() {
104-
local rev=$1
105-
local repo=$2
106-
do_extract_helper ${rev} ${repo}
107-
(
108-
cd ${repo}
109-
# Apply diffs before generating files.
110-
patch -p1 < "${src}/FREEBSD-diffs"
111-
find . -name '*.orig' -delete
112-
# Generate files.
113-
do_autogen
114-
gmake dist
115-
)
116-
}
53+
# Clean up the mess
54+
git clean -f .
11755

118-
do_extract_namespace() {
119-
local rev=$1
120-
local repo=$2
121-
do_extract_helper ${rev} ${repo}
122-
(
123-
cd ${repo}
124-
# Generate files.
125-
do_autogen
126-
gmake include/jemalloc/internal/private_namespace.h
127-
)
128-
}
56+
# Save the cheat sheet
57+
cp ~/jemalloc-upd FREEBSD-upgrade
58+
git add FREEBSD-upgrade
59+
git commit --amend
12960

130-
do_extract() {
131-
local rev=$1
132-
do_fetch ${rev}
133-
do_extract_diff ${rev} ${work_repo}
134-
do_extract_namespace ${rev} ${namespace_repo}
135-
}
61+
# Remove hash.c from lib/libc/stdlib/malloc/jemalloc/Makefile.inc
62+
# mutex_pool.c prng.c
63+
# Add
64+
# bin_info.c san.c san_bump.c counter.c prof_data.c prof_log.c prof_recent.c prof_stats.c prof_sys.c
65+
# emap.c edata.c edata_cache.c pa.c pa_extra.c pac.c decay.c hpa.c hpa_hooks.c fxp.c hpdata.c pai.c
66+
# ecache.c ehooks.c eset.c sec.c cache_bin.c peak_event.c psset.c inspect.c exp_grow.c thread_event.c
67+
#
13668

137-
do_diff() {
138-
(
139-
cd ${work_repo}
140-
find . -name '*.orig' -delete
141-
find . -name '*.rej' -delete
142-
git add -A
143-
git diff --cached
144-
) > FREEBSD-diffs
145-
}
69+
# Manually comment out the following in lib/libc/stdlib/malloc/jemalloc/include/jemalloc/jemalloc.h
70+
# /* #define JEMALLOC_HAVE_ATTR_FORMAT_GNU_PRINTF */
71+
# Add
72+
# #define JEMALLOC_OVERRIDE_VALLOC
73+
# and #include "jemalloc_FreeBSD.h"
74+
# With some adjustments to the old jemalloc_FreeBSD.h, but git can help
14675

147-
command=$1
148-
shift
149-
case "${command}" in
150-
merge-changes) # Merge local changes that were made since the previous import.
151-
rev=`cat VERSION |tr 'g' ' ' |awk '{print $2}'`
152-
# Extract code corresponding to most recent import.
153-
do_extract ${rev}
154-
# Compute local differences to the upstream+patches and apply them.
155-
(
156-
cd ${tmpdir}
157-
diff -ru -X ${src}/FREEBSD-Xlist ${work} ../jemalloc > ${changes} || true
158-
)
159-
(
160-
cd ${work_repo}
161-
patch -p1 < ${changes} || true
162-
find . -name '*.orig' -delete
163-
)
164-
# Update diff.
165-
do_diff
166-
;;
167-
extract) # Extract upstream sources, apply patches, copy to contrib/jemalloc.
168-
rev=$1
169-
do_extract ${rev}
170-
# Delete existing files so that cruft doesn't silently remain.
171-
rm -rf ChangeLog COPYING VERSION doc include src
172-
# Copy files over.
173-
tar cf - -C ${work_repo} -X FREEBSD-Xlist . |tar xvf -
174-
internal_dir="include/jemalloc/internal"
175-
grep -v ' isthreaded ' \
176-
"${namespace_repo}/${internal_dir}/private_namespace.h" \
177-
> "${internal_dir}/private_namespace.h"
178-
;;
179-
rediff) # Regenerate diffs based on working tree.
180-
do_diff
181-
;;
182-
clean) # Remove working tree and temporary files.
183-
rm -rf ${tmpdir} ${changes}
184-
;;
185-
*)
186-
echo "Unsupported command: \"${command}\"" >&2
187-
exit 1
188-
;;
189-
esac
76+
# Had to manually remove
77+
# -#define __malloc_options_1_0 JEMALLOC_N(__malloc_options_1_0)
78+
# -#define _malloc_first_thread JEMALLOC_N(_malloc_first_thread)
79+
# -#define __malloc_message_1_0 JEMALLOC_N(__malloc_message_1_0)
80+
# -#define isthreaded JEMALLOC_N(isthreaded)
81+
#
82+
# Also had to remove the following to fix jemalloc 3 ABI compat
83+
# -#define je_allocm JEMALLOC_N(je_allocm)
84+
# -#define je_dallocm JEMALLOC_N(je_dallocm)
85+
# -#define je_nallocm JEMALLOC_N(je_nallocm)
86+
# -#define je_rallocm JEMALLOC_N(je_rallocm)
87+
# -#define je_sallocm JEMALLOC_N(je_sallocm)
88+
# Without the diff you end up with non-exported _je_je*allocm symbols. With you get symbols of the form:
89+
# 365: 000000000018e2a0 406 FUNC WEAK DEFAULT 14 rallocm@FBSD_1.3 (5)
90+
# 657: 000000000018e2a0 406 FUNC GLOBAL DEFAULT 14 __rallocm@FBSD_1.3 (5)
91+
#

0 commit comments

Comments
 (0)