Replace jemalloc dependency with custom interpose layer

[hassila]

Jemalloc statistics buys us per-thread aggregatable statistics with the downside of one additional requirement.

Should consider rolling our own interposition here (can be a separate product library), which may have slightly more performance impact, but will enable running benchmarks with malloc stats without adding the dependency (and works around problems with macOS xctest/Xcode usage of benchmarks).

We can base it off the work done here by the NIO team:

https://github.com/apple/swift-nio/tree/main/IntegrationTests/allocation-counter-tests-framework

But should expand it with statistics that covers the metrics we already support (e.g. small/large allocs etc).

[hassila]

Related at the same time is #303

[tmcdonell]

I should also point out that changing the base allocator has a fundamental impact on performance. I'm seeing swings of 50% (plus and minus) with BENCHMARK_DISABLE_JEMALLOC set or not. YMMV; this will depend heavily on the application.

Using jemalloc by default makes the results of package-benchmark utterly meaningless.

[hassila]

utterly meaningless

I guess YMMV.

[tmcdonell]

As I already said. But the fact that this is the default behaviour and it is not documented anywhere, means your users probably have no idea that the results reported by your package might be completely different from their in-production performance, and may lead them to optimise their program in the wrong direction.

Additionally your comment:

Should consider rolling our own interposition here ..., which may have slightly more performance impact, but will enable running benchmarks with malloc stats without adding the dependency

Indicates to me that you also are unaware of the impact an allocator can make (and which is why there is an entire cottage industry of custom allocators).

So this seems like a collective blind spot.

[hassila]

Apart from being a bit offensive in your assumptions of my experience with regard to custom allocators (which happens to be way off the mark, but then you neither know me or know what I have been workin on...), I'll not go further into polemics with you and will keep this case to be about removing the dependency. The YMMV comment was clearly directed to your view that this makes the whole package utterly useless, not confirming yours comment.

Anyway, won't comment more here, but I would recommend then if you are heavily malloc bound to measure the malloc usage with jemalloc to reduce it, while measuring timed metrics to do it without jemalloc if you have an issue with.

[tmcdonell]

That's fair, and I apologise for my comment being overly hostile. That was out of line.

I would recommend then ... to measure the malloc usage with jemalloc to reduce it, while measuring timed metrics to do it without jemalloc

I think this advice is worthwhile to share more broadly.

[hassila]

The foundational work of built-in interposition is progressing nicely in #333 picking up parts of the NIO interposition work, unfortunately we are a bit stuck due to a SwiftPM issue at the moment swiftlang/swift-package-manager#9062 - trying to find some way to work around it...