Does this project make use of any of these techniques?

[bejolly407]

I'm not the guy that originally came up with such question. I take the template from here: SpartanJ/eepp#59

• Build with sanitizers

• Run static analyzers

• Memory profiling

It seems none of these are being utilized. Even UBSan is not being used. Quoted the w64devkit developer on skeeto/w64devkit#241:

• The project's allocator is broken and allocates misaligned objects. I
  initially made a debug build with UBSan to investigate, and I had to give
  up on UBSan because UBSan was tripping on UB everywhere. I expect this is
  easy to fix, and it's really a lack of awareness.

I do not see any debug/development builds in their build system (how are they
actually working on this code?), but it's achievable with this:

$ make all_amd64 CFLAGS_RELEASE='-municode -g3' LDFLAGS_RELEASE=

[ghost]

I know the guy posted these questions. He asked here, too: objeck/objeck-lang#504

Thanks to the Objeck developer's answer, I know about Coverity and CodeQL.

[arakov]

No, I don't use them. And it has nothing to do with lack of awareness and everything with lack of time.

If someone would like to help and do it I wouldn't mind. But so far, my philosophy is extremely simple: if it is not broken, don't fix it. If it works I'm happy with the current state. My goal is to develop the language, so the questions about the compiler are not relevant so far. If one day it becomes popular, then we could speak about performance, optimizations and so on.

[arakov]

And just another remark. I'm developing this project mostly on Windows + MSBuild. So I actually use GCC / CLANG mostly only when I testing my changes before release. I never have time to look through all recommendations (most of them is honestly crap anyway).

So if you see any issue with the compilation, please create a ticket, so I could take a look when I will have time.

[bejolly407]

MSVC also has sanitizers. Should you use them?

[arakov]

Sure, I could use them, no problem. But my point is - do not fix something that works (for example I fixed around 200 warnings recently and none of them were critical or even buggy; does it somehow improve my code, I'm not sure at all). It was a long time since I had some serious errors due to the quality of the code. In most cases, that these tools are considering as a potential bug, is in fact some optimization hack on my side. I don't want to initialize every field, only to do it once again in the next step. And so on and on spending hours to make analyzers happy. Look, there are so many warnings about order of field initialization, but there were never a single issue due to this.

As I said if the compiler will be actually used by people, all these potential things will appear and in that state, it would be cheaper to use these tools then fixing a bug after bug. But currently I'm the only one who find and fix bugs. And so far none of them due to the code quality.

Sure I don't say that my code is perfect. I do have some serious potential problems for example for RISC CPUs, but because no one use them, I can fix them when I'm in the mood to spend hours looking into assembly code

[arakov]

• The project's allocator is broken and allocates misaligned objects. I

Just out of curiosity, what the hell he is speaking about. Which allocator?
Can someone recreate these findings?

[bejolly407]

@skeeto

[skeeto]

On Linux this time so there's a diagnostic:

$ make all_amd64 CFLAGS_RELEASE='-g3 -fsanitize=undefined -fPIC' LDFLAGS_RELEASE='-fsanitize=undefined'
$ bin/sg64-cli dat/sg/syntax60.txt
ELENA command line syntax generator 6.7.7 (C)2005-2025 by Aleksey Rakov
../../../common/lists.h:289:47: runtime error: reference binding to misaligned address 0x5555b503a4ae for type 'unsigned int', which requires 4 byte alignment
...
../../../common/lists.h:1866:23: runtime error: load of misaligned address 0x5555b503a4ae for type 'unsigned int', which requires 4 byte alignment
...
generating...
saving...
../../../common/lists.h:1842:28: runtime error: load of misaligned address 0x5555b506a4ce for type 'unsigned int', which requires 4 byte alignment
...
../../../common/lists.h:1855:19: runtime error: load of misaligned address 0x5555b506a4ce for type 'unsigned int', which requires 4 byte alignment
...

Both GCC and Clang report the same error, so it's certainly not a compiler bug. The first is on this:

      T& operator*() const { return _current->item; }

Where _current comes from this suspicious-looking pointer cast into a buffer, like this:

            _current = (Item*)_buffer->get(_current->nextOffset);

Objects are, in effect, allocated from this buffer. In normal circumstances they're not being aligned correctly, leading to UB starting at the cast and detectable by UBsan on dereference. However, they're #pragma pack allowing unaligned access of members. But then notice the "reference binding" in the error. The member's address is taken for the return reference, which is UB for unaligned members of packed structs. The receiver of the address cannot know it's the address of a packed member. It just sees, in this case, an unsigned int reference.

The options are (1) stop taking the address of packed members, or (2) stop packing them and instead align them in the buffer.

(I tell everyone to enable sanitizers during all testing and development, not to wait until later, because it immediately reveals these sorts issues, when they're first introduced and easier to address.)

[arakov]

It is exactly why I don't like that type of sanitizers.

_current = (Item*)_buffer->get(_current->nextOffset);

It is not a bug at all. The main idea is to allocate a list as a single memory block, so no need to allocate the individual elements one by one. As a result the whole list can be saved / loaded in a single operation. So it is quite a significant performance gain.

Misaligned pointer here is not a bug either. In the worst case it gives you some performance penalty. The pointer refers to allocated dynamic memory, so CPUs are ok with it. This particular code refers most probably to the symbol map (a list of strings), which practically never used. So in this case the performance penalty is minuscule.

This code works for more than a decade. So you would presume that any nasty things would be ironed long time ago.

So from one side, this sanitizer found a minor issues with performance which can slightly increase the speed of calculation at the cost of size (the copying of the string, but does not affect comparison, because a UTF8 string is a pointer to the byte and the correct alignment is 1; but any such string must be aligned to 8, because the safest alignment for the pointer is 8 in normal memory and 16 in the stack - for 64 bit architecture).

From another side it will generate a significant number of false-positive. Like complaining that the local variable is not initialized, then in fact the code specifically check if the initialization was made.

It is a trade-off, you have to resolve every time individually.

P.S. And about allocating. In most cases these structures (they are all pure structures, without any virtual methods / destructors) are copied into local variables (that's why an iterator returns the reference). So no allocation whatsoever is taken place. Once again this code works for a decade. It works exactly how it was designed.

So the option is (1) stop giving an advice in the situation you have no clues

[skeeto]

UBSan, by definition, has no false positives. It's expressing the compiler's actual view of the program. If it trips, it means the program did something at run time that the compiler assumes never happens. The compiler generates code under this assumption, and so if the assumption is wrong so may be the generated code. Lots of undefined behavior will appear to be fine, sometimes for decades, until suddenly it isn't, often subtly and indirectly. So if UBSan trips there's absolutely a bug somewhere, no question about it, either in the program or in the compiler, even if it usually works out fine by chance.

This program would definitely crash — probably with a bus error — at this point on a host with stricter memory alignment (e.g. mips, sparc). Though it's still UB even if the underlying ISA is usually flexible about alignment. These aren't low-level loads and stores, but variables in a high-level programming language which mandates aligned addresses, except for certain special cases like packed structs (a compiler extension). It has nothing to do with performance, nor with calling convention stack alignment.

(I understand you don't believe me, and I honestly don't care about changing your mind on this. I'm just here to answer "what the hell I'm speaking about." Per the linked thread, this was one of five bugs I observed while investigating something that turned out benign. This situation is pretty common, and it's wasteful not to capture such observations when I come across them. I've never regretted writing them down, and on occasion I've regretted when I didn't.)

[arakov]

It would spare a lot of time, if you simply submit the bug report. Surely I've fixed this issue with the string alignment. And normally I just thank anyone for the help.

In this case it starts with suggestion that I have no idea what I'm doing. It is not very helpful. Just look at your first post, how can I deduce the problem from this?

About alignment. The C / C++ is perfectly ok with the following code:

char tmp[10]  = {};
int* ptr = (int*)(tmp + 1);
*ptr = 0x030201;

So any language implementation must take into account such posibility.

Let's agree to disagree on alignment significance in this case. Anyway thanks for your suggestion.

But to make that type of a sanitizer happy, I've resolved it here - 2720fd2

[arakov]

Just for information. I've run a memory leak test and as expected none was detected. I've used my biggest project - system.prj.

Sorry for keep posting here, but I was really triggered by these unproven claims.

[arakov]

It is so far the longest discussion :) I have on GitHub. Maybe I was to harsh in some of my replies. For this I'm sorry.

But I would like to reiterate my position. If you see some points to improve, please create an issue with your suggestion. People did this, and I fixed them and I'm grateful for this.

But don't try to lecture other people like they have no idea what they are doing, like in this case.