Reported size of `cargo clean` is incorrect?

[RandomInsano]

Problem

The reported amount of disk space cleaned is not the same as reported by du -sh . on Mac and Linux.

Hopefully this isn't a megabyte versus mebibyte problem...

Steps

Run the following:

git clone https://github.com/rust-lang/cargo
cd cargo
cargo build --release
echo "Size before cleaning:"
du -sh .
cargo clean
echo "Size after cleaning:"
du -sh .

Possible Solution(s)

No response

Notes

Here are the results of my run on macOS:

Size before cleaning:
705M    .
     Removed 2811 files, 614.4MiB total
Size after cleaning:
 82M    .

705-614 = 91MB

Here's the results on Debian Linux 12:

Size before cleaning:
700M    .
     Removed 2938 files, 702.2MiB total
Size after cleaning:
 94M    .

700 - 702 = -2

Version

macOS: 1.89.0
Linux: 1.89.0

[RandomInsano]

Just confirmed that the "human output" for du isn't that far off:

$ du -sm .
706     .
$ du -sh .
705M    .

[hanna-kruppe]

Cargo clean sums up the sizes that fs::metadata() reports while du (1) reports size-on-disk rather than how many bytes could be read from it, unless you pass --apparent-size and (2) only counts multiple hard links to the same file once, unless you pass --count-links. There may be other differences as well but I suspect these two account for most of the delta.

[RandomInsano]

Thanks for the detailed explanation Hanna! I definitely would expect the total to provide the amount of bytes made available from cleaning up so I presume this should be classified as a bug, though a minor one. I've been using Rust in a hobby capacity for over ten years now and have never noticed.

[epage]

actual vs allocated disk space discrepancies are a common occurrence when dealing with file systems. The difference is small enough that I'm unsure how much of a difference it makes. I also did some searching, and it seems non-obvious how to get the allocated space. On the surface, it looks like you can use st_blksize * st_blocks but st_blksize seems to be OS and filesystem-defined so I'm unsure if we can properly interpret this. Unsure what du uses.

[RandomInsano]

Looks like this is it for FreeBSD (macOS):

/*
 * Record the total disk/block size of the file or directory. The fts_number
 * variable provided in FTSENT is used for keeping track of the total size.
 * See FTS(3).
 */
static void
record_file_size(FTSENT *p)
{
	p->fts_number += Aflag ?
	    howmany(p->fts_statp->st_size, cblocksize) :
	    howmany(p->fts_statp->st_blocks, cblocksize);

	p->fts_parent->fts_number += p->fts_number;
}

https://github.com/freebsd/freebsd-src/blob/main/usr.bin/du/du.c#L548

For Linux (coreutils' code is much harder to read)

duinfo_set (&dui,
              (apparent_size
               ? (usable_st_size (sb) ? MAX (0, sb->st_size) : 0)
               : (uintmax_t) STP_NBLOCKS (sb) * ST_NBLOCKSIZE),
              (time_type == time_mtime ? get_stat_mtime (sb)
               : time_type == time_atime ? get_stat_atime (sb)
               : get_stat_ctime (sb)));

https://github.com/coreutils/coreutils/blob/master/src/du.c#L569C3-L576C1

As far as the "The difference is small enough" comment, I don't think I did a very good job of calling it out before. For Linux specifically, you'll see that -2MB difference versus the 94 is a total difference of 96MB. Considering the total is 702MB reported, that's a 13% difference. How many people really care about that difference, I have no idea but it's a weird one once you notice it.

[hanna-kruppe]

96 MB / 2986 files seems too large to be down to block size rounding. If it was due to sparse files, du would report smaller size than cargo clean. Hard links seem a more plausible explanation for the bulk of the delta (you could repeat your experiment with du --count-links to verify). This may be easier to address in a portable manner than block size.

[RandomInsano]

Just on my mac at the moment, so I'm in FreeBSD user land right now. I did try accounting for hard links with -l:

     -l      If a file has multiple hard links, count its size multiple times.  The default behavior of du is to count
             files with multiple hard links only once.  When the -l option is specified, the hard link checks are
             disabled, and these files are counted (and displayed) as many times as they are found.

That didn't make much difference:

$ cargo build --release
$ du -sh .
704M	.
$ du -l -sm .
705	.
$ cargo clean
     Removed 2811 files, 614.3MiB total
du -l -sm .
82	.

705-614.3=90.7 so we're out 90.7-82=8.7

Another thought; could this be file slack? 8,700,000 / 2811 files is 3,094 bytes. That feels plausible, and it looks like the minimum block size on my APFS volume is 4K from a test creating a simple text file. Finder reports "6 bytes (4 KB on disk)"