Add a directory depth flag to show N number of sub-directories

[nickjj]

By default if I run dua I see this:

   1.52 GiB src
   1.65 GiB .local
  12.01 GiB .cache
  16.95 GiB .docker
  33.39 GiB total

This helps me know at a glance what's pretty big on my file system.

But it would be great to know why .cache and .docker are so big.

The dust tool outputs this for reference:

1.1G   │   ┌── firefox         │██████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                                                                                                        │   3%
1.1G   │ ┌─┴ mozilla           │██████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                                                                                                        │   3%
1.1G   │ │ ┌── elephant-bin    │██████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                                                                                                        │   3%
1.1G   │ ├─┴ yay               │██████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                                                                                                        │   3%
3.0G   │ │   ┌── elephant-bin  │███████████████▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░░░░░░░                                                                                                        │   9%
4.6G   │ │   ├── docker-desktop│███████████████████████▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░░░░░░░                                                                                                        │  14%
9.1G   │ │ ┌─┴ clone           │████████████████████████████████████████████░░░░░░░░░░░░░░░                                                                                                        │  27%
9.1G   │ ├─┴ paru              │████████████████████████████████████████████░░░░░░░░░░░░░░░                                                                                                        │  27%
 12G   ├─┴ .cache              │███████████████████████████████████████████████████████████                                                                                                        │  36%
 16G   │         ┌── Docker.raw│███████████████████████████████████████████████████████████████████████████████████                                                                                │  51%
 16G   │       ┌─┴ data        │███████████████████████████████████████████████████████████████████████████████████                                                                                │  51%
 16G   │     ┌─┴ 0             │███████████████████████████████████████████████████████████████████████████████████                                                                                │  51%
 16G   │   ┌─┴ vms             │███████████████████████████████████████████████████████████████████████████████████                                                                                │  51%
 16G   │ ┌─┴ desktop           │███████████████████████████████████████████████████████████████████████████████████                                                                                │  51%
 16G   ├─┴ .docker             │███████████████████████████████████████████████████████████████████████████████████                                                                                │  51%
 33G ┌─┴ nick                  │██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ │ 100%

At a glance I can see what's up.

Of course dua has interactive mode which is nice for casually combing through a directory to get these details but I find interactive mode to take more time than getting a dump for some use cases.

If dua had a way to show the biggest N sub-directories with a --depth flag that would be great. You could run it with something like dua --depth 3 and it would list the 3 biggest sub-directories in the parent directory. dust has this option and it defaults to a reasonable number to let you quickly scan things.

If dua had this feature then I think I wouldn't need to install dust.

What do you think?

[Byron]

Thanks for sharing, that's an interesting proposition.

Funnily enough, personally I tend to use pdu to get an initial idea on huge directories, whereas dua is just perfect for the typical source directory. So in a way, it seems we are talking about making pdu-like functionality available via dua.

The problem I see here is that the underlying jwalk simply isn't as fast and suffers from huge overhead, and I don't think dua is ever going to be better or faster than pdu that way.

So it seems that one could just pull pdu in as a library and call it a day 😅.
And it looks like doing so would be perfectly possible.

But then again, without any additional benefits, it feels like keeping the programs separate would be preferable, unless one would have an actual library version of pdu that doesn't add substantial size to the dua binary.

[nickjj]

Thanks for the response, I didn't even know pdu existed. The readme makes it seem like it's a slightly faster version of dust.

This post came from of an urge to simplify things by having 1 tool to quickly scan and understand what's taking up disk space. One that had the best of both worlds, a quick dump with "just enough" info to usually get answers but also a way to interactively explore.

Realistically it's not a big deal to use 2 different tools. When you say it's perfect for the typical source directory, do you mean when you've identified a big directory and now you want to pry deeper?

[Byron]

This post came from of an urge to simplify things by having 1 tool to quickly scan and understand what's taking up disk space. One that had the best of both worlds, a quick dump with "just enough" info to usually get answers but also a way to interactively explore.

I saw that, I think that need is coming from getting an initial analysis of huge folders. dua isn't great for that as it's too coarse there.

When you say it's perfect for the typical source directory, do you mean when you've identified a big directory and now you want to pry deeper?

No, it's meant to be used in well-known folders where it's known that certain directories grow fast, like target in Rust projects. dua can be used to check if it's a good time for cargo clean. And that's really what it's meant to be good for.

However… it kind of seems that pdu is better at that, in principle, as it uses less lines than dua. With some tuning, pdu could have the better visualisation, maybe one that is easier to grasp as well.

❯ pdu
234.1M     ┌──objects                   │██                                                                                                                                                           │  1%
237.1M   ┌─┴.git                        │██                                                                                                                                                           │  1%
388.2M   │ ┌──cache                     │███                                                                                                                                                          │  2%
411.5M   ├─┴.turbo                      │███                                                                                                                                                          │  2%
851.3M   │ ┌──.pnpm                     │███████                                                                                                                                                      │  4%
851.5M   ├─┴node_modules                │███████                                                                                                                                                      │  4%
513.1M   │     ┌──build                 │████▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░              │  3%
674.7M   │     ├──libgitbutler_tauri.a  │█████▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░              │  3%
674.7M   │     │ ┌──libgitbutler_tauri.a│█████▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░              │  3%
  4.3G   │     ├─┴deps                  │██████████████████████████████████▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░              │ 22%
  5.7G   │   ┌─┴debug                   │█████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░              │ 28%
  5.7G   │ ┌─┴tauri                     │█████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░              │ 28%
630.8M   │ │ ┌──build                   │█████▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░              │  3%
 11.7G   │ │ ├──deps                    │████████████████████████████████████████████████████████████████████████████████████████████▒▒▒▒▒▒▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░              │ 59%
 12.6G   │ ├─┴debug                     │███████████████████████████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░              │ 63%
 18.3G   ├─┴target                      │███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████              │ 91%
 20.0G ┌─┴.                             │█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│100%

gitbutler ( fix) [$] via 
❯ dua
      0  B etc
   4.10 KB .cargo
   4.10 KB .envrc
   4.10 KB .gitattributes
   4.10 KB .gitignore
   4.10 KB .mailmap
   4.10 KB .npmrc
   4.10 KB .nvmrc
   4.10 KB .prettierignore
   4.10 KB .prettierrc.mjs
   4.10 KB .taurignore
   4.10 KB CODEOWNERS
   4.10 KB CODE_OF_CONDUCT.md
   4.10 KB CONTRIBUTING.md
   4.10 KB LINUX.md
   4.10 KB SECURITY.md
   4.10 KB aoc.md
   4.10 KB claude.local.md
   4.10 KB dad-jokes.md
   4.10 KB ignore-vercel.sh
   4.10 KB package.json
   4.10 KB pnpm-workspace.yaml
   4.10 KB rust-toolchain.toml
   4.10 KB rustfmt-nightly.toml
   4.10 KB tsconfig.json
   4.10 KB turbo.json
   8.19 KB .vscode
   8.19 KB Cargo.toml
   8.19 KB LICENSE.md
   8.19 KB README.md
   8.19 KB debug-graph-00.svg
   8.19 KB eslint.config.js
  12.29 KB deny.toml
  16.38 KB DEVELOPMENT.md
  40.96 KB scripts
  49.15 KB debug-graph-01.svg
  98.30 KB .github
 110.59 KB readme-preview.webp
 266.24 KB .idea
 294.91 KB Cargo.lock
 512.00 KB pnpm-lock.yaml
 622.59 KB e2e
  33.32 MB crates
  85.02 MB packages
 128.86 MB apps
 237.07 MB .git
 411.51 MB .turbo
 851.46 MB node_modules
  18.26 GB target
  20.01 GB total

If the top-level of each folder was printed in bold, I think the big ones would pop even more.

Maybe that's an avenue? Maybe dua should evolve to effectively print like pdu?
If pdu was usable as a library, I think this is what should happen. What do you think @KSXGitHub? I think I would have fun contributing the 'libraryfication' of pdu so it can be the default visualisation of dua, which would also finally get rid of jwalk at least for one common invocation.

[KSXGitHub]

@Byron

If pdu was usable as a library, I think this is what should happen. What do you think @KSXGitHub? I think I would have fun contributing the 'libraryfication' of pdu so it can be the default visualisation of dua, which would also finally get rid of jwalk at least for one common invocation.

I think the parallel-disk-usage library crate already has pure Rust APIs to turn a data structure into visualization. The pdu binary also has --json-input mode to turn a JSON data structure into visualization.

However, I don't know yet what APIs pdu still lacks to provide dua necessary building blocks.

If you can contribute, I'd appreciate.

Do note that I'd like to keep both the binary and the library in one single crate for now. The minor version (which, when major is 0, indicates breaking changes) should bump regardless of whether the breaking changes come from the CLI or the library APIs.

Maybe there's an advantage to making a separate parallel-disk-usage-lib, especially once pdu reach 1.0. But pdu is like a personal project with low commitment, so it's unlikely that I'd ever make it happen.

[Byron]

Thanks a lot for chiming in, it's much appreciated!

It makes total sense not to want to add additional burden by supporting parralel-disk-usage as library, or even reviewing contributions.

Thus, I think it's best if I'd just copy the needed pieces over here if the license permits, and with proper attributions. From there the code can develop independently.

Does that sound more feasible to you?

[KSXGitHub]

Using it as a library crate is fine. But copying the code to your repo is fine too.

If you choose to use the library crate, I'd expect you to make requests, PRs, or other contributions for missing APIs you need.

If you choose the later, everything in your "fork" would be your responsibility alone.