Please support generating an erofs from a manifest/pack-file

[bdrung]

I want to construct a erofs with files from the host system (take Dracut as use-case). Currently I have to create a directory and copy all files into it to map their path to a path in the erofs. Then I can call mkfs.erofs to create an erofs. Copying the files is one extra step that I like to avoid.

Please support generating an erofs from a manifest/pack-file. Other projects that have this support:

• 3cpio supports a manifest file: https://github.com/bdrung/3cpio/blob/main/man/3cpio.1.adoc#manifest
• mksquashfs (from squashfs-tools >= 4.5) support a pseudo file: https://manpages.debian.org/unstable/squashfs-tools/mksquashfs.1.en.html#PSEUDO_FILE_DEFINITION_FORMAT
• gensquashfs (from squashfs-tools-ng) support a pack file: https://manpages.debian.org/trixie/squashfs-tools-ng/gensquashfs.1.en.html#PACK_FILE_FORMAT

[hsiangkao]

@bdrung the feature itself it fine, but the issue is that those formats are highly fragmented, I'd like to find a standardized format first, but this feature is hard to finish in the upcoming erofs-utils 1.9 version.

[bdrung]

That is quite a valid issue. I am the author of 3cpio and failed to find a standardized format, so I came up with something that works as manifest file but is also compatible by just using find to generate the manifest. I would be happy to change 3cpio use a standardized format instead.

What do you think about collaborating in coming up with a standard format? A good place for this kind of specification would be https://uapi-group.org/.

[hsiangkao]

My perference is this one: https://www.ibm.com/docs/en/aix/7.3.0?topic=m-mkproto-command
and it seems it supported since unix era (xfs also supports it for historial reasons).

I'm not sure if it looks good to you, or just enhance it as an improvement.

[bdrung]

The mkproto prototype file is hard to generate, since you cannot operate line-based on it. Instead of adding

dir1/regfile3    ---- 644 0 0   /tmp/proto.examp/dir1/regfile3

you have to modify the file to insert that line at the correct place:

 dir1    d--- 755 0 2
  regfile3    ---- 644 0 0   /tmp/proto.examp/dir1/regfile3
  $

I would much prefer to be able to just add a file by adding a file to the manifest.

The second thing that looks weird to me is column two which requires four characters. I would make the mode take values to set the set UID/GID and sticky bit.

I would probably use https://manpages.debian.org/unstable/squashfs-tools/mksquashfs.1.en.html#PSEUDO_FILE_DEFINITION_FORMAT as starting point.

Looking at all the file formats, I would say we should use a format that is:

• text based
• line based (one line per file) (no JSON or such)
• allows comments (e.g. starting with #)

[bdrung]

Bonus point if the format allows to just use find . to create a simple version of that manifest (or some find . | sed '$simplerule').

The UAPI location do discuss such specification is: https://github.com/uapi-group/specifications/issues

[hsiangkao]

The mkproto prototype file is hard to generate, since you cannot operate line-based on it. Instead of adding

dir1/regfile3    ---- 644 0 0   /tmp/proto.examp/dir1/regfile3

you have to modify the file to insert that line at the correct place:

 dir1    d--- 755 0 2
  regfile3    ---- 644 0 0   /tmp/proto.examp/dir1/regfile3
  $

I would much prefer to be able to just add a file by adding a file to the manifest.

So, it seems you just need to enhance the format a little bit,like:

dir1    d--- 755 0 2 $
dir1/regfile3    ---- 644 0 0   /tmp/proto.examp/dir1/regfile3

The second thing that looks weird to me is column two which requires four characters. I would make the mode take values to
set the set UID/GID and sticky bit.

Then just try to support the modified version ({ - | d | b | c | l | L | p } + numerial modes), I don't see it's totally incompatible.

I still perfer a modernized modified protofile instead since it's supported for decades since unix era., and it seems can just meet your requirements (including comments.)

[hsiangkao]

I would probably use https://manpages.debian.org/unstable/squashfs-tools/mksquashfs.1.en.html#PSEUDO_FILE_DEFINITION_FORMAT as starting point.

Obviously, it is against your line based (one line per file) (no JSON or such), and the xattr representation (especially for binary xattr values), and another command set of +timestamp version are all odd to me.

I'm fine to support random formats like squashfs pseudo files as long as someone implements it for erofs-utils, but it seems not to be either the most reasonable one or most compatible one (unix proto files), so personally I will try to improve protofiles instead for better compatible reasons.

[bdrung]

I just noticed that https://www.ibm.com/docs/en/aix/7.3.0?topic=m-mkproto-command and https://www.unix.com/man-page/v7/1M/mkfs differ (d--755 vs space VS d--- 755 with space). A modernized modified protofile would still be a new format.

Let me write down what I had in mind:

#foobar v1
# First line is a file specification marker (needed for detection and highlighting), comments start with #
<filename> f <mode> <uid> <gid> <mtime> <filesize> <souce_filename>
<filename> d <mode> <uid> <gid> <mtime>
<filename> b <mode> <uid> <gid> <mtime> <major> <minor>
<filename> c <mode> <uid> <gid> <mtime> <major> <minor>
<filename> l <mode> <uid> <gid> <mtime> <target>
<filename> p <mode> <uid> <gid> <mtime>
<filename> s <mode> <uid> <gid> <mtime>

• filename: Path in the archive
• f = file, d = directory, b =block device, c = character device, l = symbolic link, p = pipe/fifo, s = socket
• mode: File mode specified in octal (e.g. 2755)
• uid: User ID (owner) of the file specified in decimal.
• gid: Group ID of the file specified in decimal.
• mtime: Modification time of the file specified as seconds since the Epoch (1970-01-01 00:00 UTC).
• filesize: Size of the input file in bytes.
• major: Major block/character device number in decimal.
• minor: Minor block/character device number in decimal.
• target: Target of the symbolic link.

Rules:

• Tokens are separated by space or tab.
• filename, source_filename, target can be quoted (to include spaces)

[hsiangkao]

My own suggestion:
for regular files, I don't think makes sense since it should follow the source file all the time.
Also I think mtime should be optional since erofs can use a default unique time instead if not given, how about adding a optional_prefix like "mtime:" so that the format can be identified, and arrange it in the following order:
<filename> f <mode> [mtime:<mtime>] <uid> <gid> <souce_filename>
so that mtime: won't be confused with special <source_filename>

[bdrung]

for regular files, I don't think makes sense since it should follow the source file all the time.

which part are you referring to?

We could make tokens optional: Use - to mark them as not specified.

Alternative: Use a prefix. That would make the whole extensible.

#foobar v1
# First line is a file specification marker (needed for detection and highlighting), comments start with #
<filename> f [mode:<mode>] [uid:<uid>] [gid:<gid>] [mtime:<mtime>] [size:<filesize>] [source:<souce_filename>]
<filename> d [mode:<mode>] [uid:<uid>] [gid:<gid>] [mtime:<mtime>]
<filename> b [mode:<mode>] [uid:<uid>] [gid:<gid>] [mtime:<mtime>] <major> <minor>
<filename> c [mode:<mode>] [uid:<uid>] [gid:<gid>] [mtime:<mtime>] <major> <minor>
<filename> l [mode:<mode>] [uid:<uid>] [gid:<gid>] [mtime:<mtime>] <target>
<filename> p [mode:<mode>] [uid:<uid>] [gid:<gid>] [mtime:<mtime>]
<filename> s [mode:<mode>] [uid:<uid>] [gid:<gid>] [mtime:<mtime>]

What needs to be figured out: how to handle hardlinks, xattr, and acls?

[bdrung]

We could add another type where you can just say: take the information (type, mode, uid, gid, mtime) from the source file.

[bdrung]

Or use short prefixes: m: for mode, u: for uid, g: for gid, etc.

[hsiangkao]

for regular files, I don't think makes sense since it should follow the source file all the time.

which part are you referring to?

sorry I missed the word, I meant the filesize.