Support more/larger xattrs? (now = "no space left on device")

[pjotrek-b]

Bug report

Bug description:

When trying to copy all embedded metadata key/value information from the IPTC standard-reference image, I get the following error:

[Errno 28] No space left on device: 'xattr-write-target-filename'

os.setxattr(target, prefix + strkey, strval.encode())

The total size of the exiftool-json is 15kB. More than 4k, but:

On kernel.org Wiki (ext4:xattrs), it says:

The paragraph is titled "Large Extended Attribute Values":

To enable ext4 to store extended attribute values that do not fit in the inode or in the single extended attribute block attached to an inode, the EA_INODE feature allows us to store the value in the data blocks of a regular file inode. This “EA inode” is linked only from the extended attribute name index and must not appear in a directory entry.

So I guess ext4 should be able to handle it?
My code works fine for writing shorter/less xattrs.

Does my blocksize matter? (512 bytes)
Is there any way to check what the limit is?
And which component in the stack actually decides this "out of diskspace" error?

Thank you very much in advance 😄
(PS: This was on Xubuntu 20.04.6 - python 3.8.10. Wasn't available in the dropdown)

CPython versions tested on:

3.8.10

Operating systems tested on:

Linux

[ronaldoussoren]

Have you tried setting the xattr using the fsetfattr(1) command-line tool? If that results in the same error this is a limitation in linux or ext4.

My reading of https://man7.org/linux/man-pages/man5/ext4.5.html is that EA_INODE is enabled by a filesystem feature flag, the file system your testing on likely does not have the feature enabled. Tune2fs can enable this feature (according to its manage), but cannot disable it. I've no idea if enabling this feature can have side effects that cause problems in other software, so please test carefully before enabling on a filesystem with important data.

[pjotrek-b]

Thanks @ronaldoussoren, for this practical information!

Indeed, trying to see what setfattr does is a good idea.
I'll read up on EA_INODE - and see if I can format/set a (test-)partition.

[pjotrek-b]

I've now created and formatted a specific new partition for testing this.
Even with enabling ea_inode. 😭
(I've checked if ea_inode and ext_attr is set (see below)).

And it setfattr also throws the same error if I do setfattr --restore=test.xa: No space left on device
Where test.xa is the output of getfattr -d -m - > test.xa - with 1 more key/value pair.

What I find interesting is, that du -sh shows that the 0-Byte file imported with setfattr --restore is using 8.0 kB:

4.0K x <- xattrs set with python.os.setxattr()
8.0K x2 <- xattrs set with setfattr

So you mean it's not a cpython issue, but with filesystem/kernel/setup?

$ sudo tune2fs -l /dev/sde1

tune2fs 1.45.5 (07-Jan-2020)
Filesystem volume name:   XA_Test
Last mounted on:          /media/pb/XA_Test
Filesystem UUID:          581780d3-ee12-4e4f-8646-bc805709f5aa
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg ea_inode sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              262144
Block count:              1048576
Reserved block count:     52428
Free blocks:              1011633
Free inodes:              262132
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      511
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Sun Nov 24 20:25:47 2024
Last mount time:          Mon Nov 25 09:22:54 2024
Last write time:          Mon Nov 25 09:22:54 2024
Mount count:              5
Maximum mount count:      -1
Last checked:             Sun Nov 24 20:25:47 2024
Check interval:           0 (<none>)
Lifetime writes:          66 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      7ba32842-e3de-40df-8d64-f7302283f2ee
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0xa716f555

[pjotrek-b]

I can confirm: python's os.setxattr() works as expected.

I've just tested my code on an XFS-formatted partition: Works flawlessly. All key/value pairs from the IPTC metadata reference image copied as xattrs.

The issue was/is indeed with ext4 (at least on my system).

Not sure if I understand the kernel.org docs on ext4 correctly correctly, which says:

"As of Linux 3.11, it is not possible for this block to contain a pointer to a second extended attribute block (or even the remaining blocks of a cluster). In theory it is possible for each attribute’s value to be stored in a separate data block, though as of Linux 3.11 the code does not permit this."

Sounds like maybe ext4 is after all not capable of storing "more than n xattrs?"
I do wonder what EA_INODE is for then, and what "large" (in Bytes) is meant in other kernel.org ext4 docs

xattrs stay misteriously under-documented... 😉

Case closed.