archive/tar: add support for arbitrary pax vendor extensions

[jstarks]

Feature request: archive/tar should support reading and writing arbitrary pax vendor extensions. This could be done either with additions to tar.Header, or with new functions on tar.Reader and tar.Writer. This would allow go code to work with metadata that archive/tar does not yet support.

Here's the motivation: Docker (written in go) uses tar as the archive format for container images in Linux. We'd like to make sure this works for Windows, too, but doing so requires extending the tar format to include some non-POSIX Windows metadata (security descriptor, creation time, etc.).

The natural way to extend the tar format is to encode this additional metadata in pax vendor extensions, e.g. "MSWINDOWS.sd" for the security descriptor. Currently, however, archive/tar does not expose pax vendor extensions other than those that start with SCHILY.xattr (used for POSIX extended attributes), which are exposed in Header.Xattrs. It's probably not reasonable to ask archive/tar to support these specific "MSWINDOWS" extensions that are neither formally defined nor used anywhere today, but archive/tar could expose the ability to read and write arbitrary vendor extensions.

This seems like it would be useful outside of this use case as well, e.g. to support POSIX ACLs (SCHILY.acl.*).

@dsnet

[dsnet]

I'm not the owner of archive/tar, but I personally think that this is a fairly reasonable feature request. I think it's unfortunate that tar.Header.Xattrs is specific to the SCHILY.xattrs.* headers only, which is too narrow in functionality.

If the package allows for arbitrary vendor extensions, there are some edge cases to think about. For example #13548 requests sparse file support, which can be accomplished using the GNU.sparse.* extended headers. Since these extended headers have semantic meaning in regards to how to read the file, what happens when the user overwrites GNU.sparse.realsize or something? More than likely, the user will end up creating a corrupt file.

This will take a little thought, but it's certainly solvable. If I have time, I can try and tackle this in the go1.7 cycle.

[jstarks]

Yeah, I thought about that, too, but I don't have a good solution. The same might be said e.g. for SCHILY.acl.*, if someone adds support for POSIX ACLs to Header directly. If the caller to WriteHeader() fills out both the specific Header field and an entry in the PAX extension map, which takes precedence?

Thanks for considering this change. For now, I'm going to have to fork archive/tar to add the necessary functionality, but I'd like to converge back on the official version if/when this feature makes it in.

[rsc]

@dsnet:

I'm not the owner of archive/tar

I think you are now. :-)

Anyway, it sounds like you are in favor and so a CL just needs to be written. This isn't going to block the release, but if you want to write a CL or someone else wants to write it and send it to you for review, that sounds fine.

[dsnet]

Pushing to Go 1.9.

I'm going to close off all Reader related issues in this cycle and let Go 1.9 be about Writer.

[wking]

Another way to deal with this (for readers anyway) is to add:

func (tr *Reader) NextRaw() (*Header, error)

(or some similar name) that gets you the next entry without invoking the current automatic pax-extention-header consumption and similar. Users who want to handle custom extended headers could call NextRaw() (or whatever) to advance one entry at a time, regardless of the entry type. This would also be useful for low-level code that wanted to validate the tar files (e.g. if wanted to make sure there were no pax extended headers because some downstream library didn't understand pax).

[dsnet]

@wking

NextRaw ... gets you the next entry without invoking the current automatic pax-extention-header consumption and similar. Users who want to handle custom extended headers could call NextRaw() (or whatever) to advance one entry at a time, regardless of the entry type.

If I understand your suggestion correctly, you are proposing that we try to read the archive as straight USTAR format (since PAX is really an extension built ontop of USTAR)? What happens when the underlying format is the old GNU format, which does not conform to USTAR? It also seems pretty painful, that'd you would have to parse all of the PAX extended headers yourself.

This would also be useful for low-level code that wanted to validate the tar files (e.g. if wanted to make sure there were no pax extended headers because some downstream library didn't understand pax).

Is this a real use-case that you have? Validating that a tar file is of a given format is actually pretty hard. Do you validate it according to strict conformance to the POSIX specification or GNU manual? Do you validate it according to what the GNU implementation actually does? Do you validate it according to what the world seems to believe those formats are? If there is a problem with implementations properly conforming to HTTP specifications, then the conformance of tar implementations to "specifications" is even worse.

I'm not sure the standard library should provide this feature. On the other hand, I'm willing to explore providing an option where the Writer guarantees to output only specific formats; please file another issue if you care for it.