Sync and Recursive results

[turetske]

What follows is a table showing what we want as the appropriate behavior for various types of uses of pelican sync and pelican get/put --recursive @bbockelm @brianaydemir Please double check to make sure I got every case correct.

Command	Left Argument Type	Right Argument Type	Result	Description
pelican obj sync pelican://foo/bar /tmp/test	namespace prefix/collection	local file (exists)	Error	A sync from prefix/collection to a file should fail with an error if the local file exists
pelican obj sync pelican://foo/bar /tmp/test	namespace prefix/collection	local directory (exists)	/tmp/test/baz.txt	Puts the objects with the foo/bar prefix into the tmp/test folder
pelican obj sync pelican://foo/bar /tmp/test	namespace prefix/collection	local directory (does not exist)	/tmp/test/baz.txt	Creates the /tmp/test folder and puts the objects with the foo/bar prefix into the tmp/test folder
pelican obj sync pelican::/foo/bar.txt /tmp/test	object	local directory (exists)	tmp/test/bar.txt	Puts the object in the tmp/test directory
pelican obj sync pelican::/foo/bar.txt /tmp/test	object	local path (does not exist)	tmp/test/	Writes foo/bar.txt into tmp/test and creates test
pelican obj sync pelican::/foo/bar.txt /tmp/test	object	local path (does exist)	/tmp/test	Overwrites foo/bar.txt as /tmp/test
pelican obj sync /tmp/test pelican://foo/bar	local directory	a namespace prefix	pelican://foo/bar/baz.txt	Puts the files in /temp/test/ at /foo/bar, creating namespace prefixes as needed
pelican obj sync /tmp/test.txt pelican://foo/bar.txt	local file	a directory on a posix backend	Error	On a posix backend, throw an error and do not sync a file with a directory
pelican obj sync /tmp/test.txt pelican://foo/bar.txt	local file	object	pelican::/foo/bar.txt	On an S3 backend, or a posix backend where bar.txt is an object, overwrite the file
pelican obj sync /tmp/test.txt pelican://foo/bar.txt	local file	Does not exist	pelican::foo/bar.txt	If the target location doesn't exist, write the file to that location and create whatever prefixes as necessary

When there are multiple sources specified on the command-line, we will iterate over each source and treat each one according to the table above.

[brianaydemir]

There is one wrinkle that we did not discuss: What should happen when there are multiple sources specified on the command-line.

Anyway…

For the case:

• Left argument: (Pelican) object
• Right argument: local path (does not exist)

I believe the result should be /tmp/test, same as "local path (does exist)".

I do not see anything else in the table that needs changing.

===

Then, the summary of this all almost becomes:

• If the left argument is a Pelican namespace prefix/collection, then the right argument must be an existing directory or else a path that could be created as a directory.

  Conversely, if the left argument is a directory, then the right argument must be a Pelican namespace prefix/collection.

• If the left argument is a Pelican object, then the right argument must be an existing file or else a path that could be created as a file.

  Conversely, if the left argument is a file, then the right argument must be a Pelican object.

I say almost because we have a concession to the common practice of "copy a singular file/object to this already existing local directory (that tab-completion filled in for me)": If the left argument is a Pelican object, and the right argument is an already existing directory, we treat the right argument as if the user had appended the object's basename to it. The converse of this case, where the left argument is a local file, and the right argument maps to a POSIX directory behind the Pelican origin, is, in the table, flagged as an error.

The lack of symmetry is perhaps unappealing, but in the former case, the user knows that the destination is a POSIX filesystem and thus knows how it behaves. It is a whole separate can of worms to specify and enforce a POSIX-compatible scheme on object names.

[turetske]

@brianaydemir @bbockelm Updated the table and added a line on multiple sources.

[turetske]

@brianaydemir @bbockelm

Ran into one case I don't think we can easily solve:

Command	Left Argument Type	Right Argument Type	Result	Description
pelican obj sync pelican::/foo/bar.txt /tmp/test.txt	object	local path (does exist)	/tmp/test.txt	Overwrites foo/bar.txt as /tmp/test.txt

Our current code sees that /tmp/test.txt exists and, in the case that test.txt is smaller than bar.txt, believes it's been partially synced and appends the rest of the bytes onto that file, rather than overwriting it.

If the test.txt is larger than bar.txt, it fails with the following error:
local copy of file is larger than remote copy bad content length

This actually is in line with what we expect sync to do without implementing an algorithm to check if two files are the same. I think it's not an easy problem to fix without changing how sync works.

[turetske]

@bbockelm Your opinion?

[bbockelm]

Two thoughts:

1. This doesn't have to change one way or the other in the next release (that is, object sync (and object get --recursive) are not idempotent #1616 and Recursive and sync transfers fail for a single file #1588 can be done independently of this).
2. I think the current behavior makes more sense for internal retries (partial download from cache A, followed by a failure, followed by a completion from cache B). I feel OK in assuming the partial file is the same in both cache A and B.
3. When running as a standalone tool, I'm OK with deleting the local file prior to invoking the download code.

[turetske]

@bbockelm So, for the current release, leave the behavior as is (the smushing together)?

Also, for S3, what should we do when we are syncing from an S3 origin and the objects have the naming scheme of <bucket>/foo, <bucket>/foo/bar and <bucket>/foo/baz? Because if foo is an object then we can't create a directory with the same name to place bar and baz into.

A possible suggestion is to sync foo/baz and foo/bar but give a warning for foo that it's unable to be synced. Should this be an issue for a later release and just say sync behavior is undefined for S3 for now? Or some other stop gap?

[bbockelm]

For the current release - I'm indifferent to whether we change it. I think I saw you mention somewhere you were already tackling it. I'll leave it up to you whether it makes more sense to finish or set aside.

For the S3 case -- I'm not sure that's possible since, on the server side, we already map to "directories" and should have generated an error there. That said, I'm OK with erroring out when we get to it.

[turetske]

For the S3 case, I was more referring to when there isn't a directory at all on the local side, so it would try to create foo as both a directory and also an object and probably error at that point. I'll just let it error out, then. I'm trying to get it out the door as quickly as possible, at this point.