Copier _exclude removes entire directory

[savtrip]

Hi guys,

I followed the docs example that shows how to perform an exclude for a copier update: https://copier.readthedocs.io/en/latest/configuring/#exclude

The example says the following:

Templating exclude patterns using _copier_operation allows to have files that are rendered once during copy, but are never updated:

_exclude:
- "{% if _copier_operation == 'update' -%}src/*_example.py{% endif %}"

The difference with skip_if_exists is that it will never be rendered during an update, no matter if it exists or not.

For my situation I have a folder called layout that I want to copy over on first run. Then on updates I want to ignore all changes to that directory. Therefore, I added the following to my copier config:

_exclude:
  - "{% if _copier_operation == 'update' %}layouts/{% endif %}"

However, when I update my theme with copier update instead of ignoring/excluding the updates as promised it completely deletes the folder.

Have I misunderstood the behaviour here? Is there another way to achieve my solution?

Is this occurring because the _exclude configuration didn't exist within my copier template config from when I copied the repo I am trying to update?

I saw on the issue tracker that someone had the same issue but for his use case he decided to use _skip_if_exists, which doesn't make sense in my use case: #2135

My copier version: 9.10.2

Any help from the community would be much appreciated, thanks guys 👍

[sisp]

I've just tried your scenario according to my understanding of it, it seems to work fine. The conditionally excluded directory is not deleted when updating. We also have a test case for conditionally excluded files (not directories though).

Could you provide a reproducible example please?

[sisp]

Perhaps we should remove all excluded files from the diff that's computed in the update algorithm.

[savtrip]

Thanks @sisp for the fast reply, sorry to confuse you, I tried my best to explain as I know it's a very subtle issue.

Right, I assumed that was happening. As a user of copier I would have expected it to ignore those changes.

For me it makes sense for copier to ignore anything in _excludes of the "new" version of the theme.

In my mind I treated it like the behaviour of .gitignore, in other words, copier please ignore the files/folders that match this pattern, it's not your concern.

In the current case, as you confirmed, Copier is performing the comparison and therefore clearing a folder I intended to ignore in an update, not delete.

It depends on how you guys intended the update to work. Would there be unexpected consequences to ignore all updates for files/folders that match the patterns in the _excludes?

[lkubb]

Since I was summoned as the author: Yes, the cause of this behavior is the introduction of new excluded paths between template versions where the paths existed in previous versions, causing the update diff dircmp to indicate the files have been deleted, so they are removed in _remove_old_files.

I completely agree that this is unexpected and that items in _exclude should be ignored when applying the diff, which is the only sensible behavior imho.

Having written the above, I noticed you @sisp already created a patch. Thanks for taking care of that, looks great!

Side note: When I looked into creating a patch for this issue, I noticed two inconsistencies (did not verify that they cause an issue) related to these lines:

copier/copier/_main.py

Lines 1294 to 1301 in 9640dc9

	extra_exclude = [
	filename.split("!! ").pop()
	for filename in ignored_files.splitlines()
	]
	for skip_pattern in chain(
	self.skip_if_exists, self.template.skip_if_exists, extra_exclude
	):
	apply_cmd = apply_cmd["--exclude", skip_pattern]

• (minor) The output of the git command actually includes other lines like M modified file, which are also added to the skip patterns - maybe the lines should be filtered for those beginning with !! first.
• (possibly more serious) Items in skip_if_exists can be templated, but the skip patterns are passed to git verbatim. This likely means rendered skip_if_exists don't work as expected.

See:

copier/copier/_main.py

Lines 705 to 712 in 9640dc9

	def match_skip(self) -> Callable[[Path], bool]:
	"""Get a callable to match paths against all skip-if-exists patterns."""
	return self._path_matcher(
	map(
	self._render_string,
	tuple(chain(self.skip_if_exists, self.template.skip_if_exists)),
	)
	)

WDYT?

[sisp]

In my mind I treated it like the behaviour of .gitignore, in other words, copier please ignore the files/folders that match this pattern, it's not your concern.

It depends on how you guys intended the update to work. Would there be unexpected consequences to ignore all updates for files/folders that match the patterns in the _excludes?

@savtrip I think your expected behavior is correct. With #2349 this should be fixed.

---

@lkubb I think "summoned" is too harsh a word. 😄 My intent was to loop you in in case you have time and interest in engaging in the discussion. Thanks for double-checking and confirming my thoughts on the problem and solution. 🙇 And thanks for bringing two additional observations to our attention! 👍 I'll look into them.

[lkubb]

I think "summoned" is too harsh a word.

Sorry, I didn't intend for it to sound harsh at all! Just a phrase since I plopped into a discussion. :) I appreciate you keeping me in the loop here, please feel free to mention me at any time.

Thanks for fixing the issue and maintaining Copier in general!

[savtrip]

Awesome thanks guys, @sisp that PR fixes the issue.

By the way, I didn't interpret "summoned" as harsh (in Australia it's whimsical in this context), I thought he was being funny, glad he confirmed it. Peace to my Copier friends, have a great day ✌️

[sisp]

Glad to hear it's working now, @savtrip! 👍

I'm not an English native speaker. 🫣 Just trying to clarify that I was only looping @lkubb in without any intention of blaming him or declaring him responsible for the bug. Anyway, I think we're all on the same page. 🙂