How to sort out extension ehowing huge amount of "pending changes"

[dibus2]

When starting the VS code extension it shows 110K Pending Files.
Application is extremely laggy to the point where it makes using the EC2 machine impossible.
The DVC section in the GIT tab shows nothing (as I suspect there are too many files).
The way the repo is setup is as such:

• project_1 / pipeline_1 / dvc.yaml
• project_1 / pipeline_2/ dvc.yaml
• project_1 / pipeline_3/ dvc.yaml
• ...
• project_1 / pipeline_13/ dvc.yaml
• project_2/ ...

I also tried to specialized the dvc extension to a single pipeline but it doens't seem to be helping. This is what I have in my settings.json
"dvc.focusedProjects": [
"project_1/pipeline_1"]
Note that pipeline_1 folder contains the dvc.yaml file.

Any help on sorting this out is appreciated.

Thanks.

[mattseddon]

Have you tried running dvc pull before opening the extension? Is there a folder that should be gitignored that isn't?

[dibus2]

Hi Matt,

thanks for looking at this. I had not done a pull just before starting it to be fair so I did, got the number down to 108K. My git status shows nothing so this is all the status added by DVC.

You are bringing a good point though, that some of these pipilines are maintained only by some teams and ideally other teams don't have to pull the data from all other pipelines if they don't need it.

Cheers,

[dibus2]

Here is the breakdown of the git DVC section actually. It wasn't starting earlier but this time it managed to start.

I honestly don't understand why there are all these files in the Uncommitted section. When I look at these files they are actually files for which there is a .dvc file in the repo but that I have not pulled because they are not part of any pipelines and are as such not needed in the current commit of the repo.

[dibus2]

Also, shouldn't these all be ignored since I specified a specific folder through dvc.focusedProjects that does not include any of these .dvc file in it?

[mattseddon]

I would guess that there are duplicates between uncommitted and not in cache as there is an unknown uncommitted status that things fall into when they are not in the cache (I forget the history on this), you can search the DVC issues if you want to know more.

For focused projects, these relate to - "A subset of paths to the workspace's available DVC projects. Using this option will override project auto-discovery." I am guessing that you've set this to a pipeline instead of a project, so it failed validation and bypassed the option. Try using the "Select Project(s) to Focus (set dvc.focusedProjects)" quick pick to select a project to focus. This should give a list of valid options.

[dmpetrov]

@dibus2 thank you for reporting this!

[shcheklein]

@dibus2 thanks for creating the issue! a few questions:

• where do you have .dvc directories for those projects? (just to better understand how things are isolated from each other)
• 50K+ files - is it a single directory? what is the part in the tree to it? are those parquet files? is it input for the pipeline?

[dibus2]

@shcheklein

• so each one of the pipeline_x folders contain a .dvc.yaml file.
• the files are spread across different projects/ in which we have a /data folder in each one of them.

For instance in project_1/data/

• data_set_1.dvc
• data_set_2.dvc
• data_set_3.dvc
  ...
• data_set_10.dvc

Now most of these are usually not being pulled because they are not used anymore or not yet and maybe that's a problem to keep the .dvc files around?
From a content perspective, these are mostly JSON files actually, there are few parquet but they are mostly json files.

[dibus2]

I would guess that there are duplicates between uncommitted and not in cache as there is an unknown uncommitted status that things fall into when they are not in the cache (I forget the history on this), you can search the DVC issues if you want to know more.

For focused projects, these relate to - "A subset of paths to the workspace's available DVC projects. Using this option will override project auto-discovery." I am guessing that you've set this to a pipeline instead of a project, so it failed validation and bypassed the option. Try using the "Select Project(s) to Focus (set dvc.focusedProjects)" quick pick to select a project to focus. This should give a list of valid options.

I actually do not find the quick pick to select the project option @mattseddon

[shcheklein]

@dibus2 sorry, just to clarify. I specifically mean .dvc directories (not files). Do you have a single one in the root or multiple (I'm referring to the --subdir option).

Now most of these are usually not being pulled because they are not used anymore or not yet and maybe that's a problem to keep the .dvc files around?

yes, if you don't need them - better to drop them I guess (you can always recover them from the Git history - that's the beauty of DVC).

[dibus2]

ah sorry, yes I have only one in the top folder @shcheklein