is it possible to add a subfolder using dvc add

[nvinayvarma189]

I have a folder called data that is being tracked by dvc. There is also a corresponding data.dvc file in my repo.
below is the structure

data
`-- 0.0.4

when the data folder is in the above state, it was properly added and pushed to AWS s3. data.dvc is also properly updated.

I now have added two folders to it and this is the structure now

data
|-- 0.0.4
|-- 0.0.5
`-- 0.0.6

Problem:
Now I want to do dvc add separately for 0.0.5 and 0.0.6 instead of doing dvc add data. when I do dvc add data/0.0.5 I get this error

ERROR: bad DVC file name 'data/0.0.5.dvc' is git-ignored

I guess this is because there is only data.dvc folder

Reason
The reason I want to add the subfolders separately instead of adding the whole data folder (with dvc add data) is that if I do dvc add data, the data.dvc will get updated with the hash corresponding to both the 0.0.5 and 0.0.6 folders. Then when I do dvc pull, I will get both the 0.0.5 and 0.0.6 folders downloaded.

But I just want to associate each data version folder with a unique data.dvc file. Currently, deleting all folders except one and then doing dvc add data seems to be the only way.

[karajan1001]

If you don't want to make 0.0.4, 0.0.5, and 0.0.6 combined with each other, maybe you need to dvc add data/0.0.4 them separately. But I have a question here is 0.0.4, 0.0.5, and 0.0.6 representing different versions of data? If so, then will they be used at the same time? If They are different versions and will never appear at one time.
Maybe moving contents from data/0.04 to data and using dvc add data to track them is a better choice. Then you can commit the data.dvc to Git and replace the contents with other versions with a dvc commit following. By doing this, you can make a quick switch between different versions with git checkout and dvc checkout commands.

[nvinayvarma189]

Thanks a lot for your reply @karajan1001

If you don't want to make 0.0.4, 0.0.5, and 0.0.6 combined with each other, maybe you need to dvc add data/0.0.4 them separately.

Yup....but dvc add data/<subfolder> is throwing the error: ERROR: bad DVC file name 'data/0.0.5.dvc' is git-ignored

But I have a question here is 0.0.4, 0.0.5, and 0.0.6 representing different versions of data? If so, then will they be used at the same time? If They are different versions and will never appear at one time.

Yes, They are different versions of the same data points. But I need these folders together on my local machine because I also have a folder called metrics under each data version folder. So, after I train the model with each version of the data (0.0.4, 0.0.5, 0.0.6), I save the resulting metrics under (0.0.4/metrics, 0.0.5/metrics, 0.0.6/metrics). For comparison purposes, I'm keeping multiple data version folders at a time.

Maybe moving contents from data/0.04 to data and using dvc add data to track them is a better choice

This is a possible solution.

1. I have to move out 0.0.5, and 0.0.6 from the data folder and then do dvc add data/. This data.dvc will only have the hash of 0.0.4
2. Then, I have to move out 0.0.4, and 0.0.6 from the data folder and then do dvc add data/. This data.dvc will only have the hash of 0.0.5
3. Then, I have to move out 0.0.4, and 0.0.5 from the data folder and then do dvc add data/. This data.dvc will only have the hash of 0.0.6

Which is what I'm looking for. But I was thinking if there is an easy alternative which does the same thing.

I haven't really tried out dvc commit and dvc checkout. Will experiment with that.

Thanks a lot once again!!

[karajan1001]

Yup....but dvc add data/<subfolder> is throwing the error: ERROR: bad DVC file name 'data/0.0.5.dvc' is git-ignored

This is because you had already tracked the whole data folder. In this case, you can not track the subdirs in it again. If you want to track each folder independently, first use dvc remove data to remove the versioning of whole directory and then dvc add dta/0.0.5/0.006/0.007.

Yes, They are different versions of the same data points. But I need these folders together on my local machine because I also have a folder called metrics under each data version folder. So, after I train the model with each version of the data (0.0.4, 0.0.5, 0.0.6), I save the resulting metrics under (0.0.4/metrics, 0.0.5/metrics, 0.0.6/metrics). For comparison purposes, I'm keeping multiple data version folders at a time.

DVC can compare different version of metrics, details are in the official doc