extract metadata on a directory fails if an invalid file exists

[rmflight]

As of mwtab v 1.2.5, if given a directory of mwtab json files, and calling extract metadata, if there is an invalid file, mwtab will fail.

Ideally, it would warn the user there was an error, and keep going.

Happened when I did a download all, and then tried to get metadata entries from all of the downloaded files.

[ptth222]

What do you mean by invalid file? It doesn't look like the download all command does any validation. I'm not trying to be pedantic, but when the package does contain an actual "validation" module I think it's import to be precise. Are you saying that when you tried to download a set of studies, after encountering an error when trying to download one of the studies it stopped downloading the rest of them? There is not a try block around the download function in the loop, so that is consistent with what I just described. I just want to make sure that's the issue you experienced.

[rmflight]

The CLI docs state that we can do:

• download all files
• extract metadata from that directory of files

That was my goal. I wanted to use the mwtab CLI to download everything, and then extract the metadata from them all so I didn't need to fully parse all the JSON files to get the study information.

Yes, I know there is a validation function. And if I was using the API and writing a python script, I might do a validation before pulling the metadata.

But I was using the CLI, and the docs say I can do this. Download everything, and then extract all the metadata or metabolites for the full directory. If there is an invalid file encountered during the extraction of the metadata, I expect a warning, and the function to keep going.

I don't expect that I have to validate my directory of files before extracting the metadata. I'm not sure how I would do that on a full directory, as it looks like the validation CLI takes just a single file.

Does that make more sense?

[ptth222]

The CLI docs state that we can do:

download all files
extract metadata from that directory of files
That was my goal. I wanted to use the mwtab CLI to download everything, and then extract the metadata from them all so I didn't need to fully parse all the JSON files to get the study information.

The original post just referred to downloading not extracting metadata, so I'm glad I asked. I can see that pretty much all of the CLI commands would have this problem of completely failing if any one of the files encountered an issue. I will make this change to all of the relevant commands and not just download.

Yes, I know there is a validation function. And if I was using the API and writing a python script, I might do a validation before pulling the metadata.

The CLI also has a validate command. It's not just the API. It's also an option for the download command, "--validate". It looks like this option is actually not implemented correctly for the command however.

But I was using the CLI, and the docs say I can do this. Download everything, and then extract all the metadata or metabolites for the full directory. If there is an invalid file encountered during the extraction of the metadata, I expect a warning, and the function to keep going.

The whole reason I asked for more information was the use of the term "invalid". In common parlance that term means something more general than what could be meant when talking in context about a package that "validates" things. In the context of the mwtab package "invalid" could mean that the validation function (CLI, API, whatever) reported at least one error. Using the more general meaning of "invalid" to refer to a file that causes an error during the execution of other functions can be ambiguous. A file could fail to open or be read properly due to an encoding issue for instance, but this doesn't necessarily mean that the contents would not validate if the correct encoding was used to read it. I was trying to clarify how you were using the term. I am pretty sure you are using the general meaning though.

I don't expect that I have to validate my directory of files before extracting the metadata. I'm not sure how I would do that on a full directory, as it looks like the validation CLI takes just a single file.

The CLI validate works on a directory, but the docstring does give the impression that it's just a single file. I will fix this also.

Does that make more sense?

Yes. This has once again led me to more issues than the one that started the thread, but I will deal with them all.

For me:

1. Wrap all multi-file commands in a try block to report issues with a file and keep going.
2. Change the CLI and documentation to be clear about which commands operate on single files or directories.
3. Investigate the "--validate" option as it appears to be used incorrectly by the "validate" command and is not used by the "download" command despite being in the options list for it.
4. Update docstrings for changed functions.
5. Edit documentation to explain some undocumented things such as "download study path_to_dir" reading from a JSON file and assuming what is in there are analysis IDs.