Skip to content

add an optional final report to dict_sweep helper function #303

@newgene

Description

@newgene

utils.dataload.dict_sweep is used to:

Remove keys whose values are ".", "-", "", "NA", "none", " "; and remove empty dictionaries

It's quite powerful, but could be potentially dangerous to remove some fields we want to keep. So, it would be nice to have a final report at the end of its run, something like this:

<na_value_1>: 
     - <removed_field_name_1>: cnt_of_removed_values,
        ...
<na_value_2>:
    ...

can add a new report parameter to this function, default can be True.

Also worth checking if setting to True will significantly increase the processing time in a typical data plugin, if that is the case, we might set default to False and use it only if needed for debugging purpose.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions