Skip to content

Add the ability to DQ on s3 buckets #35

@danielbeach

Description

@danielbeach

We should add the ability to do simple DQ checks on s3 buckets. Say something like ...

results = tm.check_s3_uri(bucket='somebucket', prefix='some_prefix', check_type='files_exisit')
True

Could probably implement with boto3
Or something along those lines ...

  • check files exist at some location
  • count the number of files at some location
  • check for empty files
  • check for missing files over a date range
  • check file names meet some regex
  • etc.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions