Clean up unit tests for onestop-python-client

[kenhtanaka]

Summary

As a	Developer
I want to be able to	Change all constructor signatures to not take a config file path but parameters and improve unit tests for the onestop-python-client module
So that I can	have better test coverage without errors and not pass a config file into every constructor.

• Change classes so none have config files passed in, only parameters needed.
• Updated effected documentation
• Get unit tests working
• Change unittests to not load config but have their own dict they pass in, no need for loading from a config file for unittests.
• Move unit tests to onestop-python-client/tests/unit and put test_WebPublisher in onestop-python-client/tests/integration
• Update Circle CI to run all tests, had to rename/move unit tests to test/unit/test*.py so python unittest discovery can find them. Moved integration test to test/integration
• Set build environment variables for integration tests to use.
  - [ ] Update scripts to at least accurate, unsure if all can work. Moved to new story
• Refactored S3Utils.connect so clearer, adds one more param but is clearer.

[erinreeves]

Questions:

• Lots of merged but not deleted branches, so unsure if should look at any for if these fixes have been done.
• Future story: Should we refactor out non-extractor code from Extractor class? Such as the connecting to S3?
• Refactor Chris did puts the parsing of config files on whatever calls S3MessageAdapter, S3Utils, SqsConsumer. Reflected in current script s3_notification_handler. Verification ok?
• Someone give me an example of executing python3 scripts/sqs-to-registry/s3_notification_handler.py?

[erinreeves]

Also need find story about updating circleCI config to remove CLI build.

[erinreeves]

Only have log_level in one config?

[erinreeves]

Check if relevant for story 1233:
Follow upstory for S3Utils:

1. Rename method def get_csv_s3(self, boto_client, bucket, key):
   https://github.com/cedardevs/onestop-clients/blob/master/onestop-python-client/onestop/util/S3Utils.py#L99
   so directly below we have read_bytes_s3 ...
   so maybe we have get_s3_bytes
   get_s3_file
   read_bytes_s3 -> get_s3_bytes
   get_csv_s3 -> get_s3_file

2. For CsbExtractor, shouldn't be doing any connecting
   https://github.com/cedardevs/onestop-clients/blob/master/onestop-python-client/onestop/extract/CsbExtractor.py#L42

[erinreeves]

S3MessageAdapter:

• look into transform method used to return im_message.file_format, headers, - gone now. why?

[erinreeves]

Classes todo modification:

• S3Utils

• CsbExtractor

• WebPublisher

• KafkaConsumer

• KafkaPublisher

• SqsConsumer

• Change unittests to not load config but have their own dict they pass in, no need for loading from a config file for unittests.

[erinreeves]

What is the incoming collection_uuid in KafkaPublisher class for publish_collection and publish_granule?

[erinreeves]

Build passing!!

[erinreeves]

Which scripts can be run locally versus have to be copied to a pod to be executed?

archive_client_integration.py
bucket_automation.py
launch_delete_handler.py
launch_e2e.py
launch_kafka_publisher.py
launch_pyconsumer.py

sme/
sme.py
smeFunc.py
spatial.py

sqs-to-registry/
s3_notification_handler.py

[erinreeves]

cedardevs/onestop-clients#57