Bug in onestop-client avro schemas

[erinreeves]

Bug Description
A few of the details with the python-client avro schemas seem questionable. Need clarity on what's correct and fix either unit tests or schemas.

• Change SqsHandlers/S3MessageAdapter's transform to take a single record, since only operates on 1. (snuck this in since was hurting to look at).
• Fix relationships['type'], which should it be an enum which allows for:
  {"relationships":[{"id":"5b58de08-afef-49fb-99a1-9c5d5c003bde","type":"COLLECTION"}]}
  (from onestep-test-data GOES/granules/*.json files)
• Fix fileLocationType to an enum
• Fix fileLocations in ParsedRecord to be a Map/Dict[str, FileLocation}
  "fileLocations":{
  "s3://noaa-goes16/ABI-L1b-RadF/2019/303/09/OR_ABI-L1b-RadF-M6C10_G16_s20193030950389_e20193031000109_c20193031000158.nc":{
  "serviceType":"Amazon:AWS:S3",
  "deleted":False,
  "restricted":False,
  "asynchronous":False,
  "locality":"us-east-1",
  "lastModified":1572430074000,
  "type":"ACCESS",
  "optionalAttributes":{
  },
  "uri":"s3://noaa-goes16/ABI-L1b-RadF/2019/303/09/OR_ABI-L1b-RadF-M6C10_G16_s20193030950389_e20193031000109_c20193031000158.nc"
  }
  }

Severity
Severity: MEDIUM (High/Medium/Low)
Priority: MEDIUM (High/Medium/Low)

To Reproduce
Steps to reproduce the behavior:

1. Run test_ParsedRecord.py unit test on this branch: 1508-avro-schemas

Expected Result
All tests should pass

Actual Result
ERROR: test_fileLocations (test.unit.schemas.psiSchemaClasses.org.cedar.schemas.avro.psi.test_ParsedRecord.test_ParsedRecord)
TypeError: Superfluous parameters in call: {'s3://noaa-goes16/ABI-L1b-RadF/2019/303/09/OR_ABI-L1b-RadF-M6C10_G16_s20193030950389_e20193031000109_c20193031000158.nc'}

ERROR: test_relationships (test.unit.schemas.psiSchemaClasses.org.cedar.schemas.avro.psi.test_ParsedRecord.test_ParsedRecord)
TypeError: Key type has incorrect type: str instead of Optional[RelationshipType].

FAIL: test_discovery (test.unit.schemas.psiSchemaClasses.org.cedar.schemas.avro.psi.test_ParsedRecord.test_ParsedRecord)
assert len(args) == 2
AssertionError

Additional context
Found bug by running sme/sme.py script with data loaded in local OneStop from OneStop-test-data repo (GOES triggered the errors mostly)

[erinreeves]

Relationships tested with scripts/launch_pyconsumer.py using copy of /etc/config/config.yml with log_level=Debug:

Old payload web publisher, after transform
"relationships": [{"type": {"type": "COLLECTION"}, "id": "fdb56230-87f4-49f2-ab83-104cfd073177"}], "errors": [{"title": null, "detail": null, "status": null, "code": null, "source": null}]}

New:
"relationships": [{"type": "COLLECTION", "id": "fdb56230-87f4-49f2-ab83-104cfd073177"}]

relationships=[Relationship(type='COLLECTION', id='fdb56230-87f4-49f2-ab83-104cfd073177')]

Note, manually did this but added to next fixing story:
needed try/except around get_uuid_metadata with a return in except since some records were causing "(404) when calling the HeadObject operation: Not Found" 'cuz I think they didn't exist, were deleted records.

[erinreeves]

Also changed SqsHandlers and S3MessageAdapter to operate/transform on a single record, and only take 1 in as a parameter. Since it already only operated on 1 record.

[erinreeves]

1508-avro-schemas branch

[erinreeves]

cedardevs/onestop-clients#60