string provided must be lowercase

[samtazzyman]

I've got json files taken from the github API as my data. I want to define a scheme to test the raw data against on its arrival. When I try to follow the steps in https://github.com/moj-analytical-services/etl_pipeline_example it doesn't really tell me how to create the schema in the first place.

When I try to use DatabaseMeta and TableMeta to define the schema I get

tab.add_column(
    name='authoredByCommitter',
    type='boolean',
    description='whether the author and committer are the same'
)

I get
ValueError: string provided must be lowercase

The issue being that by data comes out of Github's API with the field names in camel case. Obviously I could work through and rename them, but ought I to have to do this? If the data was in csv I could probably drop all of the headers and redefine them (?) but for a json with its nested structure this is potentially even more of a pain in the arse.

[isichei]

Yeah this was a self imposed rule but finding the same issue with dashes in json names. It is likely that we will start pulling more jsons that also have names that are web specific and more likely to have - than _ which will cause the same issue. I think we should loosen the rules.

[RobinL]

+1 just encountered it with an s3 path that had capitals in it