GCSToBQLoadRunnable doesn't respect GCS folder

[zachary-povey]

I have run into a problem when using the GCS->BQ batch mode of the BigQuerySinkConnector; each connector schedules it's own instance of the GCSToBQLoadRunnable which does not use the GCS folder when listing objects to load into BigQuery.

Because of this, if you have multiple connectors using the same bucket but different folders they all load all the objects in the bucket, irrespective of the folder they are in, and so you receive many duplicates in BQ. Further to this, only one instance will successfully delete the object and when the other instances try and fail, they will simply try again and again.

GCSToBQLoadRunnable can be seen here

[mtagle]

Good catch! IIRC, folders for GCS loading were added after the initial implementation, I think the GCS -> BQ step for that was just overlooked.

GCSToBQLoadRunnable definitely should be modified to respect the configured folder.

[zachary-povey]

Thought that might have been how it happened 🙂

I've raised a draft PR here with a suggested fix, need a bit of help getting a new integration test put in that re-creates the issue though.