[enhancement] dont force upload all files in a partition

[mkeskells]

From looking at the code, it seems that when one file is considered ready to upload, then the WriterManager identifies all files in that partition and commits them

WHeras this works well for partitions written to one or two files, or work badly when writing to hundreds, as may happen with different schemas, and hive partitioning based blob names

Happy to work on a PR for this if the maintaiters consider it appropriate