Consider changing the default value of es.batch.write.refresh to false

[harelsafra]

Hi,

We have multiple cases of writing into Elasticsearch with Spark.
Some of these indexes are quite complex with hundreds of fields and a few dozen of them have auto-complete meaning that refresh operation can take 1-2 seconds to complete.
When we loaded data into this index with the default value of true for es.batch.write.refresh this caused 5m refresh operations to be queued into the refresh queue thus causing high cpu usage and data visibility issues.

Could you please consider changing the default to false? When loading multiple batches of data over many executors it makes more sense to call a single refresh in the end or let the scheduled refresh pick the data up and not call refresh after every write.

Thanks,
Harel