insert into parquet file result a lot files?

[yuyang-ok]

we project use deltalake as a TableProvider and insert into deltalake result a lot of little files.

my code kind of like this.

        let table_path = self.table_path.clone();
        let data = self.data.clone();
        let total_rows: usize = data.iter().map(|r| r.num_rows()).sum();
        let stream = futures::stream::once(async move {
            let delta_table = open_table(&table_path).await?;
            let _ = DeltaOps(delta_table).write(data).await?;
            info!("Inserted into {} {} rows", table_path, total_rows);
            Ok(RecordBatch::new_empty(Arc::new(Schema::empty())))
        })
        .boxed();

how to append exists file instead create new files everytime.

[ion-elgreco]

You can't append to an existing parquet file, what you can do is run optimize often to combine all these small files.