docs: clarify `check_duplicate_files` option in the `add_files` api docs

[thijsheijden]

Question

Hi! I am trying to add 1 million existing Parquet files to an Iceberg table using the add_files procedure. I am inserting in 1000 batches of 1000 files. Every batch takes longer than the previous batch, and at this point each batch is taking around 1-2 minutes. This is much much slower than Spark, which remains consistent throughout the entire insertion. How can I speed this up? The Parquet files already have metadata so perhaps that could be exploited somehow? Below is the code I am using:

warehouse_path = "/warehouse"
catalog = load_catalog(
    "pyiceberg",
    **{
        'type': 'sql',
        "uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db",
        "warehouse": f"file://{warehouse_path}",
    },
)
catalog.create_namespace_if_not_exists("default")

# Load the batches of files to import
batches = os.listdir(args.file_dir)
first_file = os.path.join(args.file_dir, "batch_0", os.listdir(os.path.join(args.file_dir, "batch_0"))[0])

# Create table using schema of the first file
df = pq.read_table(first_file)
table = catalog.create_table_if_not_exists(
    f"default.{args.table}",
    schema=df.schema,
)

batch_idx = 1
for batch_dir in batches:
    print(f"Adding batch {batch_idx}")
    batch_dir =  os.path.join(args.file_dir, batch_dir)
    file_paths = os.listdir(batch_dir)
    file_paths = [batch_dir + '/' + s for s in file_paths]
    table.add_files(file_paths=file_paths)
    batch_idx += 1

[jayceslesar]

It would be awesome if you could run https://github.com/benfred/py-spy and attach the produced flame graph! I want to bet that it is the self._table.inspect.files() located here but py-spy will confirm. Also including the version of pyiceberg you are using will be helpful

[thijsheijden]

It is most definitely that, I did not realise there was an option to turn off duplicate file checking 🤦🏼. Running it now every batch takes the same time, and it is 100x faster! I can still run py-spy if you would like, but I think we found the culprit haha

[kevinjqliu]

I see the option is not mentioned in the add_files docs https://py.iceberg.apache.org/api/#add-files
Lets add it there so it becomes transparent

[Xiezhibin]

1. I'll check on this, could you assign this task to me?
2. Do we need to annotate the snapshot_properties parameter as Optional in iceberg-python/pyiceberg/table/__init__.py?
3. How should I modify the documentation? Should I add an example that includes snapshot_properties and check_duplicate_files, or should I modify the existing example to demonstrate the optional parameters?

[Fokko]

Thanks, @Xiezhibin, for being willing to work on this!

Do we need to annotate the snapshot_properties parameter as Optional in iceberg-python/pyiceberg/table/__init__.py?

No, I don't think this is needed, the EMPTY_DICT avoids doing null-checks :)

4. How should I modify the documentation? Should I add an example that includes snapshot_properties and check_duplicate_files, or should I modify the existing example to demonstrate the optional parameters?

I think we can add a table here:

iceberg-python/mkdocs/docs/api.md

Line 1064 in 5e975d5

tbl.add_files(file_paths=file_paths)

Similar to how it is done for Java: https://iceberg.apache.org/docs/nightly/spark-procedures/#usage_1

[Xiezhibin]

No, I don't think this is needed, the EMPTY_DICT avoids doing null-checks :)

👍 Thanks! —the EMPTY_DICT saves us from writing extra null checks 🙌,

I plan to modify the following parts according to the reference content to see if it is feasible.

• Position Notes and Warnings Prior to Examples: so users get the important caveats upfront.

• Add a table that outlines the input and output and whether they are required.

• Provide Two Examples: one simple before, one showing the full parameters.

The second example about Add files to Iceberg table with custom snapshot properties:

# Assume an existing Iceberg table object `tbl`

file_paths = [
    "s3a://warehouse/default/existing-1.parquet",
    "s3a://warehouse/default/existing-2.parquet",
]

# Custom snapshot properties
snapshot_properties = {"abc": "def"}

# Enable duplicate file checking
check_duplicate_files = True

# Add the Parquet files to the Iceberg table without rewriting
tbl.add_files(
    file_paths=file_paths,
    snapshot_properties=snapshot_properties,
    check_duplicate_files=check_duplicate_files
)

# NameMapping must have been set to enable reads
assert tbl.name_mapping() is not None

# Verify that the snapshot property was set correctly
assert tbl.metadata.snapshots[-1].summary["abc"] == "def"

[Xiezhibin]

Hi everyone! If you have the opportunity 😄, please take a look at the document and let me know if there are any areas that need modification. If you have any suggestions, I’d love to hear them. If everything looks good, I’ll go ahead and submit the merge request. Thank you!

[kevinjqliu]

this is the add_files function signature

iceberg-python/pyiceberg/table/__init__.py

Line 861 in 86bf71c

self, file_paths: List[str], snapshot_properties: Dict[str, str] = EMPTY_DICT, check_duplicate_files: bool = True

we dont need to worry about snapshot_properties
lets add documentation to mention that check_duplicate_files is by default True and this will check the new files against the existing iceberg table data files, which can be expensive.

check_duplicate_files can be turned off. but add a caution that this can lead to data consistency / table corruption issues if the same data file is added multiple times

https://py.iceberg.apache.org/api/#files