Is it possible to read from a json dir and insert to Icerbeg with Spark

[christophediprima]

Hi there, I am trying to implement a pipeline that reads from s3 and insert into an Iceberg table. I have been trying with Spark but could not find an example on how to wrap all this up... I have a running batch version that looks like this:

import ibis
from   pyspark.sql import SparkSession

session = SparkSession.builder.getOrCreate()

con = ibis.pyspark.connect(
  session,
)
json_path = 's3a://datalakehouse/bronze'
raw_json = con.read_json_dir(json_path)

raw_users = raw_json.filter(raw_json.type == "user")

new_users = raw_users.select(name=raw_users.name)

con.insert(
  "users",
  new_users,
  database="my_catalog.users",
  overwrite=True,
)

When I try to use the steaming mode I have following error:
NotImplementedError: PySpark in streaming mode does not support to_pyarrow