Skip to content

Conversation

@benamartin43
Copy link
Contributor

Added code to create parent directories when partitioning is used. I found that the code to automatically do this seems to have been removed when the iceberg partitioning was added (b0c83ea). I have based my code on the previous code. Please let me know if I need to change the PR in any way.

I was able to reproduce #976 and verify the fix using the following query with the nexmark data source.

CREATE TABLE sink (
  window_start TIMESTAMP,
  total DOUBLE,
  bid_count BIGINT,
  name TEXT
) WITH (
  connector = 'filesystem',
  type = 'sink',
  path = 'file:////tmp/',
  format = 'parquet',
  'partitioning.time_pattern' = 'year=%Y/month=%m/day=%d/hour=%H/minute=%M'
);

INSERT INTO sink
SELECT 
    window.start as window_start,
    total,
    bid_count,
    name
FROM (
    SELECT 
        TUMBLE(INTERVAL '1 minute') as window,
        person.name as name,
        SUM(CAST(bid.price AS DOUBLE)) as total,
        COUNT(*) as bid_count
    FROM nexmark
    GROUP BY 
        TUMBLE(INTERVAL '1 minute'),
        person.name
);

Copy link
Member

@mwylde mwylde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for the slow review and thanks for the fix!

@mwylde mwylde merged commit b1b4fae into ArroyoSystems:master Jan 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants