Skip to content

Conversation

@thePermission
Copy link

@thePermission thePermission commented Feb 17, 2025

Check List

  • Tests have been run in packages where changes made if available
  • Linter has been run for changed code
  • Tests for the changes have been added if not covered yet
  • Docs have been added / updated if required

Issue Reference this PR resolves

#7905

Description of Changes Made

Using Parquet Metadata to store and read creation time instead of using the creation time from filesystem.

@thePermission thePermission requested a review from a team as a code owner February 17, 2025 15:52
@github-actions github-actions bot added the pr:community Contribution from Cube.js community members. label Feb 17, 2025
@srh srh self-requested a review March 7, 2025 19:53
@thePermission
Copy link
Author

@srh is there anything i can do to speed this up? Thats currently blocking us from working with cube, so i would be very happy to help getting this done as fast as possible :-)

@thePermission
Copy link
Author

@igorlukanin what can i do to get this feature? Would be pretty important to us acutally.

@KSDaemon KSDaemon changed the title Issues/7905 Fix created time in cubestore parquet files Jul 17, 2025
@igorlukanin igorlukanin added the cube store Issues relating to Cube Store label Jul 18, 2025
@srh
Copy link
Contributor

srh commented Jul 21, 2025

Creation time is used here not to refer to when the first copy of the parquet file was created but when the file was copied from remote to local storage. The cleanup loop is supposed to periodically clean up files (all of them) but leave those copied more recently than cleanup_local_files_delay to the file system.

So, using in-file metadata for parquet creation time is not the value we want to use for this. It might be the case that using last modified time would work just as well (and just the same) as using creation time though.

@thePermission
Copy link
Author

thePermission commented Jul 22, 2025

The problem is, that not all filesystems have a creation_time and cube then failes to clean up. Currently Cube Store is not working for every filesystem this way.

@KSDaemon KSDaemon requested review from waralexrom and removed request for srh October 9, 2025 10:14
@KSDaemon KSDaemon unassigned srh Oct 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cube store Issues relating to Cube Store pr:community Contribution from Cube.js community members.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants