vine: vine_temp.c

[JinZhou5042]

Proposed Changes

This PR includes the following features:

• temp files related functions and logics are brought to a new file vine_temp.c, which controls file replication, redundancy cleanup, and disk load shifting.
• temp_files_to_replicate is turned into a priority queue, files with least replicas are considerred first to replicate.
• vine_prune will not trigger the removal of a record in temp_files_to_replicate, which makes the data pruning faster, the replication will fail silently if no available replicas.

Merge Checklist

The following items must be completed before PRs can be merged.
Check these off to verify you have completed all steps.

• make test Run local tests prior to pushing.
• make format Format source code to comply with lint policies. Note that some lint errors can only be resolved manually (e.g., Python)
• make lint Run lint on source code prior to pushing.
• Manual Update: Update the manual to reflect user-visible changes.
• Type Labels: Select a github label for the type: bugfix, enhancement, etc.
• Product Labels: Select a github label for the product: TaskVine, Makeflow, etc.
• PR RTM: Mark your PR as ready to merge.

[btovar]

ready to merge?

[btovar]

I see you included the cleanup of redundant replicas. We should find a way so that the redundant tasks are not created.

[JinZhou5042]

Could you elaborate a bit on what redundant tasks refer to?

[btovar]

Sorry, I misunderstood the source! Once you fix the conflicts I can merge this one.