Skip to content

[BUG] Investigate the use of memory resources in Parquet stack #21282

@mhaseeb123

Description

@mhaseeb123

Description

In the Parquet reader/writer, we may be using the _mr (the memory resource with which the output device memory should be allocated) to also allocate internal memory/vectors and (highly unlikely) vice versa with the cudf::get_current_device_resource_ref() for output memory.

Proposed fix

Create a new member variable called internal_mr or temp_mr initialized with cudf::get_current_device_resource_ref() or user provided temp_mr (as per proposed libcudf API design) and use that to allocate all internal memory (strictly no use of cudf::get_current_device_resource_ref() or not-providing an mr). xref #20780

Additional context

Hmm, since this memory is not for the output table, we should actually use cudf::get_current_device_resource_ref() for all these. That said, I think this is likely a parquet-wide issue that we need to fix (in a separate PR of course).

Originally posted by @mhaseeb123 in #21139 (comment)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcuIOcuIO issuelibcudfAffects libcudf (C++/CUDA) code.

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions