BenchMarking of pmemkv gets stuck for higher range inputs like 50k 100k

[alshamsh93]

QUESTION:

Details

Hi All,

I am first time using pmemkv. I wanted to test how much pmemkv takes time while inserting , reading , deleting 50k 100k data.
Below attached program work fine while i am giving inputs as 1k or 2k but as usual it takes longer time to insert read delete.
While i am testing with 50k 100k data it gets stuck for long time or takes much time to process like 5 min to insert / delete/ read.

can you please help me what is the mistake/issue or any bug . I am running this on ubuntu 20.

Please find the attached files
main.txt.txt
pmemkv_header.txt.txt
pmemkv_source.txt.txt

[lukaszstolarczuk]

Hi, I just run your program on my desktop (with emulated pmem) and it ended in no time, even for 500k.

What machine are you running this on? And what path are you using for this testing - I'm just assuming you're using regular disk device underneath (like SSD or HDD) so I'd expect to take some time on these devices.

 luke .../sample/build make -j4 && ./sample
[100%] Built target sample
Enter number of Integer you want to benchMark = 10000
Enter file name for pmemkv :/mnt/pmem/my_file
mmap impelementation type :mmapPmemKV
--------------------------------------------------------------
BenchMarking perform on number of records :10000
pmemkv_result.pmemkv_insert_time_micro : 57222
pmemkv_result.pmemkv_read_time_micro : 13293
pmemkv_result.pmemkv_single_delete_time_micro : 12
pmemkv_result.pmemkv_all_delete_time_micro : 54889


 luke .../sample/build make -j4 && ./sample
[100%] Built target sample
Enter number of Integer you want to benchMark = 100000
Enter file name for pmemkv :/mnt/pmem/my_file2
mmap impelementation type :mmapPmemKV
--------------------------------------------------------------
BenchMarking perform on number of records :100000
pmemkv_result.pmemkv_insert_time_micro : 583248
pmemkv_result.pmemkv_read_time_micro : 130528
pmemkv_result.pmemkv_single_delete_time_micro : 23
pmemkv_result.pmemkv_all_delete_time_micro : 565930


 luke .../sample/build make -j4 && ./sample
[100%] Built target sample
Enter number of Integer you want to benchMark = 500000
Enter file name for pmemkv :/mnt/pmem/my_file3
mmap impelementation type :mmapPmemKV
--------------------------------------------------------------
BenchMarking perform on number of records :500000
pmemkv_result.pmemkv_insert_time_micro : 2750415
pmemkv_result.pmemkv_read_time_micro : 563592
pmemkv_result.pmemkv_single_delete_time_micro : 31
pmemkv_result.pmemkv_all_delete_time_micro : 2826960

[alshamsh93]

@lukaszstolarczuk I am using ubuntu 20 . It is having HDD hard disk type. Simultaneously on the same machine I am using other libraries for memory mapped files like raw implementation, using boost mmap and googleleveldb for comparison which is fast and better to use for my projects. These library work fine on the same machine.

what to meant by emulated pmem? I have installed pmem with help of github mentioned steps. like -

first - Installed the prerequisite packages as mentioned on pmemkv github page under - Installing on ubuntu section
second - cloned the pmdk build and install
third - cloned pmemobj-cpp build and install
fourth - cloned memkind build and install
fifth - cloned pmemkv and build and install.

My code work fine in your system , even in my current mahine output is coming for smaller inputs like 1k 2k data. But while i am giving inputs as 50k 100k output is coming after 5 to 7 mins.

Is there any different way to install emulated pmem? please clarify this.

[lukaszstolarczuk]

Hi,
What I meant by emulated pmem is - if you don't have "a real PMEM" (like NVDIMMs hardware) to test your code on, it's nice to emulate the behavior of pmem in any Linux machine. I linked previously our blog post that describes it a little (https://pmem.io/blog/2016/02/how-to-emulate-persistent-memory).

PMDK and our other libraries are hardware agnostic and are working on any medium - functionally, it doesn't matter if you run a piece of code on HDD, SSD, RAM, or Persistent Memory (Non-Volatile DIMMs). It will perform differently, but functionally it will work fine on all types of medium.

To speed up the execution, I can recommend you not to use a file on an HDD device but rather on an emulated pmem (/mnt/mem/somefile as described in the blog article) or just on RAM (e.g. on tmpfs like /dev/shm/somefile).