Large file import fails with http timeout

[mohankchandrashekar]

Hi,

Some of my oracle servers produce over 100mb nmon files which includes top data. there are on a an average of 'imported : 21393570 points !' imported.

with multiple servers like this the data load fails with error:

[httpd] ::1 - root [05/Mar/2018:08:44:51 -0500] "POST /write?consistency=&db=nmon_phx2&precision=s&rp= HTTP/1.1" 500 20 "-" "InfluxDBClient" 60b5b1d8-207b-11e8-8252-000000000000 31186780
[E] 2018-03-05T13:45:22Z [500] - "timeout" service=httpd

is there a way to use async batch upload of the huge nmon files?

Thanks,
Mohan

[adejoux]

Hello,

It's a really impressive nmon file. The solution would be to send data more regularly instead of waiting the end of the file. It's a pretty good improvement. I will work on it.

I will keep you updated.

Regards,

Alain

[mohankchandrashekar]

Alain, Thank you for quick response. The issue seems to be around importing TOP metrics. After filtering CPUUTIL0.*,JFSFILE,NFSCLIV2NFSCLIV3,DISKBSIZE,DISKBUSY,DISKXFER,JFSINODE,TOP the number of points imported has been reduced to 6816960 points. Also NMON for Linux (oracle rac) produces duplicate disk stat for /dev/sd.* instead of multipath device. I m writing external script to get data for /dev/mapper/mpath*. Which should reduce the input points much further. Will send you additional data when available. I found this syntax in influxdb 1.4 which made me think to post the request: curl -F "q=@<path_to_file>" -F "async=true" http://localhost:8086/query May be nmon2influxdb can read the source nmon create a tmp file and use this batch upload to get the data loaded faster. Thank you, Mohan

…

Sent from my iPhone

On Mar 6, 2018, at 4:53 AM, adejoux ***@***.***> wrote: Hello, It's a really impressive nmon file. The solution would be to send data more regularly instead of waiting the end of the file. It's a pretty good improvement. I will work on it. I will keep you updated. Regards, Alain — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[adejoux]

Hello,

I will try your solution. Thanks.

In the mean time, you can try to skip the TOP metric section with the following parameter 👍

import_skip_metrics="JFSINODE|PCPU|TOP"

I think you have too many processes information in your nmon file so it will be useful anyway or you can limit in nmon the number of top processes.

Regards,

Alain