corrupted plotfiles when using async output?

[BenWibking]

Hi,

I can't read several plotfiles produced with my code with either yt or VisIt. (VisIt crashes when trying to read them with lots of BoxLib::Error::FABio::read_header(): expected 'F' !!! messages.)

I've examined the files manually, looking for NaNs, or other strange things in the text files, but I can't find anything. They were all produced as part of a run along with many hundreds of other plotfiles which I can all read with both yt and VisIt, so I am at a bit of a loss as to how these 36 plotfiles do not work (of a total of 1,001 plotfiles output during this run).

I've uploaded the 36 unreadable plotfiles here if someone has the chance to take a look.

Thanks,
Ben

[etpalmer63]

Hi Ben,

In the very first plotfile I was able to get the errors:

erik@Mesh:~/Downloads$ fnan.gnu.ex bad_plt01000
amrex::Error::0::FABio::read_header(): expected 'F' !!!

and

erik@Mesh:~/Downloads$ fsnapshot.gnu.ex bad_plt01000
amrex::Abort::0::ERROR: density not found in pltfile bad_plt01000 !!!

I'll start to look into this today.

Thanks for reporting the issue!

Erik

[drummerdoc]

Quick suggestion... diff the Header file and Cell_H files with ones from a plotfile that works

[etpalmer63]

Hi Ben, if its easy can you also upload one plotfile that works? I would also like to look at the diff, as @drummerdoc suggested.

Erik

[BenWibking]

This is the output from an otherwise-identical simulation with async_out=0:

$ cat plt01200/Level_2/Cell_H
1
1
11
0
(8 0
((288,448,448) (383,575,575) (0,0,0))
((384,448,448) (447,575,575) (0,0,0))
((192,480,448) (223,543,479) (0,0,0))
((224,448,448) (287,575,511) (0,0,0))
((160,448,480) (223,575,511) (0,0,0))
((160,448,512) (223,575,543) (0,0,0))
((224,448,512) (287,575,575) (0,0,0))
((192,480,544) (223,543,575) (0,0,0))
)
8
FabOnDisk: Cell_D_00002 0
FabOnDisk: Cell_D_00005 0
FabOnDisk: Cell_D_00003 0
FabOnDisk: Cell_D_00003 5767267
FabOnDisk: Cell_D_00001 0
FabOnDisk: Cell_D_00000 0
FabOnDisk: Cell_D_00007 0
FabOnDisk: Cell_D_00006 0

[BenWibking]

Something is very wrong. Here is a full diff between async_out=1 and async_out=0 for the same binary and same inputs at the same output time:

diff -r async_out/plt01800/Header no_async/plt01800/Header
15c15
< 1198899740169.0359
---
> 1198894561904.8069
27c27
< 0 8 1198899740169.0359
---
> 0 8 1198894561904.8069
54c54
< 1 8 1198899740169.0359
---
> 1 8 1198894561904.8069
81c81
< 2 8 1198899740169.0359
---
> 2 8 1198894561904.8069
84a85,86
> 6.7484375000000004e+19 7.7125e+19
> 2.4101562499999998e+19 4.3382812499999998e+19
86,87c88,89
< 4.3382812499999998e+19 6.2664062499999998e+19
< 6.7484375000000004e+19 8.6765624999999996e+19
---
> 7.7125e+19 8.6765624999999996e+19
> 4.3382812499999998e+19 5.784375e+19
89c91,92
< 6.2664062499999998e+19 7.2304687500000002e+19
---
> 6.7484375000000004e+19 7.7125e+19
> 4.3382812499999998e+19 5.784375e+19
90a94,95
> 7.7125e+19 8.6765624999999996e+19
> 5.784375e+19 7.2304687500000002e+19
92,93d96
< 7.2304687500000002e+19 7.7125e+19
< 6.7484375000000004e+19 7.7125e+19
95c98,99
< 7.2304687500000002e+19 8.1945312499999998e+19
---
> 5.784375e+19 7.2304687500000002e+19
> 6.7484375000000004e+19 8.6765624999999996e+19
96a101,102
> 7.2304687500000002e+19 8.6765624999999996e+19
> 6.7484375000000004e+19 8.6765624999999996e+19
98,104c104
< 8.1945312499999998e+19 8.6765624999999996e+19
< 6.7484375000000004e+19 7.2304687500000002e+19
< 7.2304687500000002e+19 7.7125e+19
< 7.7125e+19 8.1945312499999998e+19
< 7.2304687500000002e+19 7.7125e+19
< 6.7484375000000004e+19 7.7125e+19
< 7.2304687500000002e+19 8.1945312499999998e+19
---
> 7.2304687500000002e+19 8.6765624999999996e+19
Binary files async_out/plt01800/Level_0/Cell_D_00000 and no_async/plt01800/Level_0/Cell_D_00000 differ
Binary files async_out/plt01800/Level_0/Cell_D_00001 and no_async/plt01800/Level_0/Cell_D_00001 differ
Binary files async_out/plt01800/Level_0/Cell_D_00002 and no_async/plt01800/Level_0/Cell_D_00002 differ
Binary files async_out/plt01800/Level_0/Cell_D_00003 and no_async/plt01800/Level_0/Cell_D_00003 differ
Binary files async_out/plt01800/Level_0/Cell_D_00004 and no_async/plt01800/Level_0/Cell_D_00004 differ
Binary files async_out/plt01800/Level_0/Cell_D_00005 and no_async/plt01800/Level_0/Cell_D_00005 differ
Binary files async_out/plt01800/Level_0/Cell_D_00006 and no_async/plt01800/Level_0/Cell_D_00006 differ
Binary files async_out/plt01800/Level_0/Cell_D_00007 and no_async/plt01800/Level_0/Cell_D_00007 differ
diff -r async_out/plt01800/Level_0/Cell_H no_async/plt01800/Level_0/Cell_H
26,33c26,33
< 4.8635444976234991e-27,-5.2880385793749465e-17,-1.4680799776851224e-18,-1.1743938808111388e-18,5.1742311563002152e-12,-7.3718605992312016e-19,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935823370748756e+02,
< 7.1895403124088729e-27,4.3358734927521179e-19,-2.0106608516931199e-19,-2.0842666303337349e-19,6.1695087253489698e-11,-1.2517443359450824e-13,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.3254327161546759e+07,
< 5.1830533409800136e-27,-5.2707293903946327e-17,-8.7862721491921381e-17,-1.0658423674230198e-18,5.1742312223697367e-12,-7.7754497162376201e-18,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935823600599554e+02,
< 7.1894848709342394e-27,4.9060950306171708e-19,-1.8028012539617197e-19,-1.8538373602439889e-19,6.2528133976876061e-11,-1.8639477848874635e-13,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.3248939634383991e+07,
< 5.0091158705815039e-27,-5.1967760474506324e-17,-1.3207641526894248e-18,-8.8121982486004859e-17,5.1742312237233830e-12,-1.4035802896025223e-19,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935823605394557e+02,
< 7.1894446877768330e-27,4.0701713947772764e-19,-1.9809819779990103e-19,-1.5545344941835690e-19,5.8136360610184395e-11,-7.2659046903746869e-14,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.3253251833262667e+07,
< 4.3936120243440984e-27,-5.2324164661036084e-17,-8.8001845339983434e-17,-8.8060644693190031e-17,5.1742312325867565e-12,-6.2953593017485862e-19,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935823640252931e+02,
< 7.1894998543209545e-27,4.3992895911729357e-19,-1.9438363199802145e-19,-1.7422487905506432e-19,5.8730927384057179e-11,-1.4203308472936962e-12,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.3247565789233863e+07,
---
> 5.1078790284040915e-27,-5.2538790085901950e-17,-1.2044021677598560e-18,-8.6695088390179206e-19,5.1689295604052083e-12,-4.7554593075703147e-14,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935823664402369e+02,
> 7.1882580700999421e-27,4.1291231462038275e-19,-1.7346093688322691e-19,-1.8360470229709488e-19,5.9108780385056268e-11,-1.2515604397307770e-13,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.3215575363187697e+07,
> 4.4505814663559615e-27,-5.2172123789509019e-17,-8.7772908493072059e-17,-8.5787843332930482e-19,5.1689296053852924e-12,-2.1782065538224153e-18,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935823610172258e+02,
> 7.1567652135135797e-27,4.0777624435608459e-19,-1.6428214096393740e-19,-1.8525726830168322e-19,5.7461697291379823e-11,-5.1060808506937063e-15,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.3217654072732210e+07,
> 4.8684030359084682e-27,-5.2398390797868050e-17,-1.1400802485936617e-18,-8.7784789729585526e-17,5.1689295754074899e-12,-1.7089896550237049e-12,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935823563410275e+02,
> 7.1883182990075455e-27,4.3227403571999964e-19,-1.8110536849608758e-19,-1.8763160083743096e-19,5.9372228375258567e-11,-2.5633859761206253e-14,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.3210777259987168e+07,
> 4.7818332138428333e-27,-5.2426342016031321e-17,-8.8119769209454292e-17,-8.8003873336539528e-17,5.1689289009735920e-12,-3.0406220858467359e-16,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935823634555055e+02,
> 7.1881514910834829e-27,4.3364851890563405e-19,-1.7886659639177105e-19,-1.9365714176947027e-19,5.9844376218191598e-11,-4.0040640740579043e-14,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.3214458859654851e+07,
36,43c36,43
< 1.2321308552802354e-22,1.2622200201679219e-16,8.7963479037169127e-17,8.8139212694062588e-17,2.2463438787108334e-10,5.1794640317825937e+00,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0420076101763174e+07,
< 1.0506559226183281e-26,1.2334078716445144e-18,1.8333212911217174e-19,1.6083270828416774e-19,1.3771031642689567e-10,3.6973140921935606e-05,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.0468481193882760e+07,
< 1.2322625287223575e-22,1.2614240779924134e-16,1.0000516096943664e-18,8.7916315915452723e-17,2.2454024067501243e-10,5.1800160139620619e+00,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0435376787557229e+07,
< 1.0506393933907785e-26,1.2334202749217858e-18,1.7507909708264657e-19,2.1985213076003478e-19,1.3771214217029179e-10,3.8382911633940618e-05,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.0603301291279282e+07,
< 1.2322221222031328e-22,1.2597330585391720e-16,8.8195604875369195e-17,1.3011128061229836e-18,2.2461651131125354e-10,5.1798441205073278e+00,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0474696263914086e+07,
< 1.0503420406976328e-26,1.2333843835101845e-18,2.1470047859834130e-19,2.0670362543932719e-19,1.3770851130087662e-10,4.0780336363098181e-05,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.1058080167702693e+07,
< 1.2321497340484306e-22,1.2615521826866694e-16,1.2147214608014767e-18,1.5395530284147029e-18,2.2444617617426752e-10,5.1795384054296782e+00,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0444124644707218e+07,
< 1.0503478415979682e-26,1.2334309880204417e-18,2.0795126292481649e-19,2.2131219998817995e-19,1.3771325277314540e-10,4.0321783962202754e-05,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.0854577395798653e+07,
---
> 1.2324966271428405e-22,1.2608775877492281e-16,8.8064952561540509e-17,8.8021826084072725e-17,2.2447524433645479e-10,5.1809983698663729e+00,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0424481178164415e+07,
> 1.0465877239872188e-26,1.2329392690891432e-18,1.5052788636022393e-19,1.5522765413753566e-19,1.3764643726767507e-10,4.2686761728279422e-05,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.0331703256842963e+07,
> 1.2323301789195987e-22,1.2620440501886801e-16,1.0459675401598931e-18,8.8101519630638131e-17,2.2444496195231933e-10,5.1803005637886770e+00,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0430766871911548e+07,
> 1.0468373372946130e-26,1.2329819798959120e-18,2.6158040846364279e-19,1.9488342248066014e-19,1.3765054490125804e-10,3.9780379658571266e-05,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.0342550005515262e+07,
> 1.2324227586804876e-22,1.2617310618960692e-16,8.8078693097337046e-17,1.3084033660156113e-18,2.2459611407657154e-10,5.1806883375514454e+00,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0429134464306146e+07,
> 1.0460426714641023e-26,1.2329286740312550e-18,1.5668435935179243e-19,1.8504823141755175e-19,1.3764439053630758e-10,4.7104532792105053e-05,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.0730917011521451e+07,
> 1.2318059627877181e-22,1.2613055158222970e-16,9.5332775339275064e-19,1.3435824034197869e-18,2.2484985429766811e-10,5.1780989304910747e+00,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0461941397096246e+07,
> 1.0465503566008912e-26,1.2329706571759203e-18,2.1480484087404890e-19,1.9694182908772937e-19,1.3764904339183683e-10,4.1344897101513290e-05,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.0353983788138609e+07,
Binary files async_out/plt01800/Level_1/Cell_D_00000 and no_async/plt01800/Level_1/Cell_D_00000 differ
Binary files async_out/plt01800/Level_1/Cell_D_00001 and no_async/plt01800/Level_1/Cell_D_00001 differ
Binary files async_out/plt01800/Level_1/Cell_D_00002 and no_async/plt01800/Level_1/Cell_D_00002 differ
Binary files async_out/plt01800/Level_1/Cell_D_00003 and no_async/plt01800/Level_1/Cell_D_00003 differ
Binary files async_out/plt01800/Level_1/Cell_D_00004 and no_async/plt01800/Level_1/Cell_D_00004 differ
Binary files async_out/plt01800/Level_1/Cell_D_00005 and no_async/plt01800/Level_1/Cell_D_00005 differ
Binary files async_out/plt01800/Level_1/Cell_D_00006 and no_async/plt01800/Level_1/Cell_D_00006 differ
Binary files async_out/plt01800/Level_1/Cell_D_00007 and no_async/plt01800/Level_1/Cell_D_00007 differ
diff -r async_out/plt01800/Level_1/Cell_H no_async/plt01800/Level_1/Cell_H
16,21d15
< FabOnDisk: Cell_D_00003 0
< FabOnDisk: Cell_D_00005 0
< FabOnDisk: Cell_D_00000 0
< FabOnDisk: Cell_D_00002 0
< FabOnDisk: Cell_D_00004 0
< FabOnDisk: Cell_D_00006 0
22a17,20
> FabOnDisk: Cell_D_00006 0
> FabOnDisk: Cell_D_00002 0
> FabOnDisk: Cell_D_00000 0
> FabOnDisk: Cell_D_00005 0
23a22,23
> FabOnDisk: Cell_D_00003 0
> FabOnDisk: Cell_D_00004 0
26,33c26,33
< 4.1316533050008024e-27,-7.1130968556186583e-17,-2.8868839728716202e-18,-2.5750741045793471e-18,5.1742299599260468e-12,-1.6768587723387047e-17,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935821424526171e+02,
< 4.8084815037373262e-27,-7.3189294071821529e-17,-1.3882507554649723e-16,-2.6553447277511679e-18,5.1742308614969417e-12,-4.3106846622188267e-17,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935822957926018e+02,
< 4.2885078659708155e-27,-7.1210266523368793e-17,-2.8416823486178929e-18,-1.3975086672871597e-16,5.1742303060231892e-12,-2.3187275345133598e-18,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935822002817469e+02,
< 3.6806977010623919e-27,-7.2500186584004014e-17,-1.3926552144821324e-16,-1.3905974711714934e-16,5.1742308693004973e-12,-3.2566217649646955e-20,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935823401474369e+02,
< 5.4300439187579743e-27,4.8019182042706350e-20,-3.7457879320172402e-19,-3.7688249606913381e-19,2.9539775837583851e-11,-8.5783922768615127e-13,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.0225562528087236e+07,
< 6.2206602005825713e-27,7.6288065154039813e-21,-2.8679913427473827e-19,-3.2358267389084311e-19,3.6491788651480645e-11,-3.1474352512080810e-13,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.2017477540474441e+07,
< 4.7348830164082172e-27,1.4321010699149456e-19,-3.9308215251714751e-19,-3.1614938397081549e-19,3.6180882200753507e-11,-3.9721674048493499e-13,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.1204463781526163e+07,
< 5.6108924644292172e-27,9.3635164155379356e-20,-3.4047061773643108e-19,-3.3991174808034744e-19,3.7963192756853043e-11,-1.7475133114598102e-11,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,1.9564965790146783e+07,
---
> 4.7091119776380806e-27,-7.2049551162267563e-17,-2.1457461847184824e-18,-2.3481643538609269e-18,5.1689290956528013e-12,-2.0640563109097556e-16,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935821965383866e+02,
> 2.9812837859659447e-27,-7.2241423610837276e-17,-1.3912286943799262e-16,-1.9310374782516100e-18,5.1689292514394032e-12,-2.1499373962860112e-16,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935819107077413e+02,
> 4.4950521990869671e-27,-7.2494174302676575e-17,-2.5875037683627679e-18,-1.3853349425833013e-16,5.1689291676366052e-12,-1.1767781075864228e-16,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935770484400228e+02,
> 3.7945397648440552e-27,-7.2170944953541269e-17,-1.3934464291812474e-16,-1.3837123151134621e-16,5.1689236397583592e-12,-1.2075744991513350e-15,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935822137861896e+02,
> 6.3203948344178426e-27,9.9996224509107761e-20,-3.1932341349373495e-19,-3.1779129424923973e-19,4.3981431336193160e-11,-7.9726869983729495e-11,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.0971266129942551e+07,
> 6.0117692846833952e-27,9.5912694730043530e-20,-3.4564579135340967e-19,-4.2935443345505008e-19,3.7468399101703964e-11,-4.8355692808174537e-14,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.0832287715431735e+07,
> 5.8088827486237759e-27,7.9189280228578410e-20,-3.8368003228842007e-19,-3.8711275052742230e-19,3.4732379251936379e-11,-6.2953024942380734e-12,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.3624333364704065e+07,
> 5.7788276519084313e-27,1.0357470334111257e-19,-3.9551018970279482e-19,-3.2599845263150876e-19,3.7020175410964695e-11,-1.2991485900496706e-13,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.1131859412972234e+07,
36,43c36,43
< 2.5378692128550329e-22,3.2024842500586687e-16,1.3909898815951872e-16,1.3927556054516733e-16,3.2123934586936941e-10,1.0662298577450848e+01,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0717886023595043e+07,
< 2.5378569076181102e-22,3.1998891544968887e-16,2.6333556125730030e-18,1.3973957576440181e-16,3.2080642204546176e-10,1.0662245118957868e+01,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0683834746147066e+07,
< 2.5392227429665292e-22,3.2027515077512367e-16,1.3968855265733537e-16,3.3128405202789685e-18,3.2119622490623610e-10,1.0668011933482516e+01,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0741226946056791e+07,
< 2.5396979860307007e-22,3.2014119414854150e-16,2.7497962882541911e-18,3.2122030768993295e-18,3.2099443444114453e-10,1.0670003997666983e+01,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0645317490948692e+07,
< 1.0793388449147362e-26,1.0035234080929013e-18,3.1606246082074157e-19,3.3085721625020172e-19,1.0264086697434674e-10,1.4844648565416925e-04,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.2963801497662067e+07,
< 1.0200697044011073e-26,1.0069915726785358e-18,3.5219734437314387e-19,4.3266686538904327e-19,1.0828112624552722e-10,1.0511381188569825e-04,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.2885269636460189e+07,
< 1.0080670812473343e-26,1.0206490085307920e-18,3.5220878993225598e-19,4.3357040172967395e-19,1.0653014670110256e-10,1.5141345235346912e-04,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.2505543976709068e+07,
< 1.1655876574951522e-26,1.1641103887276665e-18,3.3328087822284976e-19,3.6189231470596086e-19,1.0652673950913294e-10,2.2818495837317163e-04,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.1987245572253115e+07,
---
> 2.5366515958892872e-22,3.1969023162573029e-16,1.3921738895587120e-16,1.3888855846510147e-16,3.2051526853733764e-10,1.0657194217395382e+01,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0733624416475900e+07,
> 2.5405205489304142e-22,3.2079988541285822e-16,2.6854129321796830e-18,1.3993859764460330e-16,3.2169261678601821e-10,1.0673444318234317e+01,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0691404919165976e+07,
> 2.5408284596583586e-22,3.2083588232705980e-16,1.3915447599512090e-16,3.0546555646190809e-18,3.2171275258614944e-10,1.0674749097734262e+01,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0640382902722895e+07,
> 2.5362959843997251e-22,3.1958852565388183e-16,3.0122711269229973e-18,3.0618912423624855e-18,3.2050166555187635e-10,1.0655709253106647e+01,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0653964782584079e+07,
> 1.1614121257909470e-26,1.0731321048940327e-18,3.1723841635453476e-19,3.6152870517370257e-19,1.0255158692245587e-10,1.7113129066484656e-04,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.3165104318909872e+07,
> 1.0250880035223150e-26,9.9159413071415649e-19,3.9047729084200283e-19,3.1497690356824060e-19,1.0532218689628165e-10,1.0174839143357214e-04,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.3494442759318195e+07,
> 1.0285405941558804e-26,1.0535383254989294e-18,3.2724363553232299e-19,2.9999642100474995e-19,1.1265175916883885e-10,1.1005530474604153e-04,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.2176105255213890e+07,
> 1.0279191924733159e-26,9.8076086247789802e-19,4.0465725602869751e-19,4.6493134672097804e-19,1.0323639940010396e-10,1.0402690007640417e-04,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.3905033783331841e+07,
Only in no_async/plt01800/Level_2: Cell_D_00000
Binary files async_out/plt01800/Level_2/Cell_D_00001 and no_async/plt01800/Level_2/Cell_D_00001 differ
Only in no_async/plt01800/Level_2: Cell_D_00002
Binary files async_out/plt01800/Level_2/Cell_D_00003 and no_async/plt01800/Level_2/Cell_D_00003 differ
Only in no_async/plt01800/Level_2: Cell_D_00004
Binary files async_out/plt01800/Level_2/Cell_D_00005 and no_async/plt01800/Level_2/Cell_D_00005 differ
Binary files async_out/plt01800/Level_2/Cell_D_00006 and no_async/plt01800/Level_2/Cell_D_00006 differ
Binary files async_out/plt01800/Level_2/Cell_D_00007 and no_async/plt01800/Level_2/Cell_D_00007 differ
diff -r async_out/plt01800/Level_2/Cell_H no_async/plt01800/Level_2/Cell_H
6,13c6,13
< ((160,448,448) (287,575,575) (0,0,0))
< ((288,448,448) (415,575,575) (0,0,0))
< ((416,448,448) (479,575,575) (0,0,0))
< ((480,448,448) (511,511,511) (0,0,0))
< ((480,512,448) (543,575,511) (0,0,0))
< ((544,448,480) (575,479,511) (0,0,0))
< ((512,480,448) (543,511,511) (0,0,0))
< ((480,448,512) (543,575,575) (0,0,0))
---
> ((160,448,448) (287,575,511) (0,0,0))
> ((160,448,512) (287,575,575) (0,0,0))
> ((288,448,448) (383,575,511) (0,0,0))
> ((288,448,512) (383,575,575) (0,0,0))
> ((384,448,448) (479,575,511) (0,0,0))
> ((384,448,512) (479,575,575) (0,0,0))
> ((480,448,448) (575,575,511) (0,0,0))
> ((480,448,512) (575,575,575) (0,0,0))
17,23c17,23
< FabOnDisk: Cell_D_00005 4720768115359713324
< FabOnDisk: Cell_D_00006 4709114260590291694
< FabOnDisk: Cell_D_00001 43254156
< FabOnDisk: Cell_D_00001 4719352842383482840
< FabOnDisk: Cell_D_00001 4719259303109005296
< FabOnDisk: Cell_D_00001 4718864219762573545
< FabOnDisk: Cell_D_00007 4716575634823662062
---
> FabOnDisk: Cell_D_00001 0
> FabOnDisk: Cell_D_00004 0
> FabOnDisk: Cell_D_00006 0
> FabOnDisk: Cell_D_00002 0
> FabOnDisk: Cell_D_00005 0
> FabOnDisk: Cell_D_00000 0
> FabOnDisk: Cell_D_00007 0
26,33c26,33
< 9.1179555555537983e-316,4.0730542995718434e-27,-9.7245788507850760e-17,-2.0894619958229578e-16,-2.0838734052218138e-16,5.1742107199178654e-12,-1.3162302184632738e-16,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,
< 0.0000000000000000e+00,0.0000000000000000e+00,2.4347154096363684e-26,1.2646872367335486e-18,5.7903477855254722e-19,5.3919210673970496e-19,1.1229612602490845e-10,7.9659094803803622e-04,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,
< 3.4011287783672556e+07,0.0000000000000000e+00,1.0850264729800355e-26,1.0574901061445551e-18,3.7190415882697045e-19,3.8419523852131728e-19,1.0806703512621756e-10,1.5628840240497168e-04,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,
< 0.0000000000000000e+00,9.8012470579721434e-27,9.6037155934752542e-19,2.9067713661534360e-19,2.8376648262093754e-19,1.0253394065137567e-10,6.6573238923778346e-05,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,
< 5.6987680776888786e-317,9.4667806189393167e-27,1.0244679847276803e-18,2.8674540930146422e-19,2.9514146273886459e-19,1.1010060923520104e-10,4.2658958303721713e-05,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,
< 1.7096255320567697e-316,9.1418828565820435e-27,9.4050697991318255e-19,1.3584257261772303e-19,1.3283249190924585e-19,9.8812742999560204e-11,3.6627955746295244e-05,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,
< 1.8520984024364121e-316,9.4188609764276075e-27,9.4029414196432386e-19,2.5920640665454952e-19,1.7981891062381272e-19,1.0123161697330351e-10,4.1516385060678289e-05,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,
< 3.3225850496607050e+07,0.0000000000000000e+00,9.4715257557045823e-27,1.0169499955150704e-18,3.2656019062883699e-19,3.1822345956458747e-19,1.0928736094171031e-10,6.5262948269583841e-05,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,
---
> 3.5513010301306641e-27,-9.8792509307127397e-17,-2.0914593634369091e-16,-3.6983161434129532e-18,5.1689176131325815e-12,-1.0645415203599877e-16,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935720805218023e+02,
> 4.0734073545516035e-27,-9.6324197602137262e-17,-2.0870175487030725e-16,-2.0799448488333103e-16,5.1688846822417756e-12,-1.0621165296577508e-19,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.9935511152072684e+02,
> 2.7728586537519547e-27,-2.2276856630160787e-19,-5.8726970549817886e-19,-5.0546034979224792e-19,1.0967247097069995e-11,-1.0687901788414875e-15,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,6.9748711518312786e+06,
> 3.2824228506941042e-27,-2.4648115981825462e-19,-5.4466369680602241e-19,-4.7632418488869270e-19,1.2061228732766177e-11,-1.8506748724049916e-15,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,7.4299216251769252e+06,
> 5.5822539815592517e-27,7.3657084673714505e-20,-3.8433494552449596e-19,-4.5997897261523756e-19,3.0143852924981019e-11,-1.3888194421753217e-17,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,1.9876828307828285e+07,
> 5.3793078787014284e-27,2.1402851831174523e-20,-4.1088639267925195e-19,-4.2518963458520135e-19,3.2040280113472572e-11,-3.7569024422373408e-15,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.0537818537777171e+07,
> 6.5294778509463574e-27,3.2101878876835215e-19,-3.1469238053344447e-19,-2.8610066492956588e-19,4.8862740790839768e-11,-3.0320586992415368e-10,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.6213524116466463e+07,
> 6.2013601091007871e-27,2.9311966803897482e-19,-2.9754506552939802e-19,-3.0689328227879921e-19,4.4128848815756681e-11,-4.4385346730836902e-12,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.5913272354024373e+07,
36,43c36,43
< 0.0000000000000000e+00,3.2765084561828027e-22,4.9102375320802490e-16,2.0977109497350081e-16,2.0959873407669863e-16,4.9063057384621991e-10,1.3758801230622302e+01,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,
< 9.1179555555537983e-316,3.3273181809597887e-27,-2.6869736533170538e-19,-6.3276663490257791e-19,-5.3620232810627691e-19,1.3225996344328993e-11,-4.8908343806687180e-17,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,
< 4.5589802234018461e-316,5.0725691617846153e-27,1.0220670482139751e-19,-4.2724288139367107e-19,-3.5432540435612488e-19,3.4132463657071187e-11,-4.2227924299189760e-17,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,
< 6.8296703190105730e-27,3.1206703504101502e-19,-2.4317377128373675e-19,-2.4822758233866543e-19,4.7692377048369194e-11,-2.7055463873459609e-15,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.4882597304232381e+07,
< 7.1460997700877401e-27,3.2814947369295672e-19,-2.6821206701345301e-19,-3.1117773469612748e-19,5.0950544706872485e-11,-1.0010435094171224e-25,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.6621354675470568e+07,
< 7.5056803093768716e-27,5.5359238925452093e-19,-1.6028586832981020e-19,-2.1943018994121564e-19,6.5084694710434207e-11,-1.0998871741134938e-29,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.6830535369049631e+07,
< 6.8738494658758185e-27,3.8935923547895832e-19,-2.4659541401027498e-19,-2.6446393627134177e-19,5.4041901957834107e-11,-6.0063446925172223e-33,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,2.6843118632798769e+07,
< 2.2794925573258699e-316,4.4823979662364099e-27,1.9816957102944218e-19,-4.1062565289609483e-19,-2.8746681729274449e-19,3.0778290313418480e-11,-7.8147463693306559e-13,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,
---
> 3.2788553383245800e-22,4.9197890328528545e-16,2.0880912682100074e-16,2.0886927523664273e-16,4.9051931285268899e-10,1.3768673318780557e+01,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0922615637399018e+07,
> 3.2819972085518831e-22,4.9331596438131763e-16,2.0670749595494314e-16,4.7310757511827013e-18,4.9198773759689839e-10,1.3781848672386719e+01,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,4.0850945724850059e+07,
> 1.8662881969880247e-26,1.1302247887002326e-18,5.3325265940896377e-19,5.5330466438188758e-19,1.0551829383748517e-10,5.8693585702680967e-04,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.4159699120178953e+07,
> 2.2602818765445281e-26,1.1592342713262754e-18,4.8061615580622600e-19,4.5471459055136762e-19,1.0558266221155428e-10,7.5605364049133653e-04,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.3322041798572421e+07,
> 1.2015874939137870e-26,1.1090345973114126e-18,4.1783686118745068e-19,3.8532492140040259e-19,1.0442061263885620e-10,1.9446460116690116e-04,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.3869102208416410e+07,
> 1.0489768305623299e-26,1.0744902475828864e-18,4.3448830261460171e-19,5.1376679004489892e-19,1.1523727804242732e-10,1.1429096020272657e-04,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.4451421288642682e+07,
> 9.5258032710514002e-27,1.0172898866950452e-18,3.1941595551639558e-19,3.0963682254463149e-19,1.0911488992470343e-10,5.5457874081919726e-05,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.1651656135242231e+07,
> 9.5494105076257575e-27,1.0636499827545892e-18,2.7735574697792467e-19,3.1495116606082929e-19,1.1376218500155368e-10,4.8989799285609624e-05,5.2900000000000001e-14,0.0000000000000000e+00,0.0000000000000000e+00,0.0000000000000000e+00,3.1959222455840357e+07,

[BenWibking]

Does either regular AMReX I/O or async I/O rely on POSIX file locking in any way?

[WeiqunZhang]

No. Only the I/O process will touch the Header file. For the data files, which process has the right to write is coordinated via MPI messaging.

[BenWibking]

I am also getting different simulation outputs if I switch on HDF5, compared with AMReX native I/O with async disabled. This is really, really bizarre...