utilization of parquet for intermediate storage

[jr3cermak]

There are several steps involved in migration to parquet for intermediate processing of slocum glider data.

• ensure dbdreader reproduces similar results to slocum binaries (Feature: optionally include first record in data payload smerckel/dbdreader#18)
• replacement of convertDbds.sh with dbdreader/parquet
• desired storage pattern for parquet (just using tables for now)

Is there a particular storage pattern or design desired for the parquet data structures?
REF: https://arrow.apache.org/docs/python/parquet.html#parquet-file-writing-options

• Enforce version 2.4? 2.6?
• Ensure the structure is queryable by time for speedy subsetting?
• If enforcing 2.6, timestamp units become less an issue
• Partitioning? Glider ID, Deployment ID, Process method (rt vs. delayed), QC'd (Level 0, 1, ...)
  • local file system or plug into directly to duckdb (https://duckdb.org/docs/guides/python/filesystems.html) or allow both

Have to nail down a potential dbdreader issue first.

[jr3cermak]

I can now confirm that dbdreader can be used to reproduce dba_merge. This code snippet will take the two files and merge the data via pandas to form a dataframe similar to that product to dba_merge.

At this stage, the merged dataframe can be tossed into the parquet framework.

    # Invoke dbdreader on sbd file
    sbdf = "%s%s" % (os.path.join("binary", bn), ".sbd")
    sbd = dbdreader.DBD(sbdf, cacheDir=cache_dir, skip_initial_line=False)

    # Invoke dbdreader on tbd file
    tbdf = "%s%s" % (os.path.join("binary", bn), ".tbd")
    tbd = dbdreader.DBD(tbdf, cacheDir=cache_dir, skip_initial_line=False)

    # Compare columns
    dbd_columns = sbd.parameterNames + tbd.parameterNames
    dbd_columns.sort()

    slo_columns = list(gu.columns)
    slo_columns.sort()

    assert set(dbd_columns) == set(slo_columns), \
        "Slocum merged and dbdreader columns do not match"

    # Simulate: dba_merge

    sbd_data = pd.DataFrame()
    for col in sbd.parameterNames:
        tm, data = sbd.get(col, return_nans=True, decimalLatLon=False)
        sbd_data[col] = data

    sbd_data = sbd_data.set_index('m_present_time')
    sbd.close()

    tbd_data = pd.DataFrame()
    for col in tbd.parameterNames:
        tm, data = tbd.get(col, return_nans=True, decimalLatLon=False)
        tbd_data[col] = data
        if col == 'sci_m_present_time':
            # Create a duplicate column for merging
            tbd_data['m_present_time'] = data

    tbd_data = tbd_data.set_index('m_present_time')
    tbd.close()

    dbd_data = pd.merge(sbd_data, tbd_data, on=['m_present_time'], how='outer', sort=True)
    dbd_data = dbd_data.reset_index()

A quick comparison of 9 separate files and values obtained via dba_merge and dbdreader:

$ python cmpData.py 


===> ascii/unit_507_2022_078_0_0_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 4.010009767796419e-06 4.94964599617731e-05
   m_battery 3.4179687524726887e-07 4.945220947227824e-05
   m_battpos 2.682209010451686e-09 4.988937377659042e-07
   m_coulomb_amphr_total 3.271484376909939e-05 0.0004809570312431788
   m_de_oil_vol 1.4968490600342221e-06 0.0004863281250209184
   m_depth 0.0 0.0004963378906381877
   m_depth_rate 0.0 4.78054046626486e-07
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 7.615256309660007e-10 4.785308838073377e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 7.77435302801166e-08 4.983215331932911e-06
   m_leakdetect_voltage_forward 1.5014648457878366e-07 4.974365234389211e-06
   m_lon 0.0 4.547473508864641e-11
   m_pitch 9.338641165923534e-09 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 9.887695284760412e-07 4.94964599617731e-05
   m_roll 2.153873440713383e-09 4.683246612735559e-08
   m_vacuum 2.639770535495245e-08 4.2085266112934505e-05
   m_water_depth 0.0 0.0004663085937579581
   m_water_vx 2.952008247569804e-08 4.5307064058808244e-08
   m_water_vy 2.8525765056281946e-13 3.911998271849404e-08
   sci_echodroid_aggindex 0.0 4.936542510880493e-07
   sci_echodroid_ctrmass 0.0 4.931030273525039e-05
   sci_echodroid_eqarea 0.0 4.97009277324878e-05
   sci_echodroid_inertia 0.0 0.00048474121092567657
   sci_echodroid_propocc 0.0 4.342842102156297e-06
   sci_echodroid_sa 0.0 4.931030273525039e-05
   sci_echodroid_sv 0.0 4.931030273525039e-05
   sci_flbbcd_bb_units 0.0 4.051462113930163e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 2.8610229518832853e-08
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1871337912694457e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3727417008956309e-07


===> ascii/unit_507_2022_078_0_1_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 1.5191650390988798e-05 1.5191650390988798e-05
   m_battery 3.0398559569988493e-05 3.076019287107101e-05
   m_battpos 4.100151062003121e-07 4.100151062003121e-07
   m_coulomb_amphr_total 0.00041699218752455636 0.0004729003906049911
   m_de_oil_vol 0.0003813476562299911 0.0003813476562299911
   m_depth 0.0 0.0
   m_depth_rate 0.0 0.0
   m_gps_lat 0.0 0.0
   m_gps_lon 1.8189894035458565e-12 1.8189894035458565e-12
   m_heading 9.155273437144729e-07 9.155273437144729e-07
   m_lat 1.8189894035458565e-12 1.8189894035458565e-12
   m_leakdetect_voltage 1.0704803465877433e-06 4.6228790284708055e-06
   m_leakdetect_voltage_forward 1.5668487547415566e-06 2.0844268799891097e-06
   m_lon 4.547473508864641e-11 4.547473508864641e-11
   m_pitch 1.534729003882518e-07 1.534729003882518e-07
   m_present_time 7.152557373046875e-07 4.5299530029296875e-06
   m_raw_altitude 3.3721923827556566e-05 3.3721923827556566e-05
   m_roll 1.9259738928689707e-08 1.9259738928689707e-08
   m_vacuum 4.727478026822496e-06 2.901611328098852e-05
   m_water_depth 0.00015649414064000666 0.00015649414064000666
   m_water_vx 2.952008247569804e-08 2.952008247569804e-08
   m_water_vy 3.911998271849404e-08 3.911998271849404e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 4.5299530029296875e-06 4.5299530029296875e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 3.356933611087243e-07 3.356933611087243e-07
   sci_water_temp 0.0 0.0


===> ascii/unit_507_2022_078_0_2_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 5.052185057863312e-06 4.674072265586915e-05
   m_battery 4.711914062482947e-06 4.655303955125589e-05
   m_battpos 2.6850545406867377e-09 4.959058761688784e-07
   m_coulomb_amphr_total 1.0986328106810106e-05 0.0004921875000150067
   m_de_oil_vol 1.300048833030587e-07 0.00042431640622453415
   m_depth 0.0 0.0004937744140676159
   m_depth_rate 0.0 4.837455749584496e-07
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 9.047412874441108e-09 4.929962158506385e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 2.4375915508301205e-07 4.974365234389211e-06
   m_leakdetect_voltage_forward 7.77435302801166e-08 4.559249878077765e-06
   m_lon 3.637978807091713e-12 4.9112713895738125e-11
   m_pitch 1.5230512619320014e-09 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 4.943847642380206e-07 4.674072265586915e-05
   m_roll 2.153873440713383e-09 4.683246612735559e-08
   m_vacuum 2.639770535495245e-08 1.972961425700248e-05
   m_water_depth 0.0 0.0004902343749790816
   m_water_vx 2.8267610071262828e-09 2.952008247569804e-08
   m_water_vy 1.6263961774565239e-09 3.911998271849404e-08
   sci_echodroid_aggindex 0.0 3.8234138488579816e-07
   sci_echodroid_ctrmass 0.0 4.7564697268853706e-05
   sci_echodroid_eqarea 0.0 4.992523193436682e-05
   sci_echodroid_inertia 0.0 0.00048474121092567657
   sci_echodroid_propocc 0.0 9.159469605002357e-07
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.931030273525039e-05
   sci_flbbcd_bb_units 0.0 4.047749638507597e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 5.0926208583490506e-08
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1894226092934446e-07
   sci_water_pressure 2.8610229740877458e-09 9.460449206244448e-07
   sci_water_temp 0.0 2.3803710913483656e-07


===> ascii/unit_507_2022_078_0_3_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 5.052185057863312e-06 5.052185057863312e-06
   m_battery 1.0815429687838218e-05 4.9467468262065495e-05
   m_battpos 1.6136550906598046e-07 4.100151062003121e-07
   m_coulomb_amphr_total 6.811523439864686e-05 0.00017114257809680566
   m_de_oil_vol 0.00041650390625136424 0.00041650390625136424
   m_depth 0.0 1.7530155181633145e-08
   m_depth_rate 0.0 0.0
   m_gps_lat 9.094947017729282e-13 9.094947017729282e-13
   m_gps_lon 1.8189894035458565e-12 1.8189894035458565e-12
   m_heading 3.634643555017192e-07 3.634643555017192e-07
   m_lat 2.7284841053187847e-12 2.7284841053187847e-12
   m_leakdetect_voltage 5.741119384339299e-07 3.140792846689777e-06
   m_leakdetect_voltage_forward 1.4114379887075756e-08 9.998321535142907e-07
   m_lon 4.547473508864641e-11 4.547473508864641e-11
   m_pitch 1.534729003882518e-07 1.534729003882518e-07
   m_present_time 4.76837158203125e-07 3.814697265625e-06
   m_raw_altitude 3.4446716306035796e-06 3.4446716306035796e-06
   m_roll 3.993892669362786e-08 3.993892669362786e-08
   m_vacuum 3.5263061519685834e-06 4.795837402404857e-05
   m_water_depth 6.762695312545475e-05 6.762695312545475e-05
   m_water_vx 2.8267610071262828e-09 2.8267610071262828e-09
   m_water_vy 1.6263961774565239e-09 1.6263961774565239e-09
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 2.1457672119140625e-06 2.1457672119140625e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 2.6702880795426154e-07 2.6702880795426154e-07
   sci_water_temp 0.0 0.0


===> ascii/unit_507_2022_078_0_4_sbd.dat
   c_wpt_lat 0.0 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 5.052185057863312e-06 4.9743652340339395e-05
   m_battery 4.3533325193578776e-06 4.867706298838925e-05
   m_battpos 2.24747955792845e-09 4.659767150749783e-07
   m_coulomb_amphr_total 8.789062519554136e-06 0.0004931640625045475
   m_de_oil_vol 1.5197753882034704e-06 0.0004482421874740794
   m_depth 0.0 0.0004984130859497782
   m_depth_rate 4.781246162877117e-10 4.788923263499001e-07
   m_gps_lat 0.0 9.094947017729282e-13
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 5.4016113359978135e-08 4.980773925744586e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 7.77435302801166e-08 4.983215331932911e-06
   m_leakdetect_voltage_forward 1.4114379887075756e-08 4.974365234389211e-06
   m_lon 1.8189894035458565e-12 4.9112713895738125e-11
   m_pitch 7.615256309660007e-10 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 3.4446716306035796e-06 4.9743652340339395e-05
   m_roll 2.153873440713383e-09 4.683246612735559e-08
   m_vacuum 1.6632079713474468e-08 4.795837402404857e-05
   m_water_depth 0.0 0.0004943847656306843
   m_water_vx 2.7995109529166395e-10 4.298452734811248e-09
   m_water_vy 1.6263961774565239e-09 4.524588585047562e-08
   sci_echodroid_aggindex 0.0 4.7902107236375e-07
   sci_echodroid_ctrmass 0.0 4.894409179456716e-05
   sci_echodroid_eqarea 0.0 4.985046386707381e-05
   sci_echodroid_inertia 0.0 0.00048474121092567657
   sci_echodroid_propocc 0.0 9.159469605002357e-07
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.7564697268853706e-05
   sci_flbbcd_bb_units 0.0 4.049237966631536e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 6.637573246592865e-08
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1917114273174434e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3727417008956309e-07


===> ascii/unit_507_2022_078_0_5_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 2.012023925956896e-05 2.012023925956896e-05
   m_battery 3.368377685575297e-05 4.551544189368428e-05
   m_battpos 1.6136550906598046e-07 1.6136550906598046e-07
   m_coulomb_amphr_total 7.641601564500888e-05 0.00019335937497544364
   m_de_oil_vol 0.00039599609374363354 0.00039599609374363354
   m_depth 0.0 1.7530155181633145e-08
   m_depth_rate 0.0 0.0
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 0.0
   m_heading 1.1898803711218875e-06 1.1898803711218875e-06
   m_lat 9.094947017729282e-13 9.094947017729282e-13
   m_leakdetect_voltage 1.1428833008864103e-06 1.68342590312065e-06
   m_leakdetect_voltage_forward 1.0210418701639412e-06 2.056198120214958e-06
   m_lon 3.8198777474462986e-11 3.8198777474462986e-11
   m_pitch 1.5993309021578028e-07 1.5993309021578028e-07
   m_present_time 0.0 4.5299530029296875e-06
   m_raw_altitude 3.840332031401772e-05 3.840332031401772e-05
   m_roll 2.4125957495679806e-08 2.4125957495679806e-08
   m_vacuum 4.447937012486136e-06 3.832702636685781e-05
   m_water_depth 0.0003625488281215894 0.0003625488281215894
   m_water_vx 2.7995109529166395e-10 2.7995109529166395e-10
   m_water_vy 4.524588585047562e-08 4.524588585047562e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 4.5299530029296875e-06 4.5299530029296875e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 6.256103510793309e-07 6.256103510793309e-07
   sci_water_temp 0.0 0.0


===> ascii/unit_507_2022_078_0_6_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 6.713867186647349e-07 4.707641601697787e-05
   m_battery 6.86645513781059e-08 4.9478149414028394e-05
   m_battpos 6.442186237590186e-11 4.7547626494393036e-07
   m_coulomb_amphr_total 1.1718749988176569e-05 0.0004953613281486469
   m_de_oil_vol 3.486938476626733e-06 0.00045166015627273737
   m_depth 0.0 0.0005000000000023874
   m_depth_rate 0.0 4.955315589943998e-07
   m_gps_lat 0.0 9.094947017729282e-13
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 5.546569825654046e-08 4.954223632669397e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 2.4375915508301205e-07 4.974365234389211e-06
   m_leakdetect_voltage_forward 7.77435302801166e-08 4.808349609586315e-06
   m_lon 1.8189894035458565e-12 4.3655745685100555e-11
   m_pitch 1.5230512619320014e-09 4.966812133799792e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 4.7607422004602995e-07 4.707641601697787e-05
   m_roll 2.153873440713383e-09 4.1464805602964283e-07
   m_vacuum 2.639770535495245e-08 4.555969238317914e-05
   m_water_depth 0.0 0.00042846679687613687
   m_water_vx 2.7995109529166395e-10 4.036182165399749e-09
   m_water_vy 1.688668727767295e-08 4.524588585047562e-08
   sci_echodroid_aggindex 0.0 4.939737320019955e-07
   sci_echodroid_ctrmass 0.0 4.931030273525039e-05
   sci_echodroid_eqarea 0.0 4.858703613308535e-05
   sci_echodroid_inertia 0.0 0.0004923706054711374
   sci_echodroid_propocc 0.0 9.159469605002357e-07
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.7564697268853706e-05
   sci_flbbcd_bb_units 0.0 4.055169820814733e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 1.0452270515770579e-07
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1917114273174434e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3727417008956309e-07


===> ascii/unit_507_2022_078_0_7_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 3.096618652520533e-05 3.096618652520533e-05
   m_battery 2.7200317383346828e-05 4.9478149414028394e-05
   m_battpos 4.6995544433769965e-07 4.6995544433769965e-07
   m_coulomb_amphr_total 7.373046872771738e-05 0.00012280273438136646
   m_de_oil_vol 0.00041650390625136424 0.00041650390625136424
   m_depth 0.0 0.0
   m_depth_rate 0.0 0.0
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 0.0
   m_heading 1.3704681398607477e-06 1.3704681398607477e-06
   m_lat 1.8189894035458565e-12 1.8189894035458565e-12
   m_leakdetect_voltage 5.529022217842794e-07 1.5880584718352964e-06
   m_leakdetect_voltage_forward 1.4944458008869788e-06 3.2573699950688706e-06
   m_lon 4.001776687800884e-11 4.001776687800884e-11
   m_pitch 5.8336257935387437e-08 5.8336257935387437e-08
   m_present_time 4.76837158203125e-07 4.76837158203125e-06
   m_raw_altitude 1.1764526366420114e-05 1.1764526366420114e-05
   m_roll 4.1464805602964283e-07 4.1464805602964283e-07
   m_vacuum 2.2567749020652172e-06 2.66418457037787e-05
   m_water_depth 0.00015234375001682565 0.00015234375001682565
   m_water_vx 1.2298977374730624e-09 1.2298977374730624e-09
   m_water_vy 1.688668727767295e-08 1.688668727767295e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 2.86102294921875e-06 2.86102294921875e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 5.79833983493927e-07 5.79833983493927e-07
   sci_water_temp 0.0 0.0


===> ascii/unit_507_2022_078_0_8_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 6.713867151120212e-07 4.933776855509109e-05
   m_battery 3.372192374229144e-07 4.8551940917462844e-05
   m_battpos 2.6850545406867377e-09 4.6995544433769965e-07
   m_coulomb_amphr_total 1.782226564728262e-05 0.0004960937499731699
   m_de_oil_vol 6.896972593040118e-08 0.00042382812500818545
   m_depth 0.0 0.0004981689453131821
   m_depth_rate 0.0 4.955315589943998e-07
   m_gps_lat 0.0 9.094947017729282e-13
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 2.3651121061618596e-09 4.9253082274880455e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 1.8020629877213423e-07 4.995574951038861e-06
   m_leakdetect_voltage_forward 1.4114379887075756e-08 4.901962280090544e-06
   m_lon 1.8189894035458565e-12 4.9112713895738125e-11
   m_pitch 2.155303935325037e-09 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 8.300781217940312e-07 4.933776855509109e-05
   m_roll 2.153873440713383e-09 4.1464805602964283e-07
   m_vacuum 1.6632079713474468e-08 3.2514953613116404e-05
   m_water_depth 0.0 0.0004304199218836402
   m_water_vx 1.2298977374730624e-09 4.4742631908484265e-09
   m_water_vy 7.65342712469308e-09 4.458808899182598e-08
   sci_echodroid_aggindex 0.0 4.809398650995611e-07
   sci_echodroid_ctrmass 0.0 4.931030273525039e-05
   sci_echodroid_eqarea 0.0 4.702453613347757e-05
   sci_echodroid_inertia 0.0 0.0004923706054711374
   sci_echodroid_propocc 0.0 3.5113906859951527e-06
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.931030273525039e-05
   sci_flbbcd_bb_units 0.0 4.053686261228789e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 1.1787414555541886e-07
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1917114273174434e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3803710913483656e-07


===> ascii/unit_507_2022_078_0_9_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 3.840332031401772e-05 3.840332031401772e-05
   m_battery 5.773925781937805e-06 4.3586730956945985e-05
   m_battpos 6.470108027123445e-08 6.470108027123445e-08
   m_coulomb_amphr_total 0.0004262695312604592 0.0004941406249940883
   m_de_oil_vol 0.00042163085936408606 0.00042163085936408606
   m_depth 0.0 3.758859635008549e-09
   m_depth_rate 0.0 0.0
   m_gps_lat 9.094947017729282e-13 9.094947017729282e-13
   m_gps_lon 0.0 0.0
   m_heading 2.923889160122428e-06 2.923889160122428e-06
   m_lat 9.094947017729282e-13 9.094947017729282e-13
   m_leakdetect_voltage 3.23616027841922e-06 3.5877227784197885e-06
   m_leakdetect_voltage_forward 2.573776245018422e-06 4.248352050595372e-06
   m_lon 4.3655745685100555e-11 4.3655745685100555e-11
   m_pitch 1.534729003882518e-07 1.534729003882518e-07
   m_present_time 9.5367431640625e-07 2.6226043701171875e-06
   m_raw_altitude 4.691772461029586e-05 4.691772461029586e-05
   m_roll 3.993892669362786e-08 3.993892669362786e-08
   m_vacuum 2.748107910122144e-06 9.059143065925923e-06
   m_water_depth 0.0002636718750181899 0.0002636718750181899
   m_water_vx 4.4742631908484265e-09 4.4742631908484265e-09
   m_water_vy 4.458808899182598e-08 4.458808899182598e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 2.384185791015625e-06 2.384185791015625e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 9.155273517080786e-08 9.155273517080786e-08
   sci_water_temp 0.0 0.0
Maximum deviation: 0.0005000000000023874
(gutils_py3_9) cermak@mom6node0:~/glider/deployments/unit_507/20220212/gutils/sfmc/dbdreader$ python cmpData.py | less
(gutils_py3_9) cermak@mom6node0:~/glider/deployments/unit_507/20220212/gutils/sfmc/dbdreader$ python cmpData.py 


===> ascii/unit_507_2022_078_0_0_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 4.010009767796419e-06 4.94964599617731e-05
   m_battery 3.4179687524726887e-07 4.945220947227824e-05
   m_battpos 2.682209010451686e-09 4.988937377659042e-07
   m_coulomb_amphr_total 3.271484376909939e-05 0.0004809570312431788
   m_de_oil_vol 1.4968490600342221e-06 0.0004863281250209184
   m_depth 0.0 0.0004963378906381877
   m_depth_rate 0.0 4.78054046626486e-07
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 7.615256309660007e-10 4.785308838073377e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 7.77435302801166e-08 4.983215331932911e-06
   m_leakdetect_voltage_forward 1.5014648457878366e-07 4.974365234389211e-06
   m_lon 0.0 4.547473508864641e-11
   m_pitch 9.338641165923534e-09 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 9.887695284760412e-07 4.94964599617731e-05
   m_roll 2.153873440713383e-09 4.683246612735559e-08
   m_vacuum 2.639770535495245e-08 4.2085266112934505e-05
   m_water_depth 0.0 0.0004663085937579581
   m_water_vx 2.952008247569804e-08 4.5307064058808244e-08
   m_water_vy 2.8525765056281946e-13 3.911998271849404e-08
   sci_echodroid_aggindex 0.0 4.936542510880493e-07
   sci_echodroid_ctrmass 0.0 4.931030273525039e-05
   sci_echodroid_eqarea 0.0 4.97009277324878e-05
   sci_echodroid_inertia 0.0 0.00048474121092567657
   sci_echodroid_propocc 0.0 4.342842102156297e-06
   sci_echodroid_sa 0.0 4.931030273525039e-05
   sci_echodroid_sv 0.0 4.931030273525039e-05
   sci_flbbcd_bb_units 0.0 4.051462113930163e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 2.8610229518832853e-08
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1871337912694457e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3727417008956309e-07


===> ascii/unit_507_2022_078_0_1_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 1.5191650390988798e-05 1.5191650390988798e-05
   m_battery 3.0398559569988493e-05 3.076019287107101e-05
   m_battpos 4.100151062003121e-07 4.100151062003121e-07
   m_coulomb_amphr_total 0.00041699218752455636 0.0004729003906049911
   m_de_oil_vol 0.0003813476562299911 0.0003813476562299911
   m_depth 0.0 0.0
   m_depth_rate 0.0 0.0
   m_gps_lat 0.0 0.0
   m_gps_lon 1.8189894035458565e-12 1.8189894035458565e-12
   m_heading 9.155273437144729e-07 9.155273437144729e-07
   m_lat 1.8189894035458565e-12 1.8189894035458565e-12
   m_leakdetect_voltage 1.0704803465877433e-06 4.6228790284708055e-06
   m_leakdetect_voltage_forward 1.5668487547415566e-06 2.0844268799891097e-06
   m_lon 4.547473508864641e-11 4.547473508864641e-11
   m_pitch 1.534729003882518e-07 1.534729003882518e-07
   m_present_time 7.152557373046875e-07 4.5299530029296875e-06
   m_raw_altitude 3.3721923827556566e-05 3.3721923827556566e-05
   m_roll 1.9259738928689707e-08 1.9259738928689707e-08
   m_vacuum 4.727478026822496e-06 2.901611328098852e-05
   m_water_depth 0.00015649414064000666 0.00015649414064000666
   m_water_vx 2.952008247569804e-08 2.952008247569804e-08
   m_water_vy 3.911998271849404e-08 3.911998271849404e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 4.5299530029296875e-06 4.5299530029296875e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 3.356933611087243e-07 3.356933611087243e-07
   sci_water_temp 0.0 0.0


===> ascii/unit_507_2022_078_0_2_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 5.052185057863312e-06 4.674072265586915e-05
   m_battery 4.711914062482947e-06 4.655303955125589e-05
   m_battpos 2.6850545406867377e-09 4.959058761688784e-07
   m_coulomb_amphr_total 1.0986328106810106e-05 0.0004921875000150067
   m_de_oil_vol 1.300048833030587e-07 0.00042431640622453415
   m_depth 0.0 0.0004937744140676159
   m_depth_rate 0.0 4.837455749584496e-07
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 9.047412874441108e-09 4.929962158506385e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 2.4375915508301205e-07 4.974365234389211e-06
   m_leakdetect_voltage_forward 7.77435302801166e-08 4.559249878077765e-06
   m_lon 3.637978807091713e-12 4.9112713895738125e-11
   m_pitch 1.5230512619320014e-09 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 4.943847642380206e-07 4.674072265586915e-05
   m_roll 2.153873440713383e-09 4.683246612735559e-08
   m_vacuum 2.639770535495245e-08 1.972961425700248e-05
   m_water_depth 0.0 0.0004902343749790816
   m_water_vx 2.8267610071262828e-09 2.952008247569804e-08
   m_water_vy 1.6263961774565239e-09 3.911998271849404e-08
   sci_echodroid_aggindex 0.0 3.8234138488579816e-07
   sci_echodroid_ctrmass 0.0 4.7564697268853706e-05
   sci_echodroid_eqarea 0.0 4.992523193436682e-05
   sci_echodroid_inertia 0.0 0.00048474121092567657
   sci_echodroid_propocc 0.0 9.159469605002357e-07
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.931030273525039e-05
   sci_flbbcd_bb_units 0.0 4.047749638507597e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 5.0926208583490506e-08
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1894226092934446e-07
   sci_water_pressure 2.8610229740877458e-09 9.460449206244448e-07
   sci_water_temp 0.0 2.3803710913483656e-07


===> ascii/unit_507_2022_078_0_3_sbd.dat
   c_wpt_lat 9.094947017729282e-13 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 5.052185057863312e-06 5.052185057863312e-06
   m_battery 1.0815429687838218e-05 4.9467468262065495e-05
   m_battpos 1.6136550906598046e-07 4.100151062003121e-07
   m_coulomb_amphr_total 6.811523439864686e-05 0.00017114257809680566
   m_de_oil_vol 0.00041650390625136424 0.00041650390625136424
   m_depth 0.0 1.7530155181633145e-08
   m_depth_rate 0.0 0.0
   m_gps_lat 9.094947017729282e-13 9.094947017729282e-13
   m_gps_lon 1.8189894035458565e-12 1.8189894035458565e-12
   m_heading 3.634643555017192e-07 3.634643555017192e-07
   m_lat 2.7284841053187847e-12 2.7284841053187847e-12
   m_leakdetect_voltage 5.741119384339299e-07 3.140792846689777e-06
   m_leakdetect_voltage_forward 1.4114379887075756e-08 9.998321535142907e-07
   m_lon 4.547473508864641e-11 4.547473508864641e-11
   m_pitch 1.534729003882518e-07 1.534729003882518e-07
   m_present_time 4.76837158203125e-07 3.814697265625e-06
   m_raw_altitude 3.4446716306035796e-06 3.4446716306035796e-06
   m_roll 3.993892669362786e-08 3.993892669362786e-08
   m_vacuum 3.5263061519685834e-06 4.795837402404857e-05
   m_water_depth 6.762695312545475e-05 6.762695312545475e-05
   m_water_vx 2.8267610071262828e-09 2.8267610071262828e-09
   m_water_vy 1.6263961774565239e-09 1.6263961774565239e-09
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 2.1457672119140625e-06 2.1457672119140625e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 2.6702880795426154e-07 2.6702880795426154e-07
   sci_water_temp 0.0 0.0


===> ascii/unit_507_2022_078_0_4_sbd.dat
   c_wpt_lat 0.0 9.094947017729282e-13
   c_wpt_lon 0.0 0.0
   m_altitude 5.052185057863312e-06 4.9743652340339395e-05
   m_battery 4.3533325193578776e-06 4.867706298838925e-05
   m_battpos 2.24747955792845e-09 4.659767150749783e-07
   m_coulomb_amphr_total 8.789062519554136e-06 0.0004931640625045475
   m_de_oil_vol 1.5197753882034704e-06 0.0004482421874740794
   m_depth 0.0 0.0004984130859497782
   m_depth_rate 4.781246162877117e-10 4.788923263499001e-07
   m_gps_lat 0.0 9.094947017729282e-13
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 5.4016113359978135e-08 4.980773925744586e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 7.77435302801166e-08 4.983215331932911e-06
   m_leakdetect_voltage_forward 1.4114379887075756e-08 4.974365234389211e-06
   m_lon 1.8189894035458565e-12 4.9112713895738125e-11
   m_pitch 7.615256309660007e-10 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 3.4446716306035796e-06 4.9743652340339395e-05
   m_roll 2.153873440713383e-09 4.683246612735559e-08
   m_vacuum 1.6632079713474468e-08 4.795837402404857e-05
   m_water_depth 0.0 0.0004943847656306843
   m_water_vx 2.7995109529166395e-10 4.298452734811248e-09
   m_water_vy 1.6263961774565239e-09 4.524588585047562e-08
   sci_echodroid_aggindex 0.0 4.7902107236375e-07
   sci_echodroid_ctrmass 0.0 4.894409179456716e-05
   sci_echodroid_eqarea 0.0 4.985046386707381e-05
   sci_echodroid_inertia 0.0 0.00048474121092567657
   sci_echodroid_propocc 0.0 9.159469605002357e-07
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.7564697268853706e-05
   sci_flbbcd_bb_units 0.0 4.049237966631536e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 6.637573246592865e-08
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1917114273174434e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3727417008956309e-07


===> ascii/unit_507_2022_078_0_5_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 2.012023925956896e-05 2.012023925956896e-05
   m_battery 3.368377685575297e-05 4.551544189368428e-05
   m_battpos 1.6136550906598046e-07 1.6136550906598046e-07
   m_coulomb_amphr_total 7.641601564500888e-05 0.00019335937497544364
   m_de_oil_vol 0.00039599609374363354 0.00039599609374363354
   m_depth 0.0 1.7530155181633145e-08
   m_depth_rate 0.0 0.0
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 0.0
   m_heading 1.1898803711218875e-06 1.1898803711218875e-06
   m_lat 9.094947017729282e-13 9.094947017729282e-13
   m_leakdetect_voltage 1.1428833008864103e-06 1.68342590312065e-06
   m_leakdetect_voltage_forward 1.0210418701639412e-06 2.056198120214958e-06
   m_lon 3.8198777474462986e-11 3.8198777474462986e-11
   m_pitch 1.5993309021578028e-07 1.5993309021578028e-07
   m_present_time 0.0 4.5299530029296875e-06
   m_raw_altitude 3.840332031401772e-05 3.840332031401772e-05
   m_roll 2.4125957495679806e-08 2.4125957495679806e-08
   m_vacuum 4.447937012486136e-06 3.832702636685781e-05
   m_water_depth 0.0003625488281215894 0.0003625488281215894
   m_water_vx 2.7995109529166395e-10 2.7995109529166395e-10
   m_water_vy 4.524588585047562e-08 4.524588585047562e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 4.5299530029296875e-06 4.5299530029296875e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 6.256103510793309e-07 6.256103510793309e-07
   sci_water_temp 0.0 0.0


===> ascii/unit_507_2022_078_0_6_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 6.713867186647349e-07 4.707641601697787e-05
   m_battery 6.86645513781059e-08 4.9478149414028394e-05
   m_battpos 6.442186237590186e-11 4.7547626494393036e-07
   m_coulomb_amphr_total 1.1718749988176569e-05 0.0004953613281486469
   m_de_oil_vol 3.486938476626733e-06 0.00045166015627273737
   m_depth 0.0 0.0005000000000023874
   m_depth_rate 0.0 4.955315589943998e-07
   m_gps_lat 0.0 9.094947017729282e-13
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 5.546569825654046e-08 4.954223632669397e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 2.4375915508301205e-07 4.974365234389211e-06
   m_leakdetect_voltage_forward 7.77435302801166e-08 4.808349609586315e-06
   m_lon 1.8189894035458565e-12 4.3655745685100555e-11
   m_pitch 1.5230512619320014e-09 4.966812133799792e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 4.7607422004602995e-07 4.707641601697787e-05
   m_roll 2.153873440713383e-09 4.1464805602964283e-07
   m_vacuum 2.639770535495245e-08 4.555969238317914e-05
   m_water_depth 0.0 0.00042846679687613687
   m_water_vx 2.7995109529166395e-10 4.036182165399749e-09
   m_water_vy 1.688668727767295e-08 4.524588585047562e-08
   sci_echodroid_aggindex 0.0 4.939737320019955e-07
   sci_echodroid_ctrmass 0.0 4.931030273525039e-05
   sci_echodroid_eqarea 0.0 4.858703613308535e-05
   sci_echodroid_inertia 0.0 0.0004923706054711374
   sci_echodroid_propocc 0.0 9.159469605002357e-07
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.7564697268853706e-05
   sci_flbbcd_bb_units 0.0 4.055169820814733e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 1.0452270515770579e-07
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1917114273174434e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3727417008956309e-07


===> ascii/unit_507_2022_078_0_7_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 3.096618652520533e-05 3.096618652520533e-05
   m_battery 2.7200317383346828e-05 4.9478149414028394e-05
   m_battpos 4.6995544433769965e-07 4.6995544433769965e-07
   m_coulomb_amphr_total 7.373046872771738e-05 0.00012280273438136646
   m_de_oil_vol 0.00041650390625136424 0.00041650390625136424
   m_depth 0.0 0.0
   m_depth_rate 0.0 0.0
   m_gps_lat 0.0 0.0
   m_gps_lon 0.0 0.0
   m_heading 1.3704681398607477e-06 1.3704681398607477e-06
   m_lat 1.8189894035458565e-12 1.8189894035458565e-12
   m_leakdetect_voltage 5.529022217842794e-07 1.5880584718352964e-06
   m_leakdetect_voltage_forward 1.4944458008869788e-06 3.2573699950688706e-06
   m_lon 4.001776687800884e-11 4.001776687800884e-11
   m_pitch 5.8336257935387437e-08 5.8336257935387437e-08
   m_present_time 4.76837158203125e-07 4.76837158203125e-06
   m_raw_altitude 1.1764526366420114e-05 1.1764526366420114e-05
   m_roll 4.1464805602964283e-07 4.1464805602964283e-07
   m_vacuum 2.2567749020652172e-06 2.66418457037787e-05
   m_water_depth 0.00015234375001682565 0.00015234375001682565
   m_water_vx 1.2298977374730624e-09 1.2298977374730624e-09
   m_water_vy 1.688668727767295e-08 1.688668727767295e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 2.86102294921875e-06 2.86102294921875e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 5.79833983493927e-07 5.79833983493927e-07
   sci_water_temp 0.0 0.0


===> ascii/unit_507_2022_078_0_8_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 6.713867151120212e-07 4.933776855509109e-05
   m_battery 3.372192374229144e-07 4.8551940917462844e-05
   m_battpos 2.6850545406867377e-09 4.6995544433769965e-07
   m_coulomb_amphr_total 1.782226564728262e-05 0.0004960937499731699
   m_de_oil_vol 6.896972593040118e-08 0.00042382812500818545
   m_depth 0.0 0.0004981689453131821
   m_depth_rate 0.0 4.955315589943998e-07
   m_gps_lat 0.0 9.094947017729282e-13
   m_gps_lon 0.0 1.8189894035458565e-12
   m_heading 2.3651121061618596e-09 4.9253082274880455e-06
   m_lat 0.0 4.547473508864641e-12
   m_leakdetect_voltage 1.8020629877213423e-07 4.995574951038861e-06
   m_leakdetect_voltage_forward 1.4114379887075756e-08 4.901962280090544e-06
   m_lon 1.8189894035458565e-12 4.9112713895738125e-11
   m_pitch 2.155303935325037e-09 4.96858596821248e-07
   m_present_time 0.0 5.0067901611328125e-06
   m_raw_altitude 8.300781217940312e-07 4.933776855509109e-05
   m_roll 2.153873440713383e-09 4.1464805602964283e-07
   m_vacuum 1.6632079713474468e-08 3.2514953613116404e-05
   m_water_depth 0.0 0.0004304199218836402
   m_water_vx 1.2298977374730624e-09 4.4742631908484265e-09
   m_water_vy 7.65342712469308e-09 4.458808899182598e-08
   sci_echodroid_aggindex 0.0 4.809398650995611e-07
   sci_echodroid_ctrmass 0.0 4.931030273525039e-05
   sci_echodroid_eqarea 0.0 4.702453613347757e-05
   sci_echodroid_inertia 0.0 0.0004923706054711374
   sci_echodroid_propocc 0.0 3.5113906859951527e-06
   sci_echodroid_sa 0.0 4.862060546884095e-05
   sci_echodroid_sv 0.0 4.931030273525039e-05
   sci_flbbcd_bb_units 0.0 4.053686261228789e-09
   sci_flbbcd_cdom_units 0.0 4.158020017186459e-08
   sci_flbbcd_chlor_units 0.0 1.1787414555541886e-07
   sci_m_present_time 0.0 5.0067901611328125e-06
   sci_rinkoii_do 0.0 3.784179682497779e-06
   sci_water_cond 0.0 1.1917114273174434e-07
   sci_water_pressure 0.0 9.460449206244448e-07
   sci_water_temp 0.0 2.3803710913483656e-07


===> ascii/unit_507_2022_078_0_9_sbd.dat
   c_wpt_lat 0.0 0.0
   c_wpt_lon 0.0 0.0
   m_altitude 3.840332031401772e-05 3.840332031401772e-05
   m_battery 5.773925781937805e-06 4.3586730956945985e-05
   m_battpos 6.470108027123445e-08 6.470108027123445e-08
   m_coulomb_amphr_total 0.0004262695312604592 0.0004941406249940883
   m_de_oil_vol 0.00042163085936408606 0.00042163085936408606
   m_depth 0.0 3.758859635008549e-09
   m_depth_rate 0.0 0.0
   m_gps_lat 9.094947017729282e-13 9.094947017729282e-13
   m_gps_lon 0.0 0.0
   m_heading 2.923889160122428e-06 2.923889160122428e-06
   m_lat 9.094947017729282e-13 9.094947017729282e-13
   m_leakdetect_voltage 3.23616027841922e-06 3.5877227784197885e-06
   m_leakdetect_voltage_forward 2.573776245018422e-06 4.248352050595372e-06
   m_lon 4.3655745685100555e-11 4.3655745685100555e-11
   m_pitch 1.534729003882518e-07 1.534729003882518e-07
   m_present_time 9.5367431640625e-07 2.6226043701171875e-06
   m_raw_altitude 4.691772461029586e-05 4.691772461029586e-05
   m_roll 3.993892669362786e-08 3.993892669362786e-08
   m_vacuum 2.748107910122144e-06 9.059143065925923e-06
   m_water_depth 0.0002636718750181899 0.0002636718750181899
   m_water_vx 4.4742631908484265e-09 4.4742631908484265e-09
   m_water_vy 4.458808899182598e-08 4.458808899182598e-08
   sci_echodroid_aggindex 0.0 0.0
   sci_echodroid_ctrmass 0.0 0.0
   sci_echodroid_eqarea 0.0 0.0
   sci_echodroid_inertia 0.0 0.0
   sci_echodroid_propocc 0.0 0.0
   sci_echodroid_sa 0.0 0.0
   sci_echodroid_sv 0.0 0.0
   sci_flbbcd_bb_units 0.0 0.0
   sci_flbbcd_cdom_units 0.0 0.0
   sci_flbbcd_chlor_units 0.0 0.0
   sci_m_present_time 2.384185791015625e-06 2.384185791015625e-06
   sci_rinkoii_do 0.0 0.0
   sci_water_cond 0.0 0.0
   sci_water_pressure 9.155273517080786e-08 9.155273517080786e-08
   sci_water_temp 0.0 0.0

Maximum deviation: 0.0005000000000023874

The deviations occur as dba_merge does not pass through the full precision of the float values when the conversion to ascii happens. As I have mentioned in the past, we need to full precision from the dbdreader to extract the embedded echograms in the echometrics data.

The code will have to be refactored a bit to handle both pairs of files. The method should remain the same. Once that is written, the convertDbds.sh can go away with the ascii conversion step.

[jr3cermak]

Or convert the ascii reader using the above to parquet and then adapt the netcdf part to pull from parquet instead of ascii.

[jr3cermak]

Yes, will make this a gradual transition. Added a hook to extra_kwargs to enable utilization of dbdreader/parquet so both pathways will continue to exist for a little while.

deployment.json:

{
    "glider": "unit_507",
    "trajectory_date": "20220212T0000",
    "filters": {
        "tsint": 20,
        "filter_z": 1,
        "filter_time": 5,
        "filter_points": 5,
        "filter_distance": 1
    },
    "extra_kwargs": {
        "enable_parquet": true,
        "echograms": {
            "enable_nc": true,
            "enable_ascii": true,
            "enable_image": true,
            "plot_type": "profile",
            "plot_cmap": "ek80",
            "svb_thresholds": [-10,-20,-25,-30,-35,-45,-55],
            "svb_limits": [-10.0, -55.0],
            "echogram_range_bins": 20,
            "echogram_range": -60.0,
            "echogram_range_units": "meters"
        }
    },
    "attributes": {
        "acknowledgement": "This work was supported by funding from NOAA/IOOS/AOOS.",
        "comment": "",

Enabling parquet also assumes dbdreader will also be used.

$ pytest -k TestEchoMetricsFive
====================================================================================== test session starts =======================================================================================
platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.0.0 -- /home/cermak/miniconda3/envs/gutils_py3_9/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/cermak/src/GUTILS, configfile: setup.cfg
plugins: anyio-3.6.2
collected 36 items / 35 deselected / 1 selected                                                                                                                                                  

gutils/tests/test_slocum.py::TestEchoMetricsFive::test_echogram 2023-06-17 14:34:08,422 - gutils.slocum - INFO - PARQUET enabled deployment
2023-06-17 14:34:08,423 - gutils.slocum - INFO - Creating echogram profile images with colormap ek80
2023-06-17 14:34:23,540 - gutils.slocum - INFO - Converted unit_507-2022-078-0-0.sbd,unit_507-2022-078-0-0.tbd to unit_507_2022_078_0_0_sbd.dat

Now to bolt on the decoding code...

... and push certain things over to the nc processing side.

Getting parquet tables to store datetime[ns, UTC] was giving me a problem initially. This gets us to the goal of parquet tables and nc/xarray and preserving the seconds since 1990-0-0 units.

Once the merge happens:

    dbd_data = pd.merge(sbd_data, tbd_data, on=['m_present_time'], how='outer', sort=True)
    dbd_data = dbd_data.reset_index()

drop sci_m_present_time and rename m_present_time to time

    # This works for xarray and parquet
    dbd_data['time'] = [datetime.datetime.fromtimestamp(tm, tz=datetime.timezone.utc) for tm in dbd_data['time'].values]

    # Convert time to datetime and preserve fractional seconds
    # NOTE: works for xarray; not for parquet
    #dbd_data['time'] = pd.to_datetime(dbd_data['time'], utc=True, unit='s')

    # Make time the index
    dbd_data = dbd_data.set_index('time')

    # Write files to storage

    # write netcdf
    fn_xr = "%s%s" % (bn, ".nc")
    ds = dbd_data.to_xarray()
    ds['time'] = ds['time'].astype('datetime64[ns]')
    encodings = {}
    for dvar in ds.data_vars:
        encodings[dvar] = {"_FillValue": None}
    encodings.update({
        "time": {
            "_FillValue": None,
            "calendar": "standard",
            "units": "seconds since 1990-01-01T00:00:00Z"
        }
    })
    ds.to_netcdf(fn_xr, encoding = encodings)

    # write parquet
    fn_pq = "%s%s" % (bn, ".parquet")
    dbd_data.to_parquet(fn_pq)

    # Verify what we wrote can be read back in

    # read netcdf
    #nc_ver = xr.open_dataset(fn_xr)
    # NOTE: initial checks showed that original values differ using xarray

    # read parquet
    #pq_ver = pq.read_table(fn_pq)

Instead of *.dat files, there will be .parquet files in the ascii directory for the nc side to utilize with the flag. Temporary until things get fully moved about.

[jr3cermak]

As expected, picking up extra resolution by utilizing dbdreader and parquet for processing.

Original m_depth and m_altitude via dba_merge:

m_altitude = _, _, _, _, _, 24.5446, _, _, _, _, _, 21.6618, _, _, _, 
    19.8181, _, _, _, _, _ ;

 m_depth = _, _, 222.494995117188, _, _, 224.837997436523, _, _, _, 
    227.069000244141, _, _, _, 229.412002563477, _, _, _, _, _, 
    231.559997558594, _ ;

via dbdreader and parquet:

m_altitude = _, _, _, _, _, 24.5445671081543, _, _, _, _, _, 
    21.6617832183838, _, _, _, 19.8180713653564, _, _, _, _, _ ;

m_depth = _, _, 222.494979858398, _, _, 224.837829589844, _, _, _, 
    227.069107055664, _, _, _, 229.411956787109, _, _, _, _, _, 
    231.559555053711, _ ;

Still some work to do...