Calibration shape error with floats with 256+ profiles

[Hanyuan-Liu]

while using pyowc.calibration.calc_piecewisefit with floats with more than 256 profiles, there would be a error as below:

File /lib/python3.8/site-packages/pyowc/calibration.py:653
in calc_piecewisefit(float_dir, float_name, system_config)
line 653 pcond_factor[0][calindex] = condslope

ValueError: shape mismatch: value array of shape (256,) could not be broadcast to indexing result of shape (440,)

This error happens every time OWC tries to calculate fit for a float with a large number of profiles.

[Hanyuan-Liu]

forgot to mention, in the example I was working on float wmo 5906393, which has 511 profiles

[kamwal]

Dear Hanyuan-Liu,
Thanks for raising this.
Could you provide more details to enable us to reproduce this error.
Such as WMO numbers for floats affected by this error, the version of the reference data you used, and the configuration you used in your analysis.

[vidkris]

hi Hanyuan-Liu,
Could you pls check the versions of the packages that you are using?
especially the numpy(1.21.0),scipy(1.10.1), pandas(2.0.3) and matplotlib(3.7.5)

Could you also pass us the .mat file you have been using and the config that gets displayed. This can help us reproduce the error.
Since it processes for few profiles, we wanted to make sure the raw data and config are not the problem.

[Hanyuan-Liu]

Thank you for the quick responses. We run the code on a server with:
"numpy==1.22.3",
"pandas==1.4.3",
"scipy==1.7.3",
"matplotlib==3.5.1",

the reference files we used for the float 5906393 are as follow:

5906393.zip

wmo_boxes.mat.zip

We also use ARGO_for_DMQC_2023V03 and CTD_for_DMQC_2024V01.

For the config, we just use the default pyowc config, other than USER_CONFIG["FLOAT_PLOTS_FORMAT"] = "png"
Thank you for the investigation!

[Hanyuan-Liu]

i also did a test with a venv with scipy(1.10.1) and pandas(2.0.3), but the issue still persists.

[vidkris]

Thanks Hanyuan. That will be helpful in debugging. We will follow up soon.

[jamclar]

I've done some investigation into this, and I've narrowed down the code to a couple of offending lines.

I've ran the code with 5906393 and used the same versions as mentioned in the above comment, along with the same files.

The error we're seeing is:

File "/Users/jamesclare/Documents/BODC/Code/ARGODMQC/argodmqc_owc/pyowc/calibration.py", line 664, in calc_piecewisefit pcond_factor[0][calindex] = condslope ValueError: shape mismatch: value array of shape (256,) could not be broadcast to indexing result of shape (511,)

This is as a result of calling fit_cond on line 657 in calibration.py

The condslope variable is returned by this function, and I've examined the variables going into the function, these were (x, y, err, & covariance) and the shapes of these are all as expected (511). This has led me to believe that this function is constructing condslope incorrectly, and causing the error we're seeing.

condslope is derived in a fairly complicated way, and a brief summary of its journey is as follows.

1. E = np.zeros((xfit.len(), best))
2. yfit = np.dot(E,A)
3. condslope = yfit.T

The source of the issue is how this E is defined, specifically the xfit.__len__()

This is done as such: xfit = np.unique(x)

print(x.shape, xfit.shape)
(10, 511) (256,)

I believe this may be the source of the issue.

to add some extra context, trying this with a different float (6901207), I see the following:

print(x.shape, xfit.shape)
(10, 93) (93,)

[jamclar]

I've discovered why this issue is occuring, and the issue is with the mat file.

Inspecting the mat file I see the following:

Name              Size              Bytes  Class     Attributes

  DATES             1x511              4088  double
  LAT               1x511              4088  double
  LONG              1x511              4088  double
  PRES            200x511            408800  single
  **PROFILE_NO        1x511               511  uint8**
  PTMP            200x511            817600  double
  SAL             200x511            817600  double
  TEMP            200x511            408800  single

I've highlighted the issue, which is PROFILE_NO, being uint8.

Because its only 8 bit, any numbers > 255 are overflowing, so therefore the array of numbers looks like:

X: [  1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17  18
  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36
  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54
  55  56  57  58  59  60  61  62  63  64  65  66  67  68  69  70  71  72
  73  74  75  76  77  78  79  80  81  82  83  84  85  86  87  88  89  90
  91  92  93  94  95  96  97  98  99 100 101 102 103 104 105 106 107 108
 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126
 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144
 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162
 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180
 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198
 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216
 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234
 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252
 253 254 255   0   1   2   3   4   5   6   7   8   9  10  11  12  13  14
  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32
  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50
  51  52  53  54  55  56  57  58  59  60  61  62  63  64  65  66  67  68
  69  70  71  72  73  74  75  76  77  78  79  80  81  82  83  84  85  86
  87  88  89  90  91  92  93  94  95  96  97  98  99 100 101 102 103 104
 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122
 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140
 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158
 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176
 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194
 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212
 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230
 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248
 249 250 251 252 253 254 255]

So when the code takes only the unique values from x, there aren't 511 of them. I've tried running this code with another float with ~400 profiles with the PROFILE_NO of double, rather than uint8, and the issue no longer persists.

@Hanyuan-Liu how are you producing these mat files?

[Hanyuan-Liu]

@jamclar Thank you for the investigation!
So for the float.mat file, we use argopy's xarray.argo.create_float_source to create it. I wonder if it is because we were using an older version (0.1.12) of argopy.

For pyOWC, do you recommend always using the lastest argopy with it? We use the legacy version as later versions don't support data fetcher from localftp anymore.

[gmaze]

Hi,
we did not modified xarray.argo.create_float_source for a while, although @kamwal reported minor issues with it (euroargodev/argopy#218, euroargodev/argopy#219)

I can have a look to the output with float 5906393

[gmaze]

Indeed the argopy code export PROFILE_NO as uint8

it's happening here: https://github.com/euroargodev/argopy/blob/4dfec15c444b07777beb2bf795305b981d908ba8/argopy/xarray.py#L1665

I can change this next week if you wish

[jamclar]

Hi @gmaze , thanks for finding that! I suspect a fix to that line would rectify the issue.

Many thanks

James-

[gmaze]

@jamclar @jamclar @kamwal @cabanesc : what would the most appropriate encoding data type for this variable then ?

I suspect I did used uint8 because that's what I found in the Matlab version, or files.

[jamclar]

@gmaze

I think using uint32 would be more than enough.

[gmaze]

so let's try to do no make this unnecessary large and use uint16,
we don't expect argo float to report more than 65535 profiles any time soon, right ?