Values loaded by NCDatasets backend with "_Unsigned" CF attribute don't correspond to Xarray

[asinghvi17]

MWE:

using FilePathsBase, AWSS3
download(p"s3://noaa-goes16/ABI-L2-SSTF/2020/210/00/OR_ABI-L2-SSTF-M6_G16_s20202100000205_e20202100059513_c20202100105456.nc", "noaa.nc")
using Rasters
r1 = Raster("noaa.nc"; lazy = true)
heatmap(r1)

heatmap(read(r1))

None of this agrees with what xarray loads:

but it looks like all CF conventions are applied...

[rafaqz]

Those just look like different colour schemes?

(Otherwise, to be clear on main NCDatasets.jl is still handling CF conversions, not Rasters)

The first one should call read in the Rasters recipe? Does it even hit that?

[asinghvi17]

No the color scheme is the same, the color limits are different though and it seems the numbers are different as well.

[asinghvi17]

Plus you can see the structure is totally strange before read

[rafaqz]

Yes read is clearly a problem. The dispatch is wrong somewhere it looks like broken iteration in DiskArrays.

But the non lazy plots without noise are just NCDatasets.jl vs XArray, Rasters is not affecting the values

[asinghvi17]

aha ok, will check out what's going on there. This is on the cf branch so hopefully it's not ignoring some obscure CF convention...

[rafaqz]

Makes way more sense this is on cf. You gave me a fright that that mess is from main...

It most likely is ignoring or getting some CF thing wrong

[asinghvi17]

FWIW I get this on main too

[rafaqz]

Both problems or just one. Like if it's both Rasters and NCDatasets CF outputs are the same on main and cf, then maybe it's Xarray... Does it even do CF scaling by default

I think cf_xarray is a separate python package, its not in xarray proper?

[rafaqz]

The DiskArrays munging I will look at tonight, seems bad if that's on main

[asinghvi17]

Both unfortunately. Yes it looks like Xarray is doing CF scaling, and the dataset is sea surface temperature, so the Xarray data range makes way more sense than the Rasters/NCDatasets one.

Left is xarray collected into a Julia Matrix, right is rasters.

[felixcremer]

Can you put the data somewhere I don't need an AWS account to download?

[asinghvi17]

You don't for this, it's public access.

[felixcremer]

I get a AWS Credentials not found error when I try your reproducer.

julia> download(p"s3://noaa-goes16/ABI-L2-SSTF/2020/210/00/OR_ABI-L2-SSTF-M6_G16_s20202100000205_e20202100059513_c20202100105456.nc", "noaa.nc")
ERROR: Can't find AWS credentials!

Stacktrace:
  [1] AWS.AWSCredentials(; profile::Nothing, throw_cred_error::Bool)
    @ AWS ~/.julia/packages/AWS/SqWtW/src/AWSCredentials.jl:145
  [2] AWSCredentials
    @ ~/.julia/packages/AWS/SqWtW/src/AWSCredentials.jl:113 [inlined]
  [3] AWS.AWSConfig()
    @ AWS ~/.julia/packages/AWS/SqWtW/src/AWSConfig.jl:32
  [4] #global_aws_config#91
    @ ~/.julia/packages/AWS/SqWtW/src/AWS.jl:81 [inlined]
  [5] global_aws_config
    @ ~/.julia/packages/AWS/SqWtW/src/AWS.jl:79 [inlined]
  [6] get_config
    @ ~/.julia/packages/AWSS3/8cxdr/src/s3path.jl:143 [inlined]
  [7] exists(fp::S3Path{Nothing})
    @ AWSS3 ~/.julia/packages/AWSS3/8cxdr/src/s3path.jl:244
  [8] cp(src::S3Path{Nothing}, dst::PosixPath; force::Bool, follow_symlinks::Bool)
    @ FilePathsBase ~/.julia/packages/FilePathsBase/Laclt/src/path.jl:560
  [9] cp
    @ ~/.julia/packages/FilePathsBase/Laclt/src/path.jl:551 [inlined]
 [10] download(url::S3Path{Nothing}, localfile::PosixPath)
    @ FilePathsBase ~/.julia/packages/FilePathsBase/Laclt/src/path.jl:697
 [11] download(url::S3Path{Nothing}, localfile::String)
    @ FilePathsBase ~/.julia/packages/FilePathsBase/Laclt/src/path.jl:700
 [12] top-level scope
    @ REPL[5]:1

[asinghvi17]

Huh! Here's a google drive link:

https://drive.google.com/file/d/1Om08vXhgsmLbcGmbEFaachfBHa6Nq5pA/view?usp=sharing

[felixcremer]

I can't reproduce your garbled up lazy raster plotting.
I get the same plot for r1 and read(r1)