Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration of CommonDataModel sources #416

Merged
merged 23 commits into from
Jan 18, 2024
Merged

Conversation

tcarion
Copy link
Contributor

@tcarion tcarion commented Apr 14, 2023

This should allow reading any dataset that implements CommonDataModel.jl. It would solve #282

@rafaqz
Copy link
Owner

rafaqz commented Apr 14, 2023

Awesome!

I would really like to get it working (properly) with lazy=true, which will need changes to CommonDataModel.jl

JuliaGeo/CommonDataModel.jl#8

@tcarion
Copy link
Contributor Author

tcarion commented Apr 14, 2023

For GRIBDataset, lazy = true on RasterStack works if you consider the PR JuliaGeo/CommonDataModel.jl#9. However, it fails with lazy = false! It actually fails when using the read method on RasterStack's (see below). I'm trying to understand why.

lazystack= RasterStack(era5; lazy=true)
read(lazystack)
ERROR: MethodError: no method matching _maybestack(::RasterStack{Rasters.OpenStack{Rasters.GRIBfile, (:z, :t), GRIBDataset{Float64, 5}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMfile, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMfile, Dict{String, Any}}, Metadata{CDMfile, Dict{String, Any}}}}, Missing}, ::Tuple{Raster{Float64, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, Array{Float64, 5}, Symbol, Metadata{CDMfile, Dict{String, Any}}, Missing}, Raster{Float64, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, Array{Float64, 5}, Symbol, Metadata{CDMfile, Dict{String, Any}}, Missing}})
Closest candidates are:
  _maybestack(::AbstractDimStack{<:NamedTuple{K}}, ::Tuple{AbstractDimArray, Vararg{AbstractDimArray}}) where K at ~/.julia/packages/DimensionalData/7iYMt/src/stack/methods.jl:72
  _maybestack(::AbstractDimStack{<:NamedTuple{K}}, ::Tuple) where K at ~/.julia/packages/DimensionalData/7iYMt/src/stack/methods.jl:69
Stacktrace:
 [1] map(f::Function, s::RasterStack{Rasters.OpenStack{Rasters.GRIBfile, (:z, :t), GRIBDataset{Float64, 5}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMfile, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMfile, Dict{String, Any}}, Metadata{CDMfile, Dict{String, Any}}}}, Missing})
   @ DimensionalData ~/.julia/packages/DimensionalData/7iYMt/src/stack/methods.jl:51
 [2] (::Rasters.var"#97#99"{UnionAll})(o::RasterStack{Rasters.OpenStack{Rasters.GRIBfile, (:z, :t), GRIBDataset{Float64, 5}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMfile, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMfile, Dict{String, Any}}, Metadata{CDMfile, Dict{String, Any}}}}, Missing})
   @ Rasters ~/.julia/dev/Rasters/src/stack.jl:326
 [3] open(f::Rasters.var"#97#99"{UnionAll}, st::RasterStack{Rasters.FileStack{Rasters.GRIBfile, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMfile, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMfile, Dict{String, Any}}, Metadata{CDMfile, Dict{String, Any}}}}, Missing}; kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Rasters ~/.julia/dev/Rasters/src/stack.jl:333
 [4] open(f::Function, st::RasterStack{Rasters.FileStack{Rasters.GRIBfile, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMfile, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMfile, Dict{String, Any}}, Metadata{CDMfile, Dict{String, Any}}}}, Missing})
   @ Rasters ~/.julia/dev/Rasters/src/stack.jl:332
 [5] modify(f::Type, s::RasterStack{Rasters.FileStack{Rasters.GRIBfile, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMfile, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMfile, Dict{String, Any}}, Metadata{CDMfile, Dict{String, Any}}}}, Missing})
   @ Rasters ~/.julia/dev/Rasters/src/stack.jl:325
 [6] read(x::RasterStack{Rasters.FileStack{Rasters.GRIBfile, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMfile, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMfile, Dict{String, Any}}, Metadata{CDMfile, Dict{String, Any}}}}, Missing})
   @ Rasters ~/.julia/dev/Rasters/src/read.jl:9

@rafaqz
Copy link
Owner

rafaqz commented Apr 14, 2023

Ahh this is just a DimensionalData.jl bug im in the process of fixing.

Downgrade DD to 0.24.4 or thereabouts for now.

Also, good to set DiskArrays.allow_scalar(false) to make sure it does really work. Then test some broadcasts with read or indexing afterwards.

(All the tests should actually have allow_scalar set to false but they were written befire I added it to DiskArrays)

Conflicts:
	src/Rasters.jl
	src/convenience.jl
	src/sources/commondatamodel.jl
@tcarion
Copy link
Contributor Author

tcarion commented Apr 17, 2023

I updated according to #421, thanks for noticing me!
Using [email protected], it goes further but still fails:

julia> read(lazystack)
ERROR: No dims found
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] dims(#unused#::Nothing)
    @ DimensionalData.Dimensions ~/.julia/packages/DimensionalData/bJAtd/src/Dimensions/dimension.jl:169
  [3] map
    @ ./tuple.jl:221 [inlined]
  [4] combinedims(xs::Nothing; kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ DimensionalData.Dimensions ~/.julia/packages/DimensionalData/bJAtd/src/Dimensions/primitives.jl:528
  [5] combinedims
    @ ~/.julia/packages/DimensionalData/bJAtd/src/Dimensions/primitives.jl:528 [inlined]
  [6] #combinedims#45
    @ ~/.julia/packages/DimensionalData/bJAtd/src/Dimensions/primitives.jl:528 [inlined]
  [7] combinedims
    @ ~/.julia/packages/DimensionalData/bJAtd/src/Dimensions/primitives.jl:528 [inlined]
  [8] (::Rasters.var"#71#72"{NamedTuple{(:z, :t), Tuple{Raster{Union{Missing, Float64}, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, Array{Union{Missing, Float64}, 5}, Symbol, Metadata{CDMsource, Dict{String, Any}}, Missing}, Raster{Union{Missing, Float64}, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, Array{Union{Missing, Float64}, 5}, Symbol, Metadata{CDMsource, Dict{String, Any}}, Missing}}}})()
    @ Rasters ~/.julia/dev/Rasters/src/stack.jl:88
  [9] #invokelatest#2
    @ ./essentials.jl:716 [inlined]
 [10] invokelatest
    @ ./essentials.jl:714 [inlined]
 [11] rebuild_from_arrays(s::RasterStack{Rasters.OpenStack{Rasters.GRIBsource, (:z, :t), GRIBDataset{Float64, 5}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing}, das::NamedTuple{(:z, :t), Tuple{Raster{Union{Missing, Float64}, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, Array{Union{Missing, Float64}, 5}, Symbol, Metadata{CDMsource, Dict{String, Any}}, Missing}, Raster{Union{Missing, Float64}, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, Array{Union{Missing, Float64}, 5}, Symbol, Metadata{CDMsource, Dict{String, Any}}, Missing}}}; data::NamedTuple{(:z, :t), Tuple{Array{Union{Missing, Float64}, 5}, Array{Union{Missing, Float64}, 5}}}, refdims::Tuple{}, metadata::Metadata{CDMsource, Dict{String, Any}}, dims::Nothing, layerdims::NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, layermetadata::NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, missingval::NamedTuple{(:z, :t), Tuple{Missing, Missing}})
    @ Rasters ~/.julia/dev/Rasters/src/stack.jl:87
 [12] rebuild_from_arrays
    @ ~/.julia/dev/Rasters/src/stack.jl:85 [inlined]
 [13] _maybestack(s::RasterStack{Rasters.OpenStack{Rasters.GRIBsource, (:z, :t), GRIBDataset{Float64, 5}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing}, das::NamedTuple{(:z, :t), Tuple{Raster{Union{Missing, Float64}, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, Array{Union{Missing, Float64}, 5}, Symbol, Metadata{CDMsource, Dict{String, Any}}, Missing}, Raster{Union{Missing, Float64}, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, Array{Union{Missing, Float64}, 5}, Symbol, Metadata{CDMsource, Dict{String, Any}}, Missing}}})
    @ DimensionalData ~/.julia/packages/DimensionalData/bJAtd/src/stack/methods.jl:99
 [14] map(f::Function, s::RasterStack{Rasters.OpenStack{Rasters.GRIBsource, (:z, :t), GRIBDataset{Float64, 5}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing})
    @ DimensionalData ~/.julia/packages/DimensionalData/bJAtd/src/stack/methods.jl:50
 [15] (::Rasters.var"#98#100"{UnionAll})(o::RasterStack{Rasters.OpenStack{Rasters.GRIBsource, (:z, :t), GRIBDataset{Float64, 5}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing})
    @ Rasters ~/.julia/dev/Rasters/src/stack.jl:316
 [16] open(f::Rasters.var"#98#100"{UnionAll}, st::RasterStack{Rasters.FileStack{Rasters.GRIBsource, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing}; kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Rasters ~/.julia/dev/Rasters/src/stack.jl:323
 [17] open(f::Function, st::RasterStack{Rasters.FileStack{Rasters.GRIBsource, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing})
    @ Rasters ~/.julia/dev/Rasters/src/stack.jl:322
 [18] modify(f::Type, s::RasterStack{Rasters.FileStack{Rasters.GRIBsource, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing})
    @ Rasters ~/.julia/dev/Rasters/src/stack.jl:315
 [19] read(x::RasterStack{Rasters.FileStack{Rasters.GRIBsource, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing})
    @ Rasters ~/.julia/dev/Rasters/src/read.jl:9

@rafaqz
Copy link
Owner

rafaqz commented Apr 17, 2023

This will give a different error on the latests DD.

But Im guessing something is missing a dims method? You are calling combinedims on nothing

@tcarion
Copy link
Contributor Author

tcarion commented Apr 18, 2023

Thanks, it indeed works with the latest commit of DD (after a minor fix)!
So GRIBDataset works, next is to fix it for NCDataset, which I'll work on once Alexander-Barth/NCDatasets.jl#205 is done

@codecov-commenter
Copy link

codecov-commenter commented Jul 5, 2023

Codecov Report

Attention: 29 lines in your changes are missing coverage. Please review.

Comparison is base (c63af55) 80.99% compared to head (a453b21) 80.93%.

❗ Current head a453b21 differs from pull request most recent head 4f38942. Consider uploading reports for the commit 4f38942 to get more accurate results

Files Patch % Lines
src/sources/commondatamodel.jl 87.39% 29 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #416      +/-   ##
==========================================
- Coverage   80.99%   80.93%   -0.06%     
==========================================
  Files          58       61       +3     
  Lines        4204     4218      +14     
==========================================
+ Hits         3405     3414       +9     
- Misses        799      804       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tcarion
Copy link
Contributor Author

tcarion commented Jul 7, 2023

@rafaqz, I tried to implement your idea here. Can you confirm that's what you had in mind?
The tests are passing locally with the DiskArray branch of NCDatasets.

If it's ok for you, I'll look into the grow! method.

@rafaqz
Copy link
Owner

rafaqz commented Jul 7, 2023

Looks good! but we probably need to define the chunking?

@rafaqz
Copy link
Owner

rafaqz commented Sep 29, 2023

Hey @tcarion where are we at with merging this now we have DiskArrays methods for NCDatasets.Variable ?

I'm sorry this has all taken so long, I can help get this over the line if there are any remaining issues.

Conflicts:
	Project.toml
	ext/RastersNCDatasetsExt/ncdatasets_source.jl
@tcarion
Copy link
Contributor Author

tcarion commented Oct 9, 2023

Since NCDatasets has been yanked, the tests are failing here, but it works locally with v0.13. If I get it well, the issue Alexander-Barth/NCDatasets.jl#230 is solved thanks to JuliaIO/DiskArrays.jl#124, is that right @Alexander-Barth ? If so, the tests should pass when we're back at v0.13 (I hope)

@rafaqz
Copy link
Owner

rafaqz commented Oct 9, 2023

It could be unyanked now that DiskArrays is fixed, @Alexander-Barth

@Alexander-Barth
Copy link

Alexander-Barth commented Oct 10, 2023

I just released NCDatasets 0.13.1 with DiskArray support.
Note at some point I would like to have a ZarrDatasets package implementing the CommonDataModel.jl interface for zarr files (based on Zarr.jl).
Maybe this will also be helpful for Rasters.jl package or does it already have zarr support by other means?

@rafaqz
Copy link
Owner

rafaqz commented Oct 10, 2023

Yes that would be good to have!

@tcarion tcarion marked this pull request as ready for review October 18, 2023 08:03
@rafaqz
Copy link
Owner

rafaqz commented Jan 1, 2024

@tcarion would it help if I finished this off? there have been a bunch of changes since this branched

Conflicts:
	Project.toml
	test/sources/ncdatasets.jl
@tcarion
Copy link
Contributor Author

tcarion commented Jan 18, 2024

@rafaqz Do you think this can be merged now ? The tests won't pass on Windows, which is normal since GRIB.jl doesn't support Windows for now

@rafaqz
Copy link
Owner

rafaqz commented Jan 18, 2024

Everything has to pass to be merged, but that may just mean skipping the grib tests on windows with a condition in the runtests.jl file. But yes otherwise this looks really good.

There are also failing documenter tests but that looks like something different.

@rafaqz
Copy link
Owner

rafaqz commented Jan 18, 2024

Ok I'm going to merge and fix the makie documentation errors later.

Thanks so much for pushing this through!!

@rafaqz rafaqz merged commit a913761 into rafaqz:main Jan 18, 2024
5 of 7 checks passed
@rafaqz
Copy link
Owner

rafaqz commented Jan 19, 2024

Note that DiskArrays doesn't actually work with this PR. chunks have to be forwarded down to the wrapped array that knows about them, and CommonDataModel.jl doesn't.

@Alexander-Barth CommonDataModel.jl variables not being disk arrays is really killing us here, the complexity of making it actually work with hacks is a huge overhead I would rather not deal with.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants