-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Integration of CommonDataModel sources #416
Conversation
Awesome! I would really like to get it working (properly) with |
For GRIBDataset, lazystack= RasterStack(era5; lazy=true)
read(lazystack)
ERROR: MethodError: no method matching _maybestack(::RasterStack{Rasters.OpenStack{Rasters.GRIBfile, (:z, :t), GRIBDataset{Float64, 5}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMfile, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMfile, Dict{String, Any}}, Metadata{CDMfile, Dict{String, Any}}}}, Missing}, ::Tuple{Raster{Float64, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, Array{Float64, 5}, Symbol, Metadata{CDMfile, Dict{String, Any}}, Missing}, Raster{Float64, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, Array{Float64, 5}, Symbol, Metadata{CDMfile, Dict{String, Any}}, Missing}})
Closest candidates are:
_maybestack(::AbstractDimStack{<:NamedTuple{K}}, ::Tuple{AbstractDimArray, Vararg{AbstractDimArray}}) where K at ~/.julia/packages/DimensionalData/7iYMt/src/stack/methods.jl:72
_maybestack(::AbstractDimStack{<:NamedTuple{K}}, ::Tuple) where K at ~/.julia/packages/DimensionalData/7iYMt/src/stack/methods.jl:69
Stacktrace:
[1] map(f::Function, s::RasterStack{Rasters.OpenStack{Rasters.GRIBfile, (:z, :t), GRIBDataset{Float64, 5}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMfile, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMfile, Dict{String, Any}}, Metadata{CDMfile, Dict{String, Any}}}}, Missing})
@ DimensionalData ~/.julia/packages/DimensionalData/7iYMt/src/stack/methods.jl:51
[2] (::Rasters.var"#97#99"{UnionAll})(o::RasterStack{Rasters.OpenStack{Rasters.GRIBfile, (:z, :t), GRIBDataset{Float64, 5}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMfile, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMfile, Dict{String, Any}}, Metadata{CDMfile, Dict{String, Any}}}}, Missing})
@ Rasters ~/.julia/dev/Rasters/src/stack.jl:326
[3] open(f::Rasters.var"#97#99"{UnionAll}, st::RasterStack{Rasters.FileStack{Rasters.GRIBfile, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMfile, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMfile, Dict{String, Any}}, Metadata{CDMfile, Dict{String, Any}}}}, Missing}; kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Rasters ~/.julia/dev/Rasters/src/stack.jl:333
[4] open(f::Function, st::RasterStack{Rasters.FileStack{Rasters.GRIBfile, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMfile, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMfile, Dict{String, Any}}, Metadata{CDMfile, Dict{String, Any}}}}, Missing})
@ Rasters ~/.julia/dev/Rasters/src/stack.jl:332
[5] modify(f::Type, s::RasterStack{Rasters.FileStack{Rasters.GRIBfile, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMfile, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMfile, Dict{String, Any}}, Metadata{CDMfile, Dict{String, Any}}}}, Missing})
@ Rasters ~/.julia/dev/Rasters/src/stack.jl:325
[6] read(x::RasterStack{Rasters.FileStack{Rasters.GRIBfile, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMfile, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMfile, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMfile, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMfile, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMfile, Dict{String, Any}}, Metadata{CDMfile, Dict{String, Any}}}}, Missing})
@ Rasters ~/.julia/dev/Rasters/src/read.jl:9 |
Ahh this is just a DimensionalData.jl bug im in the process of fixing. Downgrade DD to 0.24.4 or thereabouts for now. Also, good to set (All the tests should actually have |
Conflicts: src/Rasters.jl src/convenience.jl src/sources/commondatamodel.jl
I updated according to #421, thanks for noticing me! julia> read(lazystack)
ERROR: No dims found
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:33
[2] dims(#unused#::Nothing)
@ DimensionalData.Dimensions ~/.julia/packages/DimensionalData/bJAtd/src/Dimensions/dimension.jl:169
[3] map
@ ./tuple.jl:221 [inlined]
[4] combinedims(xs::Nothing; kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ DimensionalData.Dimensions ~/.julia/packages/DimensionalData/bJAtd/src/Dimensions/primitives.jl:528
[5] combinedims
@ ~/.julia/packages/DimensionalData/bJAtd/src/Dimensions/primitives.jl:528 [inlined]
[6] #combinedims#45
@ ~/.julia/packages/DimensionalData/bJAtd/src/Dimensions/primitives.jl:528 [inlined]
[7] combinedims
@ ~/.julia/packages/DimensionalData/bJAtd/src/Dimensions/primitives.jl:528 [inlined]
[8] (::Rasters.var"#71#72"{NamedTuple{(:z, :t), Tuple{Raster{Union{Missing, Float64}, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, Array{Union{Missing, Float64}, 5}, Symbol, Metadata{CDMsource, Dict{String, Any}}, Missing}, Raster{Union{Missing, Float64}, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, Array{Union{Missing, Float64}, 5}, Symbol, Metadata{CDMsource, Dict{String, Any}}, Missing}}}})()
@ Rasters ~/.julia/dev/Rasters/src/stack.jl:88
[9] #invokelatest#2
@ ./essentials.jl:716 [inlined]
[10] invokelatest
@ ./essentials.jl:714 [inlined]
[11] rebuild_from_arrays(s::RasterStack{Rasters.OpenStack{Rasters.GRIBsource, (:z, :t), GRIBDataset{Float64, 5}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing}, das::NamedTuple{(:z, :t), Tuple{Raster{Union{Missing, Float64}, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, Array{Union{Missing, Float64}, 5}, Symbol, Metadata{CDMsource, Dict{String, Any}}, Missing}, Raster{Union{Missing, Float64}, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, Array{Union{Missing, Float64}, 5}, Symbol, Metadata{CDMsource, Dict{String, Any}}, Missing}}}; data::NamedTuple{(:z, :t), Tuple{Array{Union{Missing, Float64}, 5}, Array{Union{Missing, Float64}, 5}}}, refdims::Tuple{}, metadata::Metadata{CDMsource, Dict{String, Any}}, dims::Nothing, layerdims::NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, layermetadata::NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, missingval::NamedTuple{(:z, :t), Tuple{Missing, Missing}})
@ Rasters ~/.julia/dev/Rasters/src/stack.jl:87
[12] rebuild_from_arrays
@ ~/.julia/dev/Rasters/src/stack.jl:85 [inlined]
[13] _maybestack(s::RasterStack{Rasters.OpenStack{Rasters.GRIBsource, (:z, :t), GRIBDataset{Float64, 5}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing}, das::NamedTuple{(:z, :t), Tuple{Raster{Union{Missing, Float64}, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, Array{Union{Missing, Float64}, 5}, Symbol, Metadata{CDMsource, Dict{String, Any}}, Missing}, Raster{Union{Missing, Float64}, 5, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, Array{Union{Missing, Float64}, 5}, Symbol, Metadata{CDMsource, Dict{String, Any}}, Missing}}})
@ DimensionalData ~/.julia/packages/DimensionalData/bJAtd/src/stack/methods.jl:99
[14] map(f::Function, s::RasterStack{Rasters.OpenStack{Rasters.GRIBsource, (:z, :t), GRIBDataset{Float64, 5}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing})
@ DimensionalData ~/.julia/packages/DimensionalData/bJAtd/src/stack/methods.jl:50
[15] (::Rasters.var"#98#100"{UnionAll})(o::RasterStack{Rasters.OpenStack{Rasters.GRIBsource, (:z, :t), GRIBDataset{Float64, 5}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing})
@ Rasters ~/.julia/dev/Rasters/src/stack.jl:316
[16] open(f::Rasters.var"#98#100"{UnionAll}, st::RasterStack{Rasters.FileStack{Rasters.GRIBsource, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing}; kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Rasters ~/.julia/dev/Rasters/src/stack.jl:323
[17] open(f::Function, st::RasterStack{Rasters.FileStack{Rasters.GRIBsource, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing})
@ Rasters ~/.julia/dev/Rasters/src/stack.jl:322
[18] modify(f::Type, s::RasterStack{Rasters.FileStack{Rasters.GRIBsource, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing})
@ Rasters ~/.julia/dev/Rasters/src/stack.jl:315
[19] read(x::RasterStack{Rasters.FileStack{Rasters.GRIBsource, (:z, :t), String, Tuple{Union, Union}, Tuple{NTuple{5, Int64}, NTuple{5, Int64}}, Tuple{DiskArrays.GridChunks{5}, DiskArrays.GridChunks{5}}, Tuple{DiskArrays.Unchunked, DiskArrays.Unchunked}}, Tuple{X{Mapped{Float64, Vector{Float64}, ForwardOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float64, Vector{Float64}, ReverseOrdered, Regular{Float64}, Points, Metadata{CDMsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Z{Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Dim{:number, Sampled{Int64, Vector{Int64}, ForwardOrdered, Regular{Int64}, Points, Metadata{CDMsource, Dict{String, Any}}}}, Ti{Sampled{DateTime, Vector{DateTime}, ForwardOrdered, Irregular{Tuple{Nothing, Nothing}}, Points, Metadata{CDMsource, Dict{String, Any}}}}}, Tuple{}, NamedTuple{(:z, :t), Tuple{Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}, Tuple{X{Colon}, Y{Colon}, Z{Colon}, Dim{:number, Colon}, Ti{Colon}}}}, Metadata{CDMsource, Dict{String, Any}}, NamedTuple{(:z, :t), Tuple{Metadata{CDMsource, Dict{String, Any}}, Metadata{CDMsource, Dict{String, Any}}}}, Missing})
@ Rasters ~/.julia/dev/Rasters/src/read.jl:9 |
This will give a different error on the latests DD. But Im guessing something is missing a |
Thanks, it indeed works with the latest commit of DD (after a minor fix)! |
src/Rasters.jl src/sources/commondatamodel.jl test/runtests.jl
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #416 +/- ##
==========================================
- Coverage 80.99% 80.93% -0.06%
==========================================
Files 58 61 +3
Lines 4204 4218 +14
==========================================
+ Hits 3405 3414 +9
- Misses 799 804 +5 ☔ View full report in Codecov by Sentry. |
Looks good! but we probably need to define the chunking? |
Hey @tcarion where are we at with merging this now we have DiskArrays methods for I'm sorry this has all taken so long, I can help get this over the line if there are any remaining issues. |
Conflicts: Project.toml ext/RastersNCDatasetsExt/ncdatasets_source.jl
Since NCDatasets has been yanked, the tests are failing here, but it works locally with v0.13. If I get it well, the issue Alexander-Barth/NCDatasets.jl#230 is solved thanks to JuliaIO/DiskArrays.jl#124, is that right @Alexander-Barth ? If so, the tests should pass when we're back at v0.13 (I hope) |
It could be unyanked now that DiskArrays is fixed, @Alexander-Barth |
I just released NCDatasets 0.13.1 with DiskArray support. |
Yes that would be good to have! |
Conflicts: Project.toml
@tcarion would it help if I finished this off? there have been a bunch of changes since this branched |
Conflicts: Project.toml test/sources/ncdatasets.jl
@rafaqz Do you think this can be merged now ? The tests won't pass on Windows, which is normal since GRIB.jl doesn't support Windows for now |
Everything has to pass to be merged, but that may just mean skipping the grib tests on windows with a condition in the runtests.jl file. But yes otherwise this looks really good. There are also failing documenter tests but that looks like something different. |
Ok I'm going to merge and fix the makie documentation errors later. Thanks so much for pushing this through!! |
Note that DiskArrays doesn't actually work with this PR. chunks have to be forwarded down to the wrapped array that knows about them, and CommonDataModel.jl doesn't. @Alexander-Barth CommonDataModel.jl variables not being disk arrays is really killing us here, the complexity of making it actually work with hacks is a huge overhead I would rather not deal with. |
This should allow reading any dataset that implements CommonDataModel.jl. It would solve #282