Writing a String always prints a warning

[johnomotani]

Describe the bug

If a String variable is written using NCDatasets, a warning is printed.

The warning makes sense as String is a variable-length data type, but as there does not seem to be anything to do about it, it is unhelpful.

The warning comes from DiskArrays.jl, but posting here as I'm guessing that the solution may be some sort of special handling for String here. Maybe a specialised version of setindex_disk!(), like

function setindex_disk!(a::NCDatasets.Variable{String, 0, NCDataset{Nothing, Missing}}, v::AbstractArray, i...)
    setindex_disk_nobatch!(a,v,i)
    v   
end

?

To Reproduce

using NCDatasets
f = NCDataset("test.nc", "c")
v = defVar(f, "foo", String, ())
v[] = "bar"

prints

┌ Warning: Can not determine size of element type. Using DiskArrays.fallback_element_size[] = 100 bytes
└ @ DiskArrays ~/.julia/packages/DiskArrays/ny95C/src/chunks.jl:354
┌ Warning: Can not determine size of element type. Using DiskArrays.fallback_element_size[] = 100 bytes
└ @ DiskArrays ~/.julia/packages/DiskArrays/ny95C/src/chunks.jl:354 

Expected behavior

The string is written with no warning.

Environment

• operating system: Linux Mint 21.1
• Julia version: julia 1.10.6, official binaries from https://julialang.org/downloads/
• Output of the julia command versioninfo()

  Julia Version 1.10.6
  Commit 67dffc4a8ae (2024-10-28 12:23 UTC)
  Build Info:
    Official https://julialang.org/ release
  Platform Info:
    OS: Linux (x86_64-linux-gnu)
    CPU: 16 × 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
    WORD_SIZE: 64
    LIBM: libopenlibm
    LLVM: libLLVM-15.0.7 (ORCJIT, tigerlake)
  Threads: 1 default, 0 interactive, 1 GC (on 16 virtual cores)
  

• NCDatasets version: v0.14.6
• Output of using Pkg; Pkg.status(mode=PKGMODE_MANIFEST)

Status `~/physics/moment_kinetics-master-clean/Manifest.toml`
  [54578032] ADNLPModels v0.8.7
  [47edcb42] ADTypes v1.9.0
  [14f7f29c] AMD v0.5.3
  [621f4979] AbstractFFTs v1.5.0
  [1520ce14] AbstractTrees v0.4.5
  [7d9f7c33] Accessors v0.1.38
⌃ [79e6a3ab] Adapt v4.1.0
  [66dad0bd] AliasTables v1.1.3
  [9b6a8646] AllocCheck v0.2.0
  [c7e460c6] ArgParse v1.2.0
⌃ [4fba245c] ArrayInterface v7.16.0
  [13072b0f] AxisAlgorithms v1.1.0
  [62783981] BitTwiddlingConvenienceFunctions v0.1.6
  [fa961155] CEnum v0.5.0
  [179af706] CFTime v0.1.3
  [2a0fbf3d] CPUSummary v0.2.6
  [d360d2e6] ChainRulesCore v1.25.0
  [fb6a15b2] CloseOpenIntervals v0.1.13
  [da1fd8a2] CodeTracking v1.3.6
⌅ [3da002f7] ColorTypes v0.11.5
⌅ [5ae59095] Colors v0.12.11
  [861a8166] Combinatorics v1.0.2
  [1fbeeb36] CommonDataModel v0.3.7
  [38540f10] CommonSolve v0.2.4
  [bbf7d656] CommonSubexpressions v0.3.1
  [34da2185] Compat v4.16.0
  [a33af91c] CompositionsBase v0.1.2
  [187b0558] ConstructionBase v1.5.8
  [adafc99b] CpuId v0.3.1
⌃ [f68482b8] Cthulhu v2.15.2
  [9a962f9c] DataAPI v1.16.0
  [864edb3b] DataStructures v0.18.20
  [163ba53b] DiffResults v1.1.0
  [b552c78f] DiffRules v1.15.1
  [3c3547ce] DiskArrays v0.4.6
⌃ [31c24e10] Distributions v0.25.112
  [ffbed154] DocStringExtensions v0.9.3
  [e2ba6199] ExprTools v0.1.10
  [7a1cc6ca] FFTW v1.8.0
  [9aa1b823] FastClosures v0.3.2
  [442a2c76] FastGaussQuadrature v1.0.2
  [5789e2e9] FileIO v1.16.4
  [1a297f60] FillArrays v1.13.0
  [6a86dc24] FiniteDiff v2.26.0
  [53c48c17] FixedPointNumbers v0.8.5
⌃ [08572546] FlameGraphs v1.0.0
  [1eca21be] FoldingTrees v1.2.1
⌃ [f6369f11] ForwardDiff v0.10.36
  [069b7b12] FunctionWrappers v1.1.3
⌅ [61eb1bfa] GPUCompiler v0.27.8
  [c27321d9] Glob v1.3.1
  [0bc81568] HAML v0.3.5
  [f67ccb44] HDF5 v0.17.2
  [3e5b6fbb] HostCPUFeatures v0.1.17
  [1baab800] HybridArrays v0.4.16
⌃ [34004b35] HypergeometricFunctions v0.3.24
  [615f187c] IfElse v0.1.1
  [9b13fd28] IndirectArrays v1.0.0
  [18e54dd8] IntegerMathUtils v0.1.2
  [a98d9a8b] Interpolations v0.15.1
  [3587e190] InverseFunctions v0.1.17
  [92d709cd] IrrationalConstants v0.2.2
  [692b3bcd] JLLWrappers v1.6.1
  [10dff2fc] JSOSolvers v0.12.1
  [aa1ae85d] JuliaInterpreter v0.9.36
  [70703baa] JuliaSyntax v0.4.10
⌃ [ba0b0d4f] Krylov v0.9.7
  [40e66cde] LDLFactorizations v0.10.1
  [929cbde3] LLVM v9.1.3
  [8ac3fa9e] LRUCache v1.6.1
  [10f19ff3] LayoutPointers v0.1.17
  [1d6d02ad] LeftChildRightSiblingTrees v0.2.0
  [3db4a2ba] LegendrePolynomials v0.4.5
  [5c8ed15e] LinearOperators v2.9.0
  [2ab3a3ac] LogExpFunctions v0.3.28
  [bdcacae8] LoopVectorization v0.12.171
  [6f1432cf] LoweredCodeUtils v3.0.5
  [2fda8390] LsqFit v0.15.0
  [da04e1cc] MPI v0.20.22
  [3da0fdf6] MPIPreferences v0.1.11
  [1914dd2f] MacroTools v0.5.13
  [d125e4d3] ManualMemory v0.1.8
  [442fdcdd] Measures v0.3.2
  [e1d29d7a] Missings v1.2.0
  [85f8d34a] NCDatasets v0.14.6
  [50b5cf9e] NLLSsolver v4.0.0
  [a4795742] NLPModels v0.21.3
  [e01155f1] NLPModelsModifiers v0.7.2
  [d41bc354] NLSolversBase v7.8.3
  [77ba4419] NaNMath v1.0.2
  [6fe1bfb0] OffsetArrays v1.14.1
  [bac558e1] OrderedCollections v1.6.3
  [90014a1f] PDMats v0.11.31
⌃ [9b87118b] PackageCompiler v2.1.21
  [eebad327] PkgVersion v0.3.3
  [1d0040c9] PolyesterWeave v0.2.2
  [aea7be01] PrecompileTools v1.2.1
  [21216c6a] Preferences v1.4.3
  [27ebfcd6] Primes v0.5.6
  [43287f4e] PtrArrays v1.2.1
  [1fd47b50] QuadGK v2.11.1
  [be4d8f0f] Quadmath v0.5.10
  [c84ed2f1] Ratios v0.4.5
  [189a3867] Reexport v1.2.2
  [05181044] RelocatableFolders v1.0.1
  [ae029012] Requires v1.3.0
  [37e2e3b7] ReverseDiff v1.15.3
  [295af30f] Revise v3.6.2
  [79098fc4] Rmath v0.8.0
  [f2b01f46] Roots v2.2.1
  [94e857df] SIMDTypes v0.1.0
  [476501e8] SLEEFPirates v0.6.43
  [6c6a2e73] Scratch v1.2.1
  [efcf1570] Setfield v1.1.1
  [ff4d7338] SolverCore v0.3.8
  [b5612192] SolverTools v0.9.0
  [a2af1166] SortingAlgorithms v1.2.1
  [9f842d2f] SparseConnectivityTracer v0.6.8
⌃ [a0a7dd2c] SparseMatricesCSR v0.6.7
⌃ [0a514795] SparseMatrixColorings v0.4.8
  [276daf66] SpecialFunctions v2.4.0
  [860ef19b] StableRNGs v1.0.2
  [a8a75453] StatProfilerHTML v1.6.0
⌅ [aedffcd0] Static v0.8.10
  [0d7ed370] StaticArrayInterface v1.8.0
  [90137ffa] StaticArrays v1.9.8
  [1e83bf80] StaticArraysCore v1.4.3
  [82ae8749] StatsAPI v1.7.0
  [2913bbd2] StatsBase v0.34.3
  [4c63d2b9] StatsFuns v1.3.2
  [b718987f] TextWrap v1.0.2
  [8290d209] ThreadingUtilities v0.5.2
  [a759f4b9] TimerOutputs v0.5.25
  [d265eb64] TypedSyntax v1.4.2
  [3a884ed6] UnPack v1.0.2
  [1986cc42] Unitful v1.21.0
⌃ [3d5dd08c] VectorizationBase v0.21.70
  [b8c1c048] WidthLimitedIO v1.0.1
  [efce3f68] WoodburyMatrices v1.0.0
  [b5ff72cc] moment_kinetics v0.1.0 `moment_kinetics`
  [0b7ba130] Blosc_jll v1.21.6+0
  [6e34b625] Bzip2_jll v1.0.8+2
  [f5851436] FFTW_jll v3.3.10+1
  [0951126a] GnuTLS_jll v3.8.4+0
⌃ [0234f1f7] HDF5_jll v1.14.2+1
⌃ [e33a78d0] Hwloc_jll v2.11.2+0
  [1d5cc7b8] IntelOpenMP_jll v2024.2.1+0
  [dad2f222] LLVMExtra_jll v0.0.34+0
  [1d63c593] LLVMOpenMP_jll v18.1.7+0
  [94ce4f54] Libiconv_jll v1.17.0+1
  [5ced341a] Lz4_jll v1.10.0+1
  [856f044c] MKL_jll v2024.2.0+0
  [7cb0a576] MPICH_jll v4.2.3+0
  [f1f71cc9] MPItrampoline_jll v5.5.1+0
  [9237b28f] MicrosoftMPI_jll v10.1.4+2
⌃ [7243133f] NetCDF_jll v400.902.209+0
⌅ [4c82536e] Nettle_jll v3.7.2+0
  [fe0851c0] OpenMPI_jll v5.0.5+0
  [458c3c95] OpenSSL_jll v3.0.15+1
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [c2071276] P11Kit_jll v0.24.1+0
  [f50d1b31] Rmath_jll v0.5.1+0
  [02c8fc9c] XML2_jll v2.13.4+0
  [ffd25f8a] XZ_jll v5.6.3+0
  [3161d3a3] Zstd_jll v1.5.6+1
  [477f73a3] libaec_jll v1.1.2+0
  [337d8026] libzip_jll v1.11.1+0
  [1317d2d5] oneTBB_jll v2021.12.0+0
  [0dad84c5] ArgTools v1.1.1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8ba89e20] Distributed
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching
  [9fa8497b] Future
  [b77e0a4c] InteractiveUtils
  [4af54fe1] LazyArtifacts
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.10.0
  [de0858da] Printf
  [9abbd945] Profile
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays v1.10.0
  [10745b16] Statistics v1.10.0
  [4607b0f0] SuiteSparse
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll v1.1.1+0
  [781609d7] GMP_jll v6.2.1+6
  [deac9b47] LibCURL_jll v8.4.0+0
  [e37daf67] LibGit2_jll v1.6.4+0
  [29816b5a] LibSSH2_jll v1.11.0+1
  [c8ffd9c3] MbedTLS_jll v2.28.2+1
  [14a3606d] MozillaCACerts_jll v2023.1.10
  [4536629a] OpenBLAS_jll v0.3.23+4
  [05823500] OpenLibm_jll v0.8.1+2
  [bea87d4a] SuiteSparse_jll v7.2.1+1
  [83775a58] Zlib_jll v1.2.13+1
  [8e850b90] libblastrampoline_jll v5.11.0+0
  [8e850ede] nghttp2_jll v1.52.0+1
  [3f19e933] p7zip_jll v17.4.0+2
Info Packages marked with ⌃ and ⌅ have new versions available. Those with ⌃ may be upgradable, but those with ⌅ are restricted by compatibility constraints from upgrading. To see why use `status --outdated -m`

Full output

In case of an error, please paste the full error message and stack trace.

[Alexander-Barth]

Thank you for your complete issue report! I agree that the warning is annoying (it should be displayed only once with this commit JuliaIO/DiskArrays.jl@324867c ).

But is the DiskArrays.setindex_disk! function not part of the private API of DiskArrays ?

Maybe a change in DiskArrays would be more appropriate. For example, if the user sets the fallback_element_size via Preferences.jl then DiskArrays would consider that as a signal that the users knows they are doing? This would "solve" the issue not only for NCDatasets but for all users of DiskArrays. A user could still override per-script DiskArrays.fallback_element_size[].

What you do think Fabian @meggart ?

[sjdaines]

From the point of view of the user of a package that depends on NCDatasets, I think any warning like this for what is actually normal use is a problem ! (and leads to misdirection if something entirely unrelated goes wrong in that package).
It really needs to "just work" with no warnings and no configuration via Preferences etc.
Perhaps NCDatasets could choose a default based on the assumption that Strings in a netcdf file will usually be relatively short metadata etc?

[Alexander-Barth]

The warning is not generated by NCDatasets but by a dependency.
We cannot address the problem in NCDatasets using the public API of DiskArrays.
Can you open a issue at DiskArrays ?

[Alexander-Barth]

Addressed in DiskArrays JuliaIO/DiskArrays.jl#209