Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

classic only build not dealing with filters/quantization well #3020

Open
edwardhartnett opened this issue Sep 12, 2024 · 4 comments
Open

classic only build not dealing with filters/quantization well #3020

edwardhartnett opened this issue Sep 12, 2024 · 4 comments

Comments

@edwardhartnett
Copy link
Contributor

For a classic only build there should be no filters or quantization.

# General
-------
NetCDF Version:		4.9.4-development
Dispatch Version:       5
Configured On:		Thu Sep 12 06:47:50 MDT 2024
Host System:		x86_64-pc-linux-gnu
Build Directory: 	/home/ed/netcdf-c
Install Prefix:         /usr/local/netcdf-c-classic
Plugin Install Prefix:  /usr/local/hdf5/lib/plugin

# Compiling Options
-----------------
C Compiler:		/usr/bin/gcc
CFLAGS:			 -fno-strict-aliasing
CPPFLAGS:		
LDFLAGS:		
AM_CFLAGS:		
AM_CPPFLAGS:		
AM_LDFLAGS:		
Shared Library:		yes
Static Library:		yes
Extra libraries:	-lm -lz -lsz -lzstd -lxml2 -lcurl 
XML Parser:             libxml2

# Features
--------
Benchmarks:		no
NetCDF-2 API:		yes
HDF4 Support:		no
HDF5 Support:		no
CDF5 Support:		yes
NC-4 Parallel Support:	no
PnetCDF Support:	no

DAP2 Support:		no
DAP4 Support:		no
Byte-Range Support:	no

S3 Support:	        no
S3 SDK:  	        none

NCZarr Support:		yes
NCZarr Zip Support:     no

Diskless Support:	yes
MMap Support:		no
ERANGE Fill Support:	no
Relaxed Boundary Check:	yes

Plugins Enabled:        yes
Plugin Install Dir:     /usr/local/hdf5/lib/plugin

Quantization:		yes
Logging:     		no
SZIP Write Support:     yes
Standard Filters:       bz2 deflate szip zstd
ZSTD Support:           yes
Parallel Filters:       no

@edwardhartnett
Copy link
Contributor Author

I will submit a fix shortly...

@czender
Copy link
Contributor

czender commented Sep 12, 2024

Whether or not there should be quantization is debatable. It's perfectly well defined and supportable by classic format, it's just that there is not any known use. I think we discussed whether classic should support quantization before, and I remember concurring that it is OK to turn it off for classic. My views have evolved since then. As long as it's working for classic, no reason to turn it off in my mind. YMMV. In any case, NCO supports quantization for classic via it's internal algorithms.

@edwardhartnett
Copy link
Contributor Author

edwardhartnett commented Sep 13, 2024

It does not work for classic.

Putting it in for classic would require diving into the libsrc code, which is not something I would choose to do without great need.

Of course it makes sense for classic, and would even provide benefit: a quantized classic file would gzip to a much smaller size.

My only concern here is that currently netcdf-fortran fails for classic-only netcdf-c library because quantize constants are missing. Obviously we want a classic only build to work.

So my thought was to detect classic only builds, and then to set quantization to "no" (since there is no quantize for classic only). Then netcdf-fortran could see that setting and know not to include the quantize constants.

@czender
Copy link
Contributor

czender commented Sep 13, 2024

OK, nevermind me, I misinterpreted your original message. Sounds like we're in agreement on the hypothetical value of having quantization in classic files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants