Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bounding box column updates #202

Merged
merged 2 commits into from
May 29, 2024
Merged

Bounding box column updates #202

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5bba669
Bounding box column updates
kylebarron Apr 15, 2024
4fb254a
Merge branch 'main' into kylebarron-patch-1
cholmes May 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion format-specs/geoparquet.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ Note: This technique to use the bounding box to improve spatial queries does not

### Bounding Box Columns

A bounding box column MUST be a Parquet group field with 4 child fields named `xmin`, `xmax`, `ymin`, and `ymax` representing the geometry's coordinate range. As with the top-level [`bbox`](#bbox) column, the values follow the GeoJSON specification (RFC 7946, section 5), which also describes how to represent the bbox for geometries that cross the antimeridian. For three dimensions the additional fields `zmin` and `zmax` MAY be present but are not required. The fields MUST be of Parquet type `FLOAT` or `DOUBLE` and all columns MUST use the same type. The repetition of a bounding box column MUST match the geometry column's [repetition](#repetition). A row MUST contain a bounding box value if and only if the row contains a geometry value. In cases where the geometry is optional and a row does not contain a geometry value, the row MUST NOT contain a bounding box value.
A bounding box column MUST be a Parquet group field with 4 or 6 child fields representing the geometry's coordinate range. For two-dimensional data, the child fields MUST be named `xmin`, `ymin`, `xmax`, and `ymax` and MUST be ordered in this same way. As with the top-level [`bbox`](#bbox) column, the values follow the GeoJSON specification (RFC 7946, section 5), which also describes how to represent the bbox for geometries that cross the antimeridian. For three dimensions the additional fields `zmin` and `zmax` MAY be present but are not required. If `zmin` is present then `zmax` MUST be present and vice versa. If `zmin` and `zmax` are present, the ordering of the child fields MUST be `xmin`, `ymin`, `zmin`, `xmax`, `ymax`, `zmax`. The fields MUST be of Parquet type `FLOAT` or `DOUBLE` and all columns MUST use the same type. The repetition of a bounding box column MUST match the geometry column's [repetition](#repetition). A row MUST contain a bounding box value if and only if the row contains a geometry value. In cases where the geometry is optional and a row does not contain a geometry value, the row MUST NOT contain a bounding box value.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I notice that this requirement about the ordering of the fields makes invalid the newly released 2024-04-16-beta.0 Overture Maps data (where the fields in the bbox group happen to be ordered xmin, xmax, ymin, and ymax).

I've missed recent meetings, and maybe this has been covered, but I'm curious about including both the name and the order of the fields in the spec - this feels "overspecified" to me. Are there clients that cannot work with the named fields in a group?

I don't know of one, but it wouldn't surprise me if there were data producers that couldn't control the order of the fields in a group (maybe using a language where unordered struct fields are serialized in a Parquet group field).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also wondering why the ordering is required? Shouldn't access go through the names that are specified in the array?

Copy link
Collaborator Author

@kylebarron kylebarron Apr 29, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the difference here is whether you consider the struct type as a hashmap or as an ordered list of tuples. I see Parquet's struct type more as the latter than the former. E.g. the struct type is really the collection of fields like so

[
    ("bbox", "xmin"),
    ("bbox", "ymin"),
    ("bbox", "xmax"),
    ("bbox", "ymax")
]

You can loop through the columns to find the column index for your desired name, but it's not a map. You can't directly access a column by name. Instead, you must first find the column index, then access the column by index.

Given that low-level access is done by index, why not strictly specify column order in the spec as well? If there were some writer that was unable to specify column order in a struct type, that might be a good reason. Does that exist?


The bounding box column MUST be at the root of the schema. The bounding box column MUST NOT be nested in a group.

Expand Down
Loading