Spec: Document Snapshot Summary Optional Fields for Standardization

[HonahX]

This PR introduces a new section, "Snapshot Summary", in the table spec under Snapshots to document optional fields in the snapshot summary, including metrics, partition-level summaries, and other fields such as Write-Audit-Publish (WAP)-related fields and ReplacePartitions indicators. The goal is to establish a clear standard for these fields, ensuring consistent naming and usage across implementations while reducing ambiguity and improving compatibility.

Proposal Here: https://docs.google.com/document/d/1Gt1ZOXVXK60IGdlmt4QlyRzaZ1iCVyYUBfMJCsiz14I/edit?usp=sharing

Marked as Draft because this is subject to change based on discussion on dev list

[RussellSpitzer]

Unclear where duplicate files are coming from

[HonahX]

This comes from a check in ManifestFilterManager:

iceberg/core/src/main/java/org/apache/iceberg/ManifestFilterManager.java

Lines 492 to 497 in 90be5d7

	if (deletedFiles.contains(file)) {
	LOG.warn(
	"Deleting a duplicate path from manifest {}: {}",
	manifest.path(),
	file.location());
	duplicateDeleteCount += 1;

"Duplicate" means there are 2 manifest entry pointing to the same data file in one Manifest.

How about

Number of duplicate files deleted, where duplicates are files recorded more than once in the manifest

[HonahX]

@RussellSpitzer Thanks for reviewing! I moved the whole section to Appendix G and update field descriptions accordingly

[HonahX]

This comes from a check in ManifestFilterManager:

iceberg/core/src/main/java/org/apache/iceberg/ManifestFilterManager.java

Lines 492 to 497 in 90be5d7

	if (deletedFiles.contains(file)) {
	LOG.warn(
	"Deleting a duplicate path from manifest {}: {}",
	manifest.path(),
	file.location());
	duplicateDeleteCount += 1;

"Duplicate" means there are 2 manifest entry pointing to the same data file in one Manifest.

How about

Number of duplicate files deleted, where duplicates are files recorded more than once in the manifest

[RussellSpitzer]

Not a helpful description here :)

[HonahX]

Thanks for catching this! Let me add more context for ReplacePartitions here.

That said, I’m on the fence about including this field. Most users likely won’t encounter it, as ReplacePartitions is generally not recommended. I decided to include it to make the appendix more comprehensive and to "reserve" the field name to prevent potential conflicts.

Link for reference: ReplacePartitions.java

[RussellSpitzer]

I'm not sure we should document this, the string representation we use for the path is not really well documented and in general we probably don't want folks including partition level summaries. Additionally it references a Table property (write.summary.partition-limit) which I really would like to avoid. I think we should stick to the table level stats above and remove this section and the associated column.

[HonahX]

Good point! I also found it challenging to document partition-level summaries while writing the appendix, as they require explanations for a lot of context. Referencing write.summary.partition-limit directly in the table spec also feels out of place. Additionally, most users won’t encounter partition-level summaries because the default value for write.summary.partition-limit is 0.

Even so, I think it’s still helpful to document this information somewhere for those who might enable or work with partition-level summaries. One option could be to move the description to the Table Configuration section and include a note under write.summary.partition-limit to explain what partition-level summaries mean. This approach ties the information to the property users would need to modify if they choose to enable this feature. The downside is that snapshot summaries would end up being documented in two places, but this might be a reasonable tradeoff to ensure clarity and usability.

Do you think this sound like a better idea?

[kevinjqliu]

nit: wydt about moving this description up one level, between "Appendix G" and "Metrics" since this is describing generally what fields belong to the Snapshot Summary. It should apply to all subheaders ( "Metrics", "Partition-Level", "Other")

Suggested change

	## Appendix G: Optional Snapshot Summary Fields
	### Metrics
	Snapshot summary can include metrics fields to track numeric stats of the snapshot. The value of these fields should be numeric strings (e.g., `"120"`).
	Some of them are also used to represent partition-level metrics, in [Partition-Level Summary](#partition-level-summary).
	Metrics must be accurate if written, as engines may rely on them for optimization.
	## Appendix G: Optional Snapshot Summary Fields
	Snapshot summary can include metrics fields to track numeric stats of the snapshot, see [Metrics](#metrics).
	Some of them are also used to represent partition-level metrics, in [Partition-Level Summary](#partition-level-summary).
	The value of these fields should be of string type (e.g., `"120"`).
	### Metrics
	Metrics must be accurate if written, as engines may rely on them for optimization.

[HonahX]

Thanks for the suggestion! I've moved the the description up one level and make it mention all the sub-sections. I removed the "partition-level summary" based on Russell's suggestion above.

[kevinjqliu]

LGTM! Thanks for adding this new appendix section