Add optional dataset characteristics?

[ranman]

It would be useful to have searchable/indexable data on the characteristics of a dataset:

• total dataset size
• AVG file size
• file type
• etc.

[chueatwork]

Hi @ranman, thanks for your feedback! Some of the attributes you are looking for can be pulled right from the CLI using the --summarize subcommand (https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html). In addition, I would encourage you to check out open.quiltdata.com. Quilt Data compiles bucket-level statistics similar to the ones you've listed (common file types and percentage of bucket, total bucket size) and has demonstrated what they can do on several open datasets.