Skip to content

Commit

Permalink
Use a constant suffix for temporary GCS part files. (#5308)
Browse files Browse the repository at this point in the history
[SC-54146](https://app.shortcut.com/tiledb-inc/story/54146/use-a-constant-suffix-for-the-gcs-temporary-part-files)

This PR changes the format of the names of temporary GCS part files from
`<object_path>__tiledb_<n>` to `<object_path>.part<n>.tmp`. Because the
new names have a constant `.tmp` suffix, this enables GCS users to set a
[lifecycle rule](https://cloud.google.com/storage/docs/lifecycle) to
automatically delete them after a certain time if they happen to be left
over (usually they get deleted at the end of the upload operation, but
it might not happen if the operation was interrupted).


![image](https://github.com/user-attachments/assets/318717fb-6b68-47de-a69d-0fa6e5423053)

Documentation will also be updated. For consistency we should also use
`.tmp` for any kind of future temporary objects we upload.

---
TYPE: IMPROVEMENT
DESC: Temporary files (currently used by GCS only) now end with `.tmp`,
allowing you to set a lifecycle rule to automatically delete them after
a certain time if they happen to be left over.
  • Loading branch information
teo-tsirpanis authored Sep 17, 2024
1 parent cf75a9c commit f016e98
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion tiledb/sm/filesystem/gcs.h
Original file line number Diff line number Diff line change
Expand Up @@ -406,7 +406,7 @@ class GCS {
std::string next_part_path() {
const uint64_t part_id = next_part_id_++;
const std::string part_path =
object_path_ + "__tiledb_" + std::to_string(part_id);
object_path_ + ".part" + std::to_string(part_id) + ".tmp";
part_paths_.emplace_back(part_path);
return part_path;
}
Expand Down

0 comments on commit f016e98

Please sign in to comment.