What kind of data will be stored in local if we use s3?

[giangndm]

I am working on a logging system which have ton of data then it needs to be stored by ObjectStorage like S3.

I read document but didn't find related information about:

• What kind of data will be stored in local disk?
• Assume that we lost entire cluster nodes but remain s3 data, can we restore from S3 or we need backup of local folder?

Can you answer above questions or give me some related documents?

Thank you very much.

[killme2008]

Sorry, the storage location document is not updated https://docs.greptime.com/user-guide/concepts/storage-location
I will update it ASAP.

If you set the storage.type to s3, the data and index files will be stored in S3, but:

What kind of data will be stored in local disk?

The storage of database and table metadata depends on your deployment type. In standalone setups, metadata like database options, table schema, and internal states are stored locally under data_home. In a cluster configuration, metadata is stored in etcd; a minimum of three nodes is recommended for the etcd cluster.

Assume that we lost entire cluster nodes but remain s3 data, can we restore from S3 or we need backup of local folder?

You cannot restore the database directly from S3. For standalone databases, you must back up the data_home regularly or use EBS volume backups. For clusters, apply the etcd disaster recovery procedures.

For more about DR solutions, please read the DR document https://docs.greptime.com/user-guide/operations/disaster-recovery/overview

[giangndm]

@killme2008 Thank you for clearly answer. I want to go to deeper into what happened in my use case with some assumptions:

• Because of it is log service, database schema isn't change
• I can backup local data daily

What happened if I remain S3 data but restored local data with yesterday version?

[giangndm]

From above DR docs, I got that local data will contains:

• Database and table metadata: it seems to not changed frequently, may be daily backup is necessary.
• WAL: act as a write buffer (or something like that) then if we lost that, we lost only small of chunk of data

Did I understand it correctly?

[killme2008]

What happened if I remain S3 data but restored local data with yesterday version

You can't read the tables or databases that were created, columns altered in some tables after backup.

Database and table metadata: it seems to not changed frequently, may be daily backup is necessary.
WAL: act as a write buffer (or something like that) then if we lost that, we lost only small of chunk of data

Yes. A daily backup is enough if you don't change the table schema or create tables frequently.
Don't worry, the WAL is in the data_home too if you are not using the remote WAL. So it can be saved as a backup too.

[killme2008]

Precise backup schema and metadata can be found in the BR document https://docs.greptime.com/user-guide/operations/disaster-recovery/back-up-&-restore-data

[MichaelScofield]

Maybe we can implement a s3 backed kvbackend, and s3 backed WAL, then all data will be located on s3 solely.

[killme2008]

I think implementing an RDS(or MySQL, Postgres) backend would be better.

[giangndm]

If we can put everything (may be not WAL) to s3, it is perfect to logging system where database schema didn't change frequently.