Using k8up for Elasticsearch Backup: Best Practices for S3 Snapshot Repositories?

[cf-sewe]

Hello everyone,

I'm exploring the best practices for backing up Elasticsearch indices using k8up. My current plan involves leveraging Elasticsearch’s snapshot API, which natively supports S3 as a target repository. You can find more details in the Elasticsearch Snapshot Documentation.

However, I'm unsure whether I should:

• Use the snapshot API to directly back up indices to S3, or
• Use a local file repository for Elasticsearch snapshots and let only k8up handle transferring the backup to S3.

I'd appreciate your insights on:

• Performance and reliability comparisons between using the native S3 support and a local file repository.
• Operational considerations and any pitfalls encountered with either approach.
• Best practices for integrating Elasticsearch snapshots with k8up in a production environment.

Looking forward to your expert opinions and experiences.

Thanks!
Sebastian

[Kidswiss]

I've used to manage some large-ish ElasticSearch clusters in the past. We've used both methods, however the setup wasn't in K8s but rather via dedicated VMs.

If you're planning to cluster ElastichSeach and go with the local file repository, be aware that it has to be a shared filesystem. E.g. NFS, CephFS, smb, etc. We used restic to move the data to S3 from that shared filesystem.

Later the backup was changed to S3 directly without any restic involved. Going directly to S3 is a lot less complex, as it involves less moving parts.

[cf-sewe]

Thanks for your insights. the reason why i prefer to use k8up for all, is because we have multiple inter-depending services (MariaDB, apps PVC, Elasticsearch) and need to backup them in a coordinated way. Also getting status from k8up would make since easier, also of course restore by one tool.

I currently consider Elasticsearch -> local storage and then let this be picked up by k8up.
I am unsure if it makes sense letting Elasticsearch snapshotting to S3 and then again letting k8up to (same) S3 again.

[Kidswiss]

I mean both variants are viable. We never had issues with the share storage backup solution (except the occasional full share ;) ). I just wanted to point out that the directly to S3 solution is the simpler one.

Also keep in mind that with the shared storage you'll have to basically restore everything twice in the worst case. First you need to restore the share storage PVC and then you'll have to tell ElasticSearch to restore the data from there. So your RTO will probably double.

But I get your requirements, so it could make sense to go with the K8up variant.

I am unsure if it makes sense letting Elasticsearch snapshotting to S3 and then again letting k8up to (same) S3 again.

I'm not quite sure what you mean by that? You'd want to use the same bucket for the K8up and ElasticSearch backups? I think it should technically work. K8up let's you use a sub folder in a bucket as the repository. However I'm not sure if ElasticSearch also allows that.