Backup Location
Backups are stored in Amazon S3. Each cluster’s backups are organized under the cluster’s S3 key prefix (found in the CR atspec.s3.keyPrefix), which ensures that backups from different clusters remain isolated even when sharing the same S3 bucket.
Manual vs API-Driven Backups
ClickHouse Private supports two backup approaches:- Manual (SQL) — Execute
BACKUP/RESTOREstatements directly on a server pod. Full control over scope and S3 paths. See Back up and restore (manual). - API-driven — Use the Private API REST endpoint, which creates
Backupcustom resources that the operator manages. See Manage backups via API and the API reference.
Backup Pod Designation
In the MultiSTS architecture, exactly one replica is designated as the backup pod (isBackupPod: true in the ReplicaStateMap). Backup operations always execute on this pod. The operator avoids marking the backup pod as Condemned during scale-in to prevent disrupting backup operations.
Restore Limitation
The Private API disallows restoring a backup to the same cluster it was taken from (instance_id == target_instance_id). This prevents placing additional load on a running production cluster and reduces the risk of data overwrites. The recommended approach is to provision a new cluster and restore into it.
If you must restore to the same cluster (for example, to recover a specific table), use manual SQL commands directly on a server pod.
Lifecycle management
There is no one size fits all approach to lifecycle management for backups. Sevaral strategies exist:- Using Cloud storage lifecycle policies (i.e. Object Lifecycle Rules). Care should be taken to avoid breaking the backup chain (see the section below)
- Custom CronJob in Kubernetes that queries the backups & determines which backups can be safely removed.