### Introduction

Our mission for ClickHouse Cloud, since its early days, has been to equip our users to develop analytics applications at scale, without having to worry about the operational overhead of infrastructure planning, sizing, and managing complex scaling operations. For this reason, the service has supported vertical autoscaling since its inception, so users never have to overprovision or monitor their usage. Based on memory and CPU usage, the service scales to accommodate the workload within the bounds specified by the user.

Now, we are excited to bring **horizontal scaling** to ClickHouse Cloud in Private Preview. This feature is available for Production instances in ClickHouse Cloud. Horizontal scaling will allow customers to control the number of replicas in their service to match their workload demands.

As part of Private Preview, this feature is available via ClickHouse Cloud APIs. We will also be enabling horizontal scaling via the cloud console very soon. To access horizontal scaling for your service, please contact the ClickHouse Support Team at support@clickhouse.com.

### Why horizontal scaling

Before we get into the details of horizontal scaling in ClickHouse Cloud, let’s first take a look at how horizontal and vertical scaling are different.

Vertical scaling usually involves increasing the resources available - adding more memory or CPU - to a single machine, or a set of machines, in a cluster. Larger machines can handle greater loads, especially ones where you might want all resources to fit in the memory of a single machine - for instance supporting large joins across tables. There are physical limits to vertical scaling though - the machines have an upper limit to how large they can get.

Horizontal scaling, on the other hand, involves adding more machines. Usually, these are all of the same size in order to scale out a cluster of homogenous machines (or replicas). The workload is then distributed among a large number of smaller machines (as opposed to a few large machines in case of vertical scaling).

Horizontal scaling provides better fault tolerance as the load is spread over a larger set of replicas. It is usually faster to scale out a service horizontally, by adding smaller machines, than it is to scale vertically by increasing the size of existing replicas. Because of the mechanics of how horizontal scaling works, it can also be less disruptive than vertical scaling, as new replicas are being added (rather than existing replicas being resized, or replaced).

Horizontal and vertical scaling help with different kinds of workloads. Workloads that need lots of resources from a single machine benefit from vertical scaling. For workloads with high concurrency, horizontal scaling can be a great alternative. For instance, you might have the need to ingest a large amount of data into ClickHouse. Adding additional replicas will allow the ingestion to proceed much faster. Horizontal scaling also works much better than vertical scaling in scenarios where the request rate is increasing, and you need to quickly add capacity to the cluster without impacting the existing capacity.

### Horizontal scaling in action

Once horizontal scaling is enabled on the service, you can use ClickHouse Cloud [public APIs](https://clickhouse.com/docs/en/cloud/manage/api/swagger#/paths/~1v1~1organizations~1:organizationId~1services~1:serviceId~1scaling/patch) to scale your service by updating the [scaling settings](https://clickhouse.com/docs/en/cloud/manage/api/swagger#/paths/~1v1~1organizations~1%7BorganizationId%7D~1services~1%7BserviceId%7D~1scaling/patch) for the service.

Let’s see how horizontal scaling works.

1. First, inspect the memory allocated to your cluster as well as the number of replicas. You can do this by looking at the cloud console dashboard as well as the number of replicas from your service settings. Here we have a 3 replica cluster with 16 GiB of memory allocated to each, resulting in a total of 48 GiB memory allocated to the cluster.

 ![unnamed (3).png](/uploads/unnamed_3_8227247d97.png)
 
 ![unnamed (4).png](/uploads/unnamed_4_fe1f1310f3.png)
 *Service scaling settings from ClickHouse Cloud console*
 
 

2. Let’s say we now want to horizontally scale the cluster to 6 replicas of 16 GiB each. This means that once scaling is complete, we should end up with a total memory of 96 GiB across six replicas. 

 > **Note**: The API currently assumes that the min and max. total memory is based on 3 replicas, which is the number of replicas in a default production ClickHouse cluster. This means that the `minTotalMemoryGb `and `maxTotalMemoryGb` have values equal to 24 and 96 respectively, which comes out to a per-node memory minimum of `8GiB` and `32GiB` respectively, as reflected by the UI. If we want to horizontally scale out from 3 to 6 replicas, we need to only change the `numReplicas` in the API, or the `Number of Nodes` in the UI, without changing the min and max memory size. Thus, for an `16GiB` replica size, our total memory will come out to be `96GiB`across 6 replicas

 To do so, we issue a patch request, to adjust the number of replicas in a cluster as shown below (screenshots from postman client).

 ![unnamed (5).png](/uploads/unnamed_5_24d9c7cab6.png)
 *PATCH request to update numReplicas*

 ![unnamed (6).png](/uploads/unnamed_6_9a4ad59745.png)
 *Response from PATCH request*
 
3. Once you issue the PATCH request, you’ll see that the scaling settings screen now displays “6” as the setting for the number of replicas in the cluster. It is important to note that this is not the current number of replicas in the cluster, it is the desired number of replicas.

 ![unnamed (7).png](/uploads/unnamed_7_d528dc717c.png)
 
 

4. Looking back at the monitoring screen for this cluster, we can now see that the cluster has indeed been scaled to a total of 96 GiB memory.

 ![unnamed (8).png](/uploads/unnamed_8_7700054f43.png)
 
 

### Looking to the future

As mentioned earlier, we will soon make horizontal scaling available via the cloud console directly. During Private Preview, we will improve the UI and API experience of horizontally scaling a ClickHouse Cloud service based on customer feedback, before making the capability generally available (GA).

Making horizontal scaling self-service and GA is just the first step. In the near future, we will also enable horizontal autoscaling, whereby a ClickHouse Cloud user is able to set minimum and maximum replica counts on their service as bounds within which their service should autoscale. Based on the load on the cluster, we will automatically horizontally scale the cluster as needed to closely match the actual utilization. Our end goal is to provide a mix of horizontal and vertical autoscaling to achieve the optimal performance for a service based on the workload.

We're excited to announce that horizontal scaling in ClickHouse Cloud is now available in Private Preview.

large_Untitled 2.jpg

medium_Untitled 2.jpg

small_Untitled 2.jpg

thumbnail_Untitled 2.jpg

Untitled 2.jpg

Horizontal Scaling in ClickHouse Cloud

<h2>Introduction</h2>

We’re excited to announce configurable backup controls are now available in ClickHouse Cloud in private preview. 


The ability to take backups of your data is table stakes for any database offering. Backups provide a safety net by ensuring that if data is lost for any unforeseen reason - whether it be accidental deletion, corruption, and so on - the service can be restored to the previous state of the last successful backup. This minimizes downtime and prevents business-critical data from being permanently lost. Backups are critical to any disaster recovery (DR) plan by ensuring that the business can quickly recover from disruptive events, maintain continuity, and resume normal operations, thus minimizing financial and reputational impact. Additionally, backups also help enterprises satisfy their compliance and legal requirements around data retention and are evidence that they have policies in place to prevent loss of data.

<h2>How it works</h2>

All ClickHouse Cloud services come with default backup policies. For Production services, backups are taken daily and retained for two days. This ensures that if data is lost for any unforeseen reason, the service can be restored to the last successful backup. Backups are a combination of full and incremental backups that constitute a sequence of backups utilized together to restore data to a new service if needed.


ClickHouse Cloud default backups (with a 24-hour backup frequency, and 48-hour retention) satisfy the business needs for some customers. However, certain customers need the additional ability to take backups more frequently and retain them for a longer duration to meet their Business Continuity, or in some cases, compliance requirements. 


Additionally, some customers prefer that backups happen at a certain time during the day. This prevents the backup process from competing with compute resources allocated to the service during critical hours, and importantly, it gives customers control over the backups being taken at a time when most of their daily data changes have been committed.


To address these needs, we are making our backup process more flexible by giving customers the ability to configure the start time, retention, and frequency for backups of their Production and Dedicated tier services. We will continue to provide the two default backups at no cost. However, changes to the backups schedule that require retention for a longer duration, or more frequent backups that require additional copies of the data to be retained, may incur additional charges.

<h2>Let’s walk through an example</h2>

To configure backups for your service and set the schedule to be different from the default, go to the service settings page and navigate to the Backups section. Click the “Change backup configuration.”



![1conf.png](https://cms.clickhouse-dev.com:1337/uploads/1conf_634d82f134.png)



The Backup configuration page lets you modify the retention, frequency, or start time of the backups for your service. You can choose any start time over a 24-hour window, and backups will start within an hour of the scheduled time. Backups can be set up to happen as frequently as every 6 hours, and as infrequently as every 24 hours, with several intermediate values supported within that range. Retention ranges from 1 day and goes up to 30 days, which refers to the ability to roll back to a certain point in time.



![2conf.png](https://cms.clickhouse-dev.com:1337/uploads/2conf_d03c87f980.png)



NOTE: Frequency and start time (Scheduled) are mutually exclusive settings for backups. For example, if you select 2 AM UTC as the start time for your backups, you won’t be able to simultaneously set a frequency of backing up the data every 6 hours.

Available backups, usage, and cost


All available backups for your service are displayed on the backups page in the Cloud Console. In the example below, backups have been configured to happen every 6 hours, with a 5-day retention. From here (under Actions) you can also select a particular backup and choose to restore it to a new service. Details of restoring a backup to a service are covered in our <a href="https://clickhouse.com/docs/en/manage/backups">public docs</a>.



![3conf.png](https://cms.clickhouse-dev.com:1337/uploads/3conf_4cd7309f93.png)



To understand the cost impact of your backup configuration, you can look at the usage breakdown for your service under the “Organization” section on the Cloud Console. If there are costs associated with the backup configuration you’ve selected, they will be displayed on this page under the column “Backups.”



![unnamed.png](https://cms.clickhouse-dev.com:1337/uploads/unnamed_a38262a70a.png)


<h2>Looking ahead</h2>

We plan to make ClickHouse Cloud backups even more seamless and flexible. We will soon enable on-demand backups, so you can kick off a backup at any point in time from the UI, or programmatically via APIs.


We will also soon support the ability to export backups cross-region. This gives you the ability to fulfill your DR (disaster recovery) requirements in situations where the primary region has an interruption in service. 


Additionally, we are looking to enable the capability to export backups to your own cloud service account. This will allow for more control over the backup lifecycle, as well as data retention for data residency or other compliance purposes.


Finally, in the coming months, we also plan to improve our backup capability to support continuous backups and point-in-time restores (PITR). This will make it possible to have even more granular <a href="https://www.druva.com/glossary/what-is-a-recovery-point-objective-definition-and-related-faqs">RPO</a> for data stored in ClickHouse Cloud.

We’re excited to announce configurable backup controls are now available in ClickHouse Cloud. 

Aashish Kohli

Blog posts by Aashish Kohli

Horizontal Scaling in ClickHouse Cloud

Configurable Backups in ClickHouse Cloud