> ## Documentation Index
> Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

> A sacrificial child process that attracts the Linux OOM killer before the ClickHouse server, giving the server a chance to shed load and survive.

# OOM canary

export const ExperimentalBadge = () => {
  return <div className="experimentalBadge">
            <div className="experimentalIcon">
            <svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
                <path strokeWidth="1.25" d="M5.5 2H10.5" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round" />
                <path strokeWidth="1.25" d="M9.50015 2V6.19625L13.4283 12.7425C13.4738 12.8183 13.4985 12.9049 13.4996 12.9934C13.5008 13.0818 13.4785 13.169 13.435 13.246C13.3914 13.323 13.3283 13.3871 13.2519 13.4317C13.1755 13.4764 13.0886 13.4999 13.0002 13.5H3.00015C2.91164 13.5 2.8247 13.4766 2.74822 13.432C2.67174 13.3874 2.60847 13.3233 2.56487 13.2463C2.52126 13.1693 2.49889 13.082 2.50004 12.9935C2.50119 12.905 2.52582 12.8184 2.5714 12.7425L6.50015 6.19625V2" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round" />
                <path strokeWidth="1.25" d="M4.47656 9.56754C5.30344 9.41254 6.47656 9.47942 7.99969 10.25C10.0153 11.2707 11.4216 11.0569 12.2184 10.7282" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round" />
            </svg>
        </div>
            Experimental feature. <u><a href="/docs/docs/beta-and-experimental-features#experimental-features">Learn more.</a></u>
        </div>;
};

<ExperimentalBadge />

<Note>
  The OOM canary is experimental and disabled by default. Its behavior may change
  between ClickHouse versions until production validation is complete.
</Note>

<h2 id="overview">
  Overview
</h2>

When a host or memory cgroup runs out of memory, the Linux OOM (out-of-memory)
killer terminates a process with `SIGKILL` — usually the largest consumer, which
on a dedicated host is `clickhouse-server` itself. The whole server is lost
instead of being given a chance to recover.

The OOM canary changes who dies first. It runs a small *sacrificial* child
process that makes itself the most attractive OOM target, so the kernel kills it
instead of the server. The server then detects the death, confirms it was an OOM
event, and sheds memory pressure so it can survive.

The canary does not raise any memory limit and is not a replacement for correct
limits (see [Memory overcommit](/docs/concepts/features/configuration/settings/memory-overcommit) and
`max_server_memory_usage`). It is a last line of defense that trades a small,
fixed amount of memory for a chance to survive a memory spike.

<h2 id="how-it-works">
  How it works
</h2>

The canary is a separate `clickhouse oom-canary` process. It sets its own
`oom_score_adj` to the maximum (`1000`) so the kernel targets it first, then
allocates, touches, and `mlock`-s `oom_canary_size` bytes (100 MB by default) so
its resident set is real. It is killed automatically if the server exits.

In the server, a monitor thread watches the canary (via `pidfd`) and reacts when
it dies:

* Killed by `SIGKILL` **with** cgroup OOM evidence → run the OOM response, then
  relaunch a fresh canary.
* Killed **without** OOM evidence (for example, a manual `kill -9`), or exited
  with a transient failure → relaunch only, no response.
* Permanent setup failure, or server shutdown → the canary disables itself.

OOM evidence comes only from the cgroup v2 `memory.events.local` `oom_kill`
counter. It is deliberately cgroup-local: hierarchical or host-wide counters can
be advanced by unrelated processes and would trigger false responses.

On a confirmed OOM the response runs these independent steps: log a `FATAL`
message, purge allocator (jemalloc) arenas, best-effort cancel all running
queries, cancel all merges and mutations, and queue an event in
[`system.crash_log`](/docs/reference/system-tables/crash_log). System logs are not
flushed synchronously, because forcing I/O under memory pressure can make things
worse.

<h2 id="requirements">
  Requirements
</h2>

* **Linux ≥ 5.3.** The monitor owns the canary via `pidfd_open`; on older kernels
  the canary disables itself at startup. It is a no-op on non-Linux platforms.
* **cgroup v2 with `memory.events.local`** for the OOM response. Without it the
  canary still relaunches after a `SIGKILL` but cannot confirm an OOM, so the
  response never runs (a warning is logged at startup).
* **`mlock` capability (optional).** Locking the canary's memory needs
  `CAP_IPC_LOCK` or a sufficient `RLIMIT_MEMLOCK`; if it fails the canary logs a
  warning and its memory may be swapped out, weakening it as an OOM target.

<Warning>
  **memory.oom.group**

  If cgroup v2 `memory.oom.group` is enabled for the server's cgroup, the kernel
  kills the entire cgroup as one unit on an OOM — the server dies together with the
  canary and the response never runs. The canary cannot protect the server in this
  mode; a warning is logged at startup.
</Warning>

<h2 id="configuration">
  Configuration
</h2>

The canary is controlled by [server settings](/docs/reference/settings/server-settings/settings),
set as top-level elements of the server configuration and applied on restart.

| Setting                              | Default              | Description                                                                                                                                                  |
| ------------------------------------ | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `oom_canary_enable`                  | `false`              | Enable the OOM canary.                                                                                                                                       |
| `oom_canary_size`                    | `104857600` (100 MB) | Bytes the canary allocates and touches. Larger values make it a more attractive OOM target.                                                                  |
| `oom_canary_relaunch`                | `true`               | Relaunch the canary after it dies (unless it was a permanent setup failure or shutdown), subject to the limits below.                                        |
| `oom_canary_max_rapid_relaunches`    | `10`                 | Maximum consecutive *rapid* relaunches before auto-relaunch is disabled, to avoid thrashing. Resets once a canary outlives `oom_canary_max_backoff_seconds`. |
| `oom_canary_initial_backoff_seconds` | `1`                  | Initial delay between relaunches; doubles each time up to the maximum.                                                                                       |
| `oom_canary_max_backoff_seconds`     | `60`                 | Maximum delay between relaunches.                                                                                                                            |

```xml theme={null}
<clickhouse>
    <oom_canary_enable>1</oom_canary_enable>
    <oom_canary_size>104857600</oom_canary_size>
</clickhouse>
```

<h2 id="observability">
  Observability
</h2>

A confirmed OOM produces a row in
[`system.crash_log`](/docs/reference/system-tables/crash_log) with `signal = 9` and a
`signal_description` mentioning `OOM Canary`:

```sql theme={null}
SELECT event_time, signal, signal_description
FROM system.crash_log
WHERE signal = 9 AND signal_description LIKE '%OOM Canary%'
ORDER BY event_time DESC;
```

The canary's lifecycle and each OOM-response step are also logged to the server log.