> ## Documentation Index
> Fetch the complete documentation index at: https://clickhouse.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring Cloudflare logs with ClickHouse Observability

> Ingest Cloudflare logs into self-managed ClickHouse Observability using ClickHouse Cloud and ClickPipes.

> [tab] Self-managed
>
> > [!NOTE]
> >
> > **TL;DR**
> >
> > This guide shows you how to ingest Cloudflare logs into ClickStack using ClickPipes. Cloudflare Logpush writes logs to S3, and ClickPipes continuously ingests new files into ClickHouse. Unlike most ClickStack integration guides that use the OpenTelemetry Collector, this guide uses [ClickPipes](https://clickhouse.com/docs/integrations/clickpipes/home.md) to pull data directly from S3.
> >
> > A demo dataset is available if you want to explore the dashboards before configuring production ingestion.
>
> ## Overview
>
> Cloudflare [Logpush](https://developers.cloudflare.com/logs/about/) exports HTTP request logs to destinations like Amazon S3. Forwarding these logs to ClickStack allows you to:
>
> * Analyze edge traffic, cache performance, and security events alongside your other observability data
> * Query logs using ClickHouse SQL
> * Retain logs beyond Cloudflare's default retention
>
> This guide uses [ClickPipes](https://clickhouse.com/docs/integrations/clickpipes/home.md) to continuously ingest Cloudflare log files from S3 into ClickHouse. S3 acts as a durable buffer between Cloudflare and ClickHouse, providing exactly-once semantics and replay capability.
>
> > [!NOTE]
> >
> > **Alternative: Direct HTTP ingestion**
> >
> > Cloudflare Logpush also supports pushing logs to [HTTP endpoints](https://developers.cloudflare.com/logs/get-started/enable-destinations/http/) directly. Since Cloudflare exports logs as newline-delimited JSON (NDJSON) and ClickHouse natively accepts this format via `JSONEachRow`, you can point Logpush directly at your ClickHouse Cloud HTTP interface using the following endpoint URL format:
> >
> > ```text theme={null}
> > https://YOUR_CLICKHOUSE_HOST:8443/?query=INSERT+INTO+cloudflare_http_logs+FORMAT+JSONEachRow&header_Authorization=Basic+BASE64_CREDENTIALS
> > ```
> >
> > Replace `YOUR_CLICKHOUSE_HOST` with your ClickHouse Cloud hostname and `BASE64_CREDENTIALS` with your Base64-encoded credentials (`echo -n 'default:YOUR_PASSWORD' | base64`).
> >
> > This is simpler to set up (no S3, SQS, or IAM configuration needed), but Cloudflare Logpush [cannot backfill historical data](https://developers.cloudflare.com/logs/logpush/) if delivery fails — so if ClickHouse is unavailable during a push, those logs are permanently lost.
>
> ## Integration with existing Cloudflare Logpush
>
> This section assumes you have Cloudflare Logpush configured to export logs to S3. If not, follow [Cloudflare's AWS S3 setup guide](https://developers.cloudflare.com/logs/get-started/enable-destinations/aws-s3/) first.
>
> ### Prerequisites
>
> * **ClickHouse Cloud service** running (ClickPipes is a Cloud-only feature — not available in ClickStack OSS)
> * Cloudflare Logpush actively writing logs to an S3 bucket
> * S3 bucket name and region where Cloudflare writes logs
>
> **1. Configure S3 authentication**
>
> ClickPipes needs permission to read from your S3 bucket. Follow the [Accessing S3 data securely](https://clickhouse.com/docs/products/cloud/guides/data-sources/accessing-s3-data-securely.md) guide to configure either IAM role-based access or credentials-based access.
>
> For full details on ClickPipes S3 authentication and permissions, see the [S3 ClickPipes reference documentation](https://clickhouse.com/docs/integrations/clickpipes/object-storage/amazon-s3/overview.md#access-control).
>
> **2. Create ClickPipes job**
>
> 1. ClickHouse Cloud Console → **Data Sources** → **Create ClickPipe**
> 2. **Source**: Amazon S3
>
> ![Clickpipe s3](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudflare/clickpipe-s3.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=0df0a2e7193512253d66522909e355b0)
>
> **Connection:**
>
> * **S3 file path**: Your Cloudflare logs bucket path with a wildcard to match files. If you enabled daily subfolders in Logpush, use `**` to match across subdirectories:
>   * No subfolders: `https://your-bucket.s3.us-east-1.amazonaws.com/logs/*`
>   * Daily subfolders: `https://your-bucket.s3.us-east-1.amazonaws.com/logs/**/*`
> * **Authentication**: Select your authentication method and provide the credentials or IAM Role ARN
>
> **Ingestion settings:**
>
> Click **Incoming data**, then configure:
>
> * Toggle on **Continuous ingestion**
> * **Ordering**: Lexicographical order
>
> ![Continuous ingestion](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudflare/continuous-ingestion.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=001061bc198ce9d54cf25f32e943183a)
>
> Cloudflare Logpush writes files with date-based naming (e.g., `20250127/...`), which is naturally lexicographical. ClickPipes polls for new files every 30 seconds and ingests any file with a name greater than the last processed file.
>
> **Schema mapping:**
>
> Click **Parse information**. ClickPipes samples your log files and auto-detects the schema. Review the mapped columns and adjust types as needed. Define a **Sorting key** for the destination table — for Cloudflare logs, a good choice is `(EdgeStartTimestamp, ClientCountry, EdgeResponseStatus)`.
>
> ![Parse information](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudflare/parse-information.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=604ffdd7c1756ccd8586ea7bcd308b7c)
>
> Click **Complete Setup**.
>
> > [!NOTE]
> >
> > When first created, ClickPipes performs an initial load of **all existing files** in the specified path before switching to continuous polling. If your bucket contains a large backlog of Cloudflare logs, this initial load may take some time.
>
> **3. Configure ClickStack UI data source**
>
> ClickPipes ingests Cloudflare logs into a flat table with Cloudflare's native field names. To view these logs in ClickStack UI, configure a custom data source that maps Cloudflare columns to ClickStack UI's log view.
>
> 1. Open ClickStack UI → **Team Settings** → **Sources**
>
> ![Add source](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudflare/add-source.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=19c53b6983ad55d66dc5186907a39607)
>
> 2. Click **Add source** and configure the following settings. Click **Configure Optional Fields** to access all fields:
>
> ![Configure optional](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudflare/configure-optional-fields.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=d0a0ba7143aa8c8d5e3934824aaa28e1)
>
> | Setting | Value |
> | - | - |
> | **Name** | `Cloudflare Logs` |
> | **Source Data Type** | Log |
> | **Database** | `default` |
> | **Table** | `cloudflare_http_logs` |
> | **Timestamp Column** | `toDateTime(EdgeStartTimestamp / 1000000000)` |
> | **Default Select** | `EdgeStartTimestamp, ClientRequestMethod, ClientRequestURI, EdgeResponseStatus, ClientCountry` |
> | **Service Name Expression** | `'cloudflare'` |
> | **Log Level Expression** | `multiIf(EdgeResponseStatus >= 500, 'ERROR', EdgeResponseStatus >= 400, 'WARN', 'INFO')` |
> | **Body Expression** | `concat(ClientRequestMethod, ' ', ClientRequestURI, ' ', toString(EdgeResponseStatus))` |
> | **Log Attributes Expression** | `map('http.method', ClientRequestMethod, 'http.status_code', toString(EdgeResponseStatus), 'http.url', ClientRequestURI, 'client.country', ClientCountry, 'client.ip', ClientIP, 'cache.status', CacheCacheStatus, 'bot.score', toString(BotScore), 'cloudflare.ray_id', RayID, 'cloudflare.colo', EdgeColoCode)` |
> | **Resource Attributes Expression** | `map('cloudflare.zone', ClientRequestHost)` |
> | **Implicit Column Expression** | `concat(ClientRequestMethod, ' ', ClientRequestURI)` |
>
> 3. Click **Save Source**
>
> ![Save source](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudflare/save-source.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=fade4c475bb186ce5b616c1439fa56a5)
>
> This maps Cloudflare's native columns directly to ClickStack UI's log viewer without any data transformation or duplication. The **Body** displays a request summary like `GET /api/v1/users 200`, and all Cloudflare fields are available as searchable attributes.
>
> **4. Verify data in ClickStack UI**
>
> Navigate to the **Search** view and select the **Cloudflare Logs** source. Set the time range to cover your data. You should see log entries with:
>
> * Request summaries in the Body column (e.g., `GET /api/v1/users 200`)
> * Severity levels color-coded by HTTP status (INFO for 2xx, WARN for 4xx, ERROR for 5xx)
> * Searchable attributes like `http.status_code`, `client.country`, `cache.status`, and `bot.score`
>
> ![Search view](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudflare/search-view.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=2c7eb327246bb2a772b6dc05a608fba8)
>
> ![Log view](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudflare/log-view.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=079790edc71602f96242d0f8722fd555)
>
> ## Demo dataset
>
> For users who want to test the integration before configuring their production Cloudflare Logpush, we provide a sample dataset with realistic HTTP request logs.
>
> **1. Start ClickPipes with the demo dataset**
>
> 1. ClickHouse Cloud Console → **Data Sources** → **Create ClickPipe**
> 2. **Source**: Amazon S3
> 3. **Authentication**: Public
> 4. **S3 file path**: `https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/cloudflare/cloudflare-http-logs.json`
> 5. Click **Incoming data**
> 6. Select **JSON** as the format
> 7. Click **Parse information** and review the detected schema
> 8. Set the **Table name** to `cloudflare_http_logs`
> 9. Click **Complete Setup**
>
> The dataset includes 5,000 HTTP request log entries spanning 24 hours with realistic patterns including traffic from multiple countries, cache hits and misses, API and static asset requests, error responses, and security events.
>
> **2. Configure ClickStack UI data source**
>
> Follow the [data source configuration steps](#configure-source-self-configured) to create a ClickStack UI source pointing to the `cloudflare_http_logs` table. If you already configured the source in the production integration section, this step is not needed.
>
> **3. Verify demo data**
>
> ```sql theme={null}
> SELECT count() FROM cloudflare_http_logs;
> -- Should return 5000
> ```
>
> Navigate to the **Search** view in ClickStack UI, select the **Cloudflare Logs** source, and set the time range to **2026-02-23 00:00:00 - 2026-02-26 00:00:00**.
>
> You should see log entries with request summaries, searchable Cloudflare attributes, and severity levels based on HTTP status codes.
>
> ![Search view](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudflare/search-view.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=2c7eb327246bb2a772b6dc05a608fba8)
>
> ![Log view](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudflare/log-view.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=079790edc71602f96242d0f8722fd555)
>
> > [!NOTE]
> >
> > **Timezone Display**
> >
> > ClickStack UI displays timestamps in your browser's local timezone. The demo data spans **2026-02-24 00:00:00 - 2026-02-25 00:00:00 (UTC)**. The wide time range ensures you'll see the demo logs regardless of your location. Once you see the logs, you can narrow the range to a 24-hour period for clearer visualizations.
>
> ## Dashboards and visualization
>
> **1. Download the dashboard configuration**
>
> [Download the dashboard configuration file](https://clickhouse-docs-assets.s3.us-east-1.amazonaws.com/examples/cloudflare-logs-dashboard.json).
>
> **2. Import dashboard**
>
> 1. ClickStack UI → **Dashboards** → **Import Dashboard**
>
> ![Import dashboard](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudflare/import-dashboard.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=1c5b1bd34ecbd420ff512b52286aa5d3)
>
> 2. Upload `cloudflare-logs-dashboard.json` → **Finish Import**
>
> ![Import dashboard](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudflare/finish-import.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=fb9a98077655fb5585bc430127e1be0a)
>
> **3. View dashboard**
>
> ![Example dashboard](https://mintcdn.com/private-7c7dfe99/LCIbqddLfKHZZnfk/images/clickstack/cloudflare/example-dashboard.webp?fit=max\&auto=format\&n=LCIbqddLfKHZZnfk\&q=85\&s=f149e46e8b8404dae36871d57997a301)
>
> > [!NOTE]
> >
> > For the demo dataset, set the time range to **2026-02-24 00:00:00 - 2026-02-25 00:00:00 (UTC)** (adjust based on your local timezone). The imported dashboard won't have a time range specified by default.
>
> ## Troubleshooting
>
> ### Data not appearing in ClickHouse
>
> Verify the table was created and contains data:
>
> ```sql theme={null}
> SHOW TABLES FROM default LIKE 'cloudflare_http_logs';
> SELECT count() FROM cloudflare_http_logs;
> ```
>
> If the table exists but is empty, check ClickPipes for errors: ClickHouse Cloud Console → **Data Sources** → Your ClickPipe → **Logs**. For authentication issues with private buckets, see the [S3 ClickPipes access control documentation](https://clickhouse.com/docs/integrations/clickpipes/object-storage/amazon-s3/overview.md#access-control).
>
> ### Logs not appearing in ClickStack UI
>
> If data is in ClickHouse but not visible in ClickStack UI, check the data source configuration:
>
> * Verify a source exists for `cloudflare_http_logs` under ClickStack UI → **Team Settings** → **Sources**
> * Ensure the **Timestamp Column** is set to `toDateTime(EdgeStartTimestamp / 1000000000)` — Cloudflare timestamps are in nanoseconds and need to be converted
> * Verify your time range in ClickStack UI covers the data. For the demo dataset, use **2026-02-23 00:00:00 - 2026-02-26 00:00:00**
>
> ## Next steps
>
> * Set up [alerts](https://clickhouse.com/docs/clickstack/observability/features/alerts.md) for security events (WAF blocks, bot traffic spikes, error rate thresholds)
> * Optimize [retention policies](https://clickhouse.com/docs/clickstack/self-configured/managing/ttl.md) based on your data volume
> * Create additional dashboards for specific use cases (API performance, cache optimization, geographic traffic analysis)
>
> ## Going to production
>
> This guide demonstrates ingesting Cloudflare logs using a public demo dataset. For production deployments, configure Cloudflare Logpush to write to your own S3 bucket and set up ClickPipes with [IAM role-based authentication](https://clickhouse.com/docs/products/cloud/guides/data-sources/accessing-s3-data-securely.md) for secure access. Select only the [Logpush fields](https://developers.cloudflare.com/logs/logpush/logpush-job/datasets/zone/http_requests/) you need to reduce storage costs and ingestion volume. Enable daily subfolders in Logpush for better file organization and use `**/*` in your ClickPipes path pattern to match across subdirectories.
>
> See the [S3 ClickPipes documentation](https://clickhouse.com/docs/integrations/clickpipes/object-storage/amazon-s3/overview.md) for advanced configuration options including [SQS-based unordered ingestion](https://clickhouse.com/docs/integrations/clickpipes/object-storage/amazon-s3/overview.md#continuous-ingestion-any-order) for handling backfills and out-of-order files.