ClickHouse 26.x · Laravel · MySQL
Your MySQL reports are slow. Here is the fix, one problem at a time.
You know Laravel and MySQL. You have never used ClickHouse. This tutorial follows one SaaS app from its first slow GROUP BY to a replicated, monitored analytics system. Each section fixes a real problem, and each fix opens the next one. When you finish, you can design tables, load data from Laravel in batches, query it fast, and avoid the classic mistakes. The tutorial targets the ClickHouse 26.x line. Check the docs for later changes.
00 Why OLAP is different
Where we are
Your Laravel SaaS app stores users and orders in MySQL. The product team wants dashboards for page views, events, and order revenue. Reports run against MySQL today. They are slow and they hurt the live app.
The problem
You run a monthly revenue report on orders:
SELECT DATE(created_at) AS day, SUM(total) AS revenue
FROM orders
WHERE created_at >= '2026-01-01'
GROUP BY day;On 40 million rows this takes minutes. MySQL reads whole rows from disk. The query competes with live traffic. Users see slow pages.
Why it happens
MySQL is an OLTP database. OLTP means online transaction processing. It stores each row together on disk. This design is good for reading or writing one order by id. It is bad for scanning one column across millions of rows.
ClickHouse is an OLAP database. OLAP means online analytical processing. It stores each column apart on disk. This is called column storage. A revenue query reads only two columns, created_at and total. It skips the rest. This reads far less data.
ClickHouse also sorts and compresses each column. Similar values sit together, so they compress well. ClickHouse claims very high scan speed on analytics queries. Treat all speed numbers here as vendor benchmark claims, not guarantees.
This tutorial targets the ClickHouse 26.x line. ClickHouse uses Year.Month.Patch versions. A minor release ships about once a month. An LTS release ships twice a year. Check the docs for changes after 26.x.
The fix
First, run ClickHouse next to your app with Docker. Create docker-compose.yml:
services:
app:
build: .
depends_on:
- clickhouse
environment:
CLICKHOUSE_HOST: clickhouse
CLICKHOUSE_PORT: 8123
clickhouse:
image: clickhouse/clickhouse-server:26.3
ports:
- "8123:8123" # HTTP interface
- "9000:9000" # native TCP
- "9004:9004" # MySQL wire protocol
ulimits:
nofile:
soft: 262144
hard: 262144
volumes:
- ch_data:/var/lib/clickhouse
volumes:
ch_data:Port 8123 is the HTTP interface. Port 9000 is the native TCP interface. Port 9004 speaks the MySQL wire protocol. Most Laravel clients use 8123.
Create your first table. In MySQL you pick InnoDB. In ClickHouse you pick a table engine. The main engine is MergeTree.
CREATE TABLE events
(
event_time DateTime,
user_id UInt64,
event_type String,
url String
)
ENGINE = MergeTree
ORDER BY (event_time);MergeTree is the base engine for large tables. It sorts data by the ORDER BY key and stores it in parts. A part is one immutable folder of sorted, compressed column files.
Run your first query:
SELECT event_type, count() AS c
FROM events
GROUP BY event_type
ORDER BY c DESC;┌─event_type─┬─────c─┐
│ page_view │ 91230 │
│ click │ 12044 │
│ purchase │ 811 │
└────────────┴───────┘Now connect Laravel. Install the integration package:
composer require glushkovds/phpclickhouse-laravelCheck Packagist for Laravel 13 and PHP 8.4 support before you pin a version. Set the environment values:
CLICKHOUSE_HOST=clickhouse
CLICKHOUSE_PORT=8123
CLICKHOUSE_DATABASE=default
CLICKHOUSE_USERNAME=default
CLICKHOUSE_PASSWORD=The package registers a clickhouse connection. You get PhpClickHouseLaravel\BaseModel for models and PhpClickHouseLaravel\Migration for migrations. It wraps the low-level client smi2/phpclickhouse, which uses the HTTP interface with curl.
Define a model:
<?php
namespace App\Models\Clickhouse;
use PhpClickHouseLaravel\BaseModel;
class Event extends BaseModel
{
protected $table = 'events';
}Migrations extend PhpClickHouseLaravel\Migration, live in database/migrations, and run with the normal php artisan migrate. Section 04 shows one.
Run your first query from Laravel:
use App\Models\Clickhouse\Event;
$rows = Event::select(['event_type'])
->selectRaw('count() AS c')
->groupBy('event_type')
->orderBy('c', 'desc')
->getRows();Tips and tricks
- You can query ClickHouse over the MySQL wire protocol on port 9004 with a normal
DB::connection(). This is handy for a quick read. It does not give you native features such as batch inserts with settings. - Keep MySQL as your source of truth for
usersandorders. Use ClickHouse for analytics only. - The low-level client is
smi2/phpclickhouse. It needs PHP 8.0 or later. The Laravel adapter builds on it. - An alternative is
bavix/laravel-clickhouse. It gives Eloquent models for Laravel 11 and 12 on PHP 8.2 or later.
The next problem
You wire up event tracking. Each web request calls Event::insert([...]), one row at a time, like Eloquent save(). After a burst of traffic ClickHouse starts to reject inserts with an error about too many parts. Your dashboard stops receiving data.
Quiz
01 The TOO_MANY_PARTS wall
Where we are
ClickHouse runs in Docker next to your app. You have an events table and a Laravel model. Your tracking code inserts one event per request. Traffic is growing. Inserts start to fail.
The problem
You see this error in your logs:
DB::Exception: Too many parts (300). Merges are processing
significantly slower than inserts. (TOO_MANY_PARTS)Each insert wrote one part. Thousands of requests wrote thousands of tiny parts. ClickHouse cannot merge them fast enough.
Why it happens
Every insert creates a new part. A part is a sorted, compressed folder on disk. ClickHouse merges small parts into bigger parts in the background. A merge is this background work that combines parts. This is why the engine is called MergeTree.
Merges cost CPU and disk. If you insert faster than ClickHouse can merge, the part count grows. When active parts pass a limit, ClickHouse stops new inserts to protect itself. In MySQL you insert one row at a time with no penalty. In ClickHouse you must insert in batches, because each insert is a part.
The fix
Insert thousands of rows in one statement:
INSERT INTO events (event_time, user_id, event_type, url) VALUES
('2026-08-25 10:00:00', 1, 'page_view', '/pricing'),
('2026-08-25 10:00:01', 2, 'page_view', '/home');
-- ... thousands of rows in one statementIn Laravel, never insert one row per request. Buffer events, then flush a batch from a queued job. Push events to a Redis list in the request:
<?php
namespace App\Services;
use Illuminate\Support\Facades\Redis;
class EventBuffer
{
public const KEY = 'events_buffer';
public function push(int $userId, string $type, string $url): void
{
Redis::rpush(self::KEY, json_encode([
'event_time' => now('UTC')->format('Y-m-d H:i:s'),
'user_id' => $userId,
'event_type' => $type,
'url' => $url,
]));
}
}Flush in a queued job that runs on a schedule:
<?php
namespace App\Jobs;
use App\Models\Clickhouse\Event;
use App\Services\EventBuffer;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Support\Facades\Redis;
class FlushEventsJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable;
public function handle(): void
{
// Take up to 10,000 items in one round trip, atomically.
[$raws] = Redis::transaction(function ($tx) {
$tx->lrange(EventBuffer::KEY, 0, 9999);
$tx->ltrim(EventBuffer::KEY, 10000, -1);
});
if ($raws === []) {
return;
}
$rows = [];
foreach ($raws as $raw) {
$row = json_decode($raw, true);
$rows[] = [$row['event_time'], $row['user_id'], $row['event_type'], $row['url']];
}
Event::insertBulk($rows, ['event_time', 'user_id', 'event_type', 'url']);
}
}Schedule the job in routes/console.php:
use App\Jobs\FlushEventsJob;
use Illuminate\Support\Facades\Schedule;
Schedule::job(new FlushEventsJob())->everyMinute()->withoutOverlapping();Read the list with one LRANGE plus one LTRIM inside a Redis transaction. A loop of 10,000 single LPOP calls is 10,000 network round trips. withoutOverlapping() stops two flushes from running at once if a batch takes longer than a minute.
You can also let ClickHouse batch small inserts with async inserts. Set them per query:
INSERT INTO events SETTINGS async_insert = 1, wait_for_async_insert = 1
VALUES ('2026-08-25 10:00:00', 1, 'page_view', '/pricing');async_insert = 1 tells the server to hold the insert in a memory buffer, then write one part for many inserts. wait_for_async_insert = 1 makes the client wait until the data reaches disk, so you still get an error if the write fails. Use wait_for_async_insert = 0 only if you accept a small risk of data loss on a crash.
Since 26.3 LTS, async inserts are on by default. Batching in your own job still gives you the most control.
Tips and tricks
- Aim for at least a few thousand rows per insert. Tens of thousands is fine.
- The default granule size is 8,192 rows. A granule is the smallest block of rows ClickHouse reads. Batches far larger than one granule are healthy.
- If you already batch in the app, do not also turn on async inserts. Async inserts add latency you do not need.
- Watch part counts with
SELECT table, count() FROM system.parts WHERE active GROUP BY table.
The next problem
A support agent reports a wrong event_type on one row. You try to fix it with ALTER TABLE events UPDATE ..., the way you would run UPDATE in MySQL. The command returns at once but the row does not change. Minutes later it still looks wrong.
Quiz
02 Update is not free
Where we are
You batch inserts from a queued job. Part counts are healthy. Data flows into events. You added an orders_analytics table for order facts. Now you need to correct one wrong row.
The problem
You run the classic mutation:
ALTER TABLE orders_analytics UPDATE status = 'refunded' WHERE order_id = 42;The statement returns fast, but the row does not change at once. You check system.mutations and see the work is still running in the background:
SELECT mutation_id, command, is_done FROM system.mutations
WHERE table = 'orders_analytics';┌─mutation_id─┬─command──────────────────────────────────┬─is_done─┐
│ mutation_3 │ UPDATE status = 'refunded' WHERE order_id = 42 │ 0 │
└─────────────┴──────────────────────────────────────────┴─────────┘On a large table this rewrites whole column files and takes a long time.
Why it happens
ALTER TABLE ... UPDATE and ALTER TABLE ... DELETE are mutations. A mutation rewrites every affected part. ClickHouse stores columns in immutable parts, so it cannot change a value in place. It must write new parts. This is heavy and asynchronous.
ClickHouse also has lightweight statements. Lightweight DELETE is generally available (GA). It marks rows as deleted and removes them at the next merge. Lightweight UPDATE (UPDATE ... SET ... WHERE) is Beta. It writes a small patch part instead of whole columns. Test it on your own data before you rely on it.
Even with these tools, ClickHouse is built for append, not for row-by-row edits. The best design avoids updates.
The fix
Prefer append plus ReplacingMergeTree. Instead of editing a row, insert a new version of it. Give the table a version column and let ClickHouse keep the latest version during merges:
CREATE TABLE orders_analytics
(
order_id UInt64,
user_id UInt64,
status LowCardinality(String),
total Decimal(12, 2),
updated_at DateTime,
version UInt64
)
ENGINE = ReplacingMergeTree(version)
ORDER BY (order_id);To change the status, append a row with a higher version:
INSERT INTO orders_analytics VALUES
(42, 7, 'refunded', 19.99, now(), 2);For a real delete of a few rows, use the lightweight DELETE:
DELETE FROM orders_analytics WHERE order_id = 42;The Beta lightweight update looks like this:
-- Beta in 26.x. Check the docs before production use.
UPDATE orders_analytics SET status = 'refunded' WHERE order_id = 42;In Laravel, stop calling update(). Insert a new version instead:
<?php
namespace App\Models\Clickhouse;
use PhpClickHouseLaravel\BaseModel;
class OrderAnalytics extends BaseModel
{
protected $table = 'orders_analytics';
}use App\Models\Clickhouse\OrderAnalytics;
OrderAnalytics::insertBulk([
[42, 7, 'refunded', '19.99', now('UTC')->format('Y-m-d H:i:s'), 2],
], ['order_id', 'user_id', 'status', 'total', 'updated_at', 'version']);Tips and tricks
- Use lightweight
DELETEfor small, rare deletes. Use partition drops for large deletes. Section 5 shows how. - Lightweight
UPDATEis Beta. Name it as Beta in your team notes. - Never build a workflow that updates one row per user action. Append and let merges do the work.
- Use the MySQL
updated_attimestamp asversion. It only moves forward. - Two updates in the same second get the same version, and then
ReplacingMergeTreekeeps whichever it saw last. If that can happen, makeupdated_ataDATETIME(6)in MySQL and send microseconds as the version, or keep an explicitversioncounter column in MySQL.
The next problem
You append new versions of orders. Then you run SELECT * FROM orders_analytics WHERE order_id = 42 and get two rows: the old one and the new one. You expected a unique primary key to keep only one.
Quiz
03 The primary key is not unique
Where we are
You append new versions of orders into a ReplacingMergeTree. Your flush job batches inserts. You expect one row per order_id. A query returns duplicates and your revenue totals are too high.
The problem
SELECT order_id, status, version FROM orders_analytics WHERE order_id = 42;┌─order_id─┬─status───┬─version─┐
│ 42 │ paid │ 1 │
│ 42 │ refunded │ 2 │
└──────────┴──────────┴─────────┘Both versions are present. In MySQL a PRIMARY KEY (order_id) rejects the second row. ClickHouse kept both.
Why it happens
The ClickHouse primary key is not a uniqueness constraint. It is a sparse index. The index stores one entry per granule, not one entry per row. A granule is a block of 8,192 rows by default. The index maps the first key value of each granule to its position, so ClickHouse can skip granules that cannot match a WHERE filter.
Because the index is sparse and does not enforce uniqueness, ClickHouse never checks for duplicate keys on insert. It accepts every row.
ReplacingMergeTree(version) removes duplicates during background merges. It keeps the row with the highest version for each ORDER BY key. Merges happen on the schedule of ClickHouse. Until a merge runs, both rows are visible. This is eventual, not immediate.
The fix
To get a correct answer before merges finish, force deduplication at read time with FINAL:
SELECT order_id, status, version
FROM orders_analytics FINAL
WHERE order_id = 42;┌─order_id─┬─status───┬─version─┐
│ 42 │ refunded │ 2 │
└──────────┴──────────┴─────────┘FINAL applies merge logic during the query. It is correct but it costs CPU.
For aggregations, argMax is often faster than FINAL. It picks the value tied to the highest version inside a GROUP BY:
SELECT
order_id,
argMax(status, version) AS status,
argMax(total, version) AS total
FROM orders_analytics
GROUP BY order_id;argMax(status, version) returns the status from the row with the largest version. It deduplicates and aggregates in one pass.
You can also stop exact duplicate batches at insert time. ClickHouse keeps a hash of each inserted block. If your job retries the same batch, the second insert is dropped:
CREATE TABLE orders_analytics
(
order_id UInt64,
user_id UInt64,
status LowCardinality(String),
total Decimal(12, 2),
updated_at DateTime,
version UInt64
)
ENGINE = ReplacingMergeTree(version)
ORDER BY (order_id)
SETTINGS non_replicated_deduplication_window = 100;In Laravel, use FINAL for a single record and argMax for reports:
use App\Models\Clickhouse\OrderAnalytics;
$order = OrderAnalytics::select(['order_id', 'status', 'version'])
->from('orders_analytics FINAL')
->where('order_id', 42)
->getRows();
$report = OrderAnalytics::select(['order_id'])
->selectRaw('argMax(status, version) AS status')
->selectRaw('argMax(total, version) AS total')
->groupBy('order_id')
->getRows();Tips and tricks
ReplacingMergeTreededuplicates only rows with the same fullORDER BYkey.- Design the
ORDER BYkey to hold the natural key you want to deduplicate on. - For a soft delete, add a column
is_deleted UInt8 DEFAULT 0and declareENGINE = ReplacingMergeTree(version, is_deleted). Insert a new version withis_deleted = 1to delete.FINALthen drops rows whose latest version is deleted. Both the column and the engine argument are needed. - Do not run
OPTIMIZE TABLE ... FINALin production on a big table to fix duplicates. It is heavy. UseFINALorargMaxat read time.
The next problem
Your dashboards read events by user and by day. Queries are still slow. You look at the migration and see ORDER BY (id), copied from a MySQL primary key. The sort key does not match how you filter.
Quiz
04 The ORDER BY mistake
Where we are
Deduplication works with ReplacingMergeTree, FINAL, and argMax. Your events table still has ORDER BY (id) with an id column you added like a MySQL primary key. Dashboard queries filter by user_id and event_time and read most of the table.
The problem
SELECT count() FROM events
WHERE user_id = 7 AND event_time >= '2026-08-01';1 row in set. Elapsed: 4.812 sec. Processed 210.30 million rowsThis scans almost every granule. EXPLAIN indexes = 1 shows why:
EXPLAIN indexes = 1
SELECT count() FROM events WHERE user_id = 7;PrimaryKey
Keys: id
Condition: true
Parts: 24/24
Granules: 25672/25672The primary key id cannot prune granules for a user_id filter.
Why it happens
ClickHouse uses the ORDER BY key to build the sparse primary index. It can skip granules only when your WHERE filter matches the leading columns of that key. If the key is id but you filter on user_id, the index cannot help. ClickHouse reads everything.
Design the key from your WHERE patterns. Put the column that almost every query filters on first. Cardinality means the number of distinct values in a column. When two columns are both filtered in most queries, put the lower-cardinality one first: it groups many rows under each index entry, and the higher-cardinality one then narrows inside those groups. Column order also shapes compression, so rows with equal leading values sit together.
A high-cardinality leading column is fine when nearly every query filters on it. user_id has millions of values, but a filter on it skips almost every granule, which is the point. The mistake is to lead with a column you rarely filter on, such as id. The rule "low cardinality first" applies to the order between filtered columns. It does not mean "put a low-cardinality column first even if you never filter on it".
The fix
Design the key from real queries. Most queries filter by user_id, then narrow by time:
CREATE TABLE events
(
event_time DateTime,
user_id UInt64,
event_type LowCardinality(String),
url String
)
ENGINE = MergeTree
ORDER BY (user_id, event_time);Now a filter on user_id skips all granules outside that user. A range on event_time then narrows further, because rows are sorted by time within each user.
PrimaryKey
Keys: user_id, event_time
Parts: 3/24
Granules: 14/25672Notice what you gave up. The daily revenue report from section 00 filters on time only. With (user_id, event_time) there is no match on the leading column, so that query still reads every granule. ClickHouse sorts a table one way. That is not a bug in the key. Time-only dashboards get their own fix in section 10 with materialized views and projections, and you will see the full scan again there.
You can set a PRIMARY KEY that is a prefix of the ORDER BY key. This keeps the index small while sorting on more columns:
ENGINE = MergeTree
PRIMARY KEY (user_id)
ORDER BY (user_id, event_time, event_type)The PRIMARY KEY must be a prefix of the ORDER BY. The extra ORDER BY columns help sorting and compression only.
Check the parts and their granules with system.parts:
SELECT name, rows, marks, round(rows / marks) AS rows_per_granule
FROM system.parts
WHERE table = 'events' AND active;You cannot change ORDER BY on a table that already holds data. Every later section that changes the events definition uses this recipe. Create the new table, copy the rows, swap the names:
-- 1. New table with the new key. Same columns.
CREATE TABLE events_new
(
event_time DateTime,
user_id UInt64,
event_type LowCardinality(String),
url String
)
ENGINE = MergeTree
ORDER BY (user_id, event_time);
-- 2. Copy old rows in date ranges. One INSERT per month keeps memory flat.
INSERT INTO events_new SELECT * FROM events
WHERE event_time >= '2026-07-01' AND event_time < '2026-08-01';
-- ... repeat per month up to a cutoff you choose
-- 3. Swap names in one atomic step, then drop the old table.
EXCHANGE TABLES events AND events_new;
DROP TABLE events_new;Pause the flush job while you copy, or copy up to a cutoff time, swap, and then copy the rows after the cutoff. Otherwise rows inserted during the copy land in the old table and are lost at the drop. EXCHANGE TABLES needs an Atomic database, which is the default.
In Laravel, write the migration:
<?php
use PhpClickHouseLaravel\Migration;
return new class extends Migration
{
public function up(): void
{
static::write('
CREATE TABLE events
(
event_time DateTime,
user_id UInt64,
event_type LowCardinality(String),
url String
)
ENGINE = MergeTree
ORDER BY (user_id, event_time)
');
}
public function down(): void
{
static::write('DROP TABLE events');
}
};Tips and tricks
- Do not copy a MySQL primary key into
ORDER BY. Design the key from your filters. - A read that touches 0.1% of rows uses the index well. A read that touches 80% does not.
- Sorted data compresses better, so a good
ORDER BYalso shrinks storage. - Use
EXPLAIN indexes = 1to confirm granule pruning before you trust a change. - You cannot change
ORDER BYon existing data. Use the copy-and-exchange recipe above. - One table serves one sort order well. For a second access pattern, use a projection or a materialized view (section 10), not a second copy you maintain by hand.
The next problem
You read that partitioning speeds up queries. You add PARTITION BY user_id to make it faster. Part counts explode and TOO_MANY_PARTS returns. You try PARTITION BY toHour(event_time) and it gets worse.
Quiz
05 Partitioning is not an index
Where we are
Your ORDER BY key on events is (user_id, event_time) and queries are fast. orders_analytics uses ReplacingMergeTree(version). You added a partition key to gain more speed. Part counts blew up and inserts fail again.
The problem
CREATE TABLE events
(
event_time DateTime,
user_id UInt64,
event_type LowCardinality(String),
url String
)
ENGINE = MergeTree
PARTITION BY user_id -- one partition per user
ORDER BY (user_id, event_time);DB::Exception: Too many partitions for single INSERT block (more than 100).
The limit is controlled by 'max_partitions_per_insert_block' setting.With one partition per user, ClickHouse creates a part for each user in every batch. Merges cannot keep up. TOO_MANY_PARTS comes back. PARTITION BY toHour(event_time) also creates far too many partitions.
Why it happens
A partition is a group of parts that share the same value of the partition expression. ClickHouse never merges parts across partitions. A high-cardinality partition key makes millions of small partitions, each with its own parts. This raises merge pressure and metadata cost.
Partitioning is for data management, not for query speed. The sort key gives you speed. Partitions give you cheap bulk delete and data lifecycle control. Partition by month for almost everything and keep the partition count low.
The fix
Partition by month. Apply it with the copy-and-exchange recipe from section 04, because the partition key cannot change in place either:
CREATE TABLE events
(
event_time DateTime,
user_id UInt64,
event_type LowCardinality(String),
url String
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(event_time) -- one partition per month
ORDER BY (user_id, event_time);toYYYYMM(event_time) makes one partition per month. Keep active partitions under about 1,000 per table.
Use TTL to expire old data. TTL means time to live. It is a rule that deletes rows after a set age. When the rule matches the partition key, ClickHouse drops whole parts instead of rewriting rows:
ALTER TABLE events MODIFY TTL event_time + INTERVAL 18 MONTH DELETE;
ALTER TABLE events MODIFY SETTING ttl_only_drop_parts = 1;ttl_only_drop_parts = 1 tells ClickHouse to wait until every row in a part expires, then drop the whole part.
Drop a month by hand in milliseconds:
ALTER TABLE events DROP PARTITION '202501';List partitions and their sizes:
SELECT partition, sum(rows) AS rows, formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
WHERE table = 'events' AND active
GROUP BY partition
ORDER BY partition;In Laravel, put the partition and TTL in the migration:
<?php
use PhpClickHouseLaravel\Migration;
return new class extends Migration
{
public function up(): void
{
static::write('
CREATE TABLE events
(
event_time DateTime,
user_id UInt64,
event_type LowCardinality(String),
url String
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(event_time)
ORDER BY (user_id, event_time)
TTL event_time + INTERVAL 18 MONTH DELETE
SETTINGS ttl_only_drop_parts = 1
');
}
public function down(): void
{
static::write('DROP TABLE events');
}
};Tips and tricks
- Never partition by
user_id, a UUID, or a raw timestamp. - A healthy partition holds from a few gigabytes to a few hundred gigabytes.
- Keep the partition key and the TTL rule on the same time unit, so partition drops stay cheap.
- Dropping a partition removes data in milliseconds, far faster than
DELETE. - A batch that spans more than 100 partitions fails by default. Monthly partitions keep each batch inside one or two partitions.
The next problem
A dashboard must show the plan and country of each user next to each event. You join events to a users table with a million rows. The query fails with MEMORY_LIMIT_EXCEEDED.
Quiz
06 The JOIN that runs out of memory
Where we are
Events are partitioned by month with a good sort key and TTL. A dashboard needs the plan and country of each user beside each event. You copied a users_raw table into ClickHouse with a million rows. You join events to it and run out of memory.
The problem
SELECT u.country, count() AS events
FROM users_raw AS u
JOIN events AS e ON e.user_id = u.user_id
WHERE e.event_time >= today()
GROUP BY u.country;DB::Exception: Memory limit (for query) exceeded: would use 9.31 GiB
(attempt to allocate chunk of 4194304 bytes), maximum: 9.31 GiB.
(MEMORY_LIMIT_EXCEEDED)The events table sits on the right side of the join. The join tries to hold it in memory.
Why it happens
The default join algorithm builds a hash table in memory from the right-hand table. It then streams the left-hand table through it. The size of the right table sets the memory cost. If you put the large table on the right, you use a lot of memory.
Since ClickHouse 24.12 the planner tries to put the smaller table on the right for two-table joins. Still learn the rule, because you can guide the planner and reason about the plan.
For a lookup like user attributes, a dictionary is better than a join. A dictionary is an in-memory key-value store that ClickHouse refreshes from a source table on a schedule. A lookup against a dictionary is very fast.
The fix
First, put the small table on the right when you write a join by hand:
SELECT u.country, count() AS events
FROM events AS e
JOIN users_raw AS u ON e.user_id = u.user_id -- users_raw is smaller
WHERE e.event_time >= today()
GROUP BY u.country;Better, create a dictionary for user attributes:
CREATE TABLE users_raw
(
user_id UInt64,
country String,
plan String
)
ENGINE = MergeTree
ORDER BY (user_id);
CREATE DICTIONARY users_dict
(
user_id UInt64,
country String,
plan String
)
PRIMARY KEY user_id
SOURCE(CLICKHOUSE(TABLE 'users_raw'))
LAYOUT(HASHED())
LIFETIME(MIN 300 MAX 600);Look up values with dictGet:
SELECT
dictGet('users_dict', 'country', user_id) AS country,
count() AS events
FROM events
WHERE event_time >= today()
GROUP BY country;Beware the one-to-many trap. A dictionary returns exactly one value per key. Do not use it where a key maps to many rows, such as one user with many orders.
Or denormalize at insert time. Write country and plan into events in your buffer service, so you never join at read time:
ALTER TABLE events
ADD COLUMN country LowCardinality(String),
ADD COLUMN plan LowCardinality(String);<?php
namespace App\Services;
use App\Models\User;
use Illuminate\Support\Facades\Redis;
class EventBuffer
{
public const KEY = 'events_buffer';
public function push(User $user, string $type, string $url): void
{
Redis::rpush(self::KEY, json_encode([
'event_time' => now('UTC')->format('Y-m-d H:i:s'),
'user_id' => $user->id,
'event_type' => $type,
'url' => $url,
'country' => $user->country, // denormalized
'plan' => $user->plan, // denormalized
]));
}
}If you must join two large tables, pick an algorithm that fits your memory. Set join_algorithm:
SELECT count()
FROM events AS e
JOIN orders_analytics AS o ON e.user_id = o.user_id
SETTINGS join_algorithm = 'grace_hash';grace_hash spills to disk instead of failing. full_sorting_merge fits when both sides are sorted on the join key.
Tips and tricks
- Aim for at most 3 or 4 joins per query. Denormalize or use dictionaries to cut the rest.
- A lookup with
dictGetis much faster than a hash join. ClickHouse benchmarks report large gains. Treat them as vendor claims. - Denormalizing at insert time trades a little storage for fast reads. In ClickHouse that trade is usually worth it.
- Use
SYSTEM RELOAD DICTIONARY users_dictafter a bulk change tousers_raw.
The next problem
You open the events table definition and wince. Every column is Nullable(String), and the timestamp is DateTime64(9). Storage is large and scans are slow. The types are the problem.
Quiz
07 Types matter
Where we are
Joins are cheap thanks to users_dict and denormalized country and plan columns. While adding country and plan, a teammate "made everything safe": every non-key column became Nullable(String) and event_time became DateTime64(9). Storage tripled and every scan reads more than it should.
The problem
CREATE TABLE events
(
event_time DateTime64(9),
user_id UInt64,
event_type Nullable(String),
url Nullable(String),
country Nullable(String),
plan Nullable(String),
revenue Nullable(String)
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(event_time)
ORDER BY (user_id, event_time);SELECT formatReadableSize(sum(bytes_on_disk)) FROM system.parts
WHERE table = 'events' AND active;┌─size──────┐
│ 41.20 GiB │
└───────────┘DateTime64(9) stores nanoseconds you never use. Every Nullable column adds a hidden mask column. revenue as a string cannot be summed without a cast. Only user_id escaped, because ClickHouse refuses a Nullable column in the sort key unless you set allow_nullable_key = 1. Do not set it.
Why it happens
ClickHouse stores each column apart and compresses it. The type controls the size and the speed. A small, fixed type reads and compresses better than a wide or nullable one.
Nullable adds a separate map that marks null rows. This costs storage and slows reads. Store a default value instead of null where you can.
LowCardinality(String) replaces repeated strings with a small dictionary of ids. It helps below about 10,000 distinct values. Enum stores a fixed set of labels as integers. Decimal stores money exactly. Date stores a day. DateTime stores seconds. DateTime64(3) stores milliseconds. Pick the smallest precision you need.
The JSON type stores flexible objects with typed paths. It is GA since 25.3. The old Object('json') type is removed. Array stores a list of values. Map stores keys and values of one type.
The fix
Choose tight types and add codecs. This is a new table definition, so apply it with the copy-and-exchange recipe from section 04:
CREATE TABLE events
(
event_time DateTime CODEC(DoubleDelta, ZSTD(1)),
user_id UInt64 CODEC(Delta, LZ4),
event_type Enum8('page_view' = 1, 'click' = 2, 'purchase' = 3),
url String,
country LowCardinality(String),
plan LowCardinality(String),
revenue Decimal(12, 2),
tags Array(LowCardinality(String)),
attributes Map(String, String),
payload JSON
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(event_time)
ORDER BY (user_id, event_time);A codec transforms a column before compression. Delta and DoubleDelta shrink values that grow in steps, such as ids and timestamps. Gorilla fits floats that change slowly. ZSTD is a strong general compressor. LZ4 is fast to decode. Chain them as CODEC(encoder, compressor).
After the change the same data is much smaller:
┌─size──────┐
│ 6.84 GiB │
└───────────┘Read a JSON path by name:
SELECT payload.utm_source AS source, count()
FROM events
GROUP BY source;In Laravel, cast scalar values before insert so types match. insertBulk is fine for scalars and, with the InsertArray helper, for Array columns:
use PhpClickHouseLaravel\InsertArray;
$rows[] = [
$row['event_time'],
(int) $row['user_id'],
$row['event_type'], // one of the Enum labels
$row['url'],
$row['country'],
$row['plan'],
(string) $row['revenue'], // Decimal sent as string keeps precision
new InsertArray($row['tags']), // Array(LowCardinality(String))
];
Event::insertBulk($rows, [
'event_time', 'user_id', 'event_type', 'url', 'country', 'plan',
'revenue', 'tags',
]);For Map and JSON columns, do not rely on the builder to guess a literal. Send the batch as JSONEachRow through the low-level client. ClickHouse parses each line into the column types itself, so a PHP assoc array becomes a Map and a nested object becomes JSON:
use Illuminate\Support\Facades\DB;
$lines = [];
foreach ($rows as $row) {
$lines[] = json_encode([
'event_time' => $row['event_time'],
'user_id' => (int) $row['user_id'],
'event_type' => $row['event_type'],
'url' => $row['url'],
'country' => $row['country'],
'plan' => $row['plan'],
'revenue' => (string) $row['revenue'],
'tags' => $row['tags'], // ["a","b"] -> Array
'attributes' => $row['attributes'], // {"k":"v"} -> Map
'payload' => $row['payload'], // nested object -> JSON
]);
}
DB::connection('clickhouse')->getClient()->write(
"INSERT INTO events FORMAT JSONEachRow\n" . implode("\n", $lines)
);Send dates as 'Y-m-d H:i:s' strings in UTC and Decimal as strings. Numbers wider than 53 bits, such as UInt64, and all Decimal values come back from the HTTP interface as strings. Cast them in PHP before you do math.
Tips and tricks
- Avoid
Nullable. Use0,'', or a default instead, unless null has real meaning. - Use
LowCardinality(String)for columns likecountryandplanwhen distinct values stay under about 10,000. - Use
Enumfor a small, fixed set of labels. Since 26.6 you can add a value withALTER TABLE ... MODIFY COLUMN ... ADD ENUM VALUESand no data rewrite. - For money, use
Decimal, neverFloat. - Check compression per column with
system.columns. Comparedata_compressed_bytesanddata_uncompressed_bytes.
The next problem
You write SELECT * in a dashboard query and read every column from the wide table. You also add FINAL to every query to be safe. Both habits read far more data than the dashboard needs.
Quiz
08 SELECT only what you read
Where we are
Your events table has tight types and codecs. orders_analytics is a ReplacingMergeTree. Dashboard queries still use SELECT * and wrap everything in FINAL. Both read more data than needed.
The problem
SELECT * FROM events FINAL WHERE user_id = 7;12 rows in set. Elapsed: 1.902 sec. Processed 3.12 million rows, 2.41 GBSELECT * reads every column, including payload and attributes that the dashboard ignores. FINAL forces merge logic on a plain MergeTree that has no duplicates to resolve.
Why it happens
ClickHouse reads only the columns you name. This is column pruning. With SELECT * you lose the main benefit of column storage, because you read all columns from disk.
FINAL is right when you must see the deduplicated state of a ReplacingMergeTree now. It is wrong as a default, because it adds merge work to every query. On an append-only MergeTree with no duplicates, FINAL gives no benefit and costs time.
The fix
Name only the columns you use:
SELECT event_time, event_type
FROM events
WHERE user_id = 7 AND event_time >= today();12 rows in set. Elapsed: 0.014 sec. Processed 16.38 thousand rows, 131 KBUse FINAL only on a ReplacingMergeTree when you need the current state:
SELECT order_id, status FROM orders_analytics FINAL WHERE order_id = 42;Tune heavy queries with settings:
SELECT event_type, count()
FROM events
GROUP BY event_type
SETTINGS max_threads = 8, max_memory_usage = 4000000000;max_threads sets how many threads run the query. max_memory_usage caps memory per query in bytes.
Find slow queries in system.query_log. This table records every finished query:
SELECT query, read_rows, formatReadableSize(read_bytes) AS read, query_duration_ms
FROM system.query_log
WHERE type = 'QueryFinish' AND event_time >= now() - INTERVAL 1 HOUR
ORDER BY query_duration_ms DESC
LIMIT 10;In Laravel, list columns in the builder and pass settings for heavy reports:
use App\Models\Clickhouse\Event;
$rows = Event::select(['event_time', 'event_type'])
->where('user_id', 7)
->where('event_time', '>=', now('UTC')->startOfDay()->format('Y-m-d H:i:s'))
->getRows();
$report = Event::select(['event_type'])
->selectRaw('count() AS c')
->groupBy('event_type')
->settings(['max_threads' => 8, 'max_memory_usage' => 4000000000])
->getRows();Tips and tricks
- Rows come back as PHP arrays of strings for
UInt64,Decimal, and dates. Cast with(int),(string), orCarbon::parseas you read them. - Compare
read_rowsto the total rows of the table. A small ratio means the index worked. - Do not add
FINALjust in case. Add it only when you need deduplicated state now. - Raise
max_threadsfor one heavy report, but watch total server load. - Set
log_commentper query so you can find your dashboard queries insystem.query_log.
The next problem
Your daily revenue totals are off by a few hours. A sale at 1 a.m. Paris time lands on the wrong day. Finance disagrees with your dashboard. The cause is time zones.
Quiz
09 Time zones
Where we are
Your queries read only needed columns and avoid needless FINAL. events and orders_analytics both carry timestamps. Daily totals look wrong by a few hours. A late-night event lands on the wrong day.
The problem
SELECT toStartOfDay(event_time) AS day, sum(revenue) AS revenue
FROM events
GROUP BY day;┌─────────────────day─┬─revenue─┐
│ 2026-08-23 00:00:00 │ 1840.00 │
│ 2026-08-24 00:00:00 │ 2210.50 │ ← finance says 2019.50
└─────────────────────┴─────────┘Your app wrote event_time in Paris local time. The server groups by its own zone. A sale at 01:00 Paris time counts on the wrong day.
Why it happens
A DateTime value maps to a moment in time. If you store local times without a fixed zone, the same clock value means different moments in different places. Grouping by day then depends on the server zone, which is fragile.
Store every timestamp in UTC. UTC is one fixed zone with no daylight saving. Convert to the zone of the user only at the edge, when you show a value or when you group for a report.
The fix
Type the column as UTC. Changing a column type on an existing table is a mutation that rewrites the column, so apply this with the copy-and-exchange recipe from section 04 as well:
CREATE TABLE events
(
event_time DateTime('UTC') CODEC(DoubleDelta, ZSTD(1)),
user_id UInt64 CODEC(Delta, LZ4),
event_type Enum8('page_view' = 1, 'click' = 2, 'purchase' = 3),
url String,
country LowCardinality(String),
plan LowCardinality(String),
revenue Decimal(12, 2),
tags Array(LowCardinality(String)),
attributes Map(String, String),
payload JSON
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(event_time)
ORDER BY (user_id, event_time);Group by day in a chosen zone with the time zone argument:
SELECT
toStartOfDay(event_time, 'Europe/Paris') AS day,
sum(revenue) AS revenue
FROM events
GROUP BY day
ORDER BY day;toStartOfDay(event_time, 'Europe/Paris') converts to Paris time, then truncates to the day. The stored value stays UTC.
In Laravel, set 'timezone' => 'UTC' in config/app.php. Cast to UTC before insert:
'event_time' => now('UTC')->format('Y-m-d H:i:s'),Convert for display in the view layer:
$day = \Carbon\Carbon::parse($row['day'], 'Europe/Paris');
echo $day->format('d.m.Y');Pass the zone of the user into the report. The zone is user input, so it must not be pasted into the SQL string. Bind it:
use Illuminate\Support\Facades\DB;
$tz = auth()->user()->timezone ?? 'UTC';
if (! in_array($tz, timezone_identifiers_list(), true)) {
$tz = 'UTC';
}
$daily = DB::connection('clickhouse')->getClient()->select(
'SELECT toStartOfDay(event_time, :tz) AS day, sum(revenue) AS revenue
FROM events
GROUP BY day
ORDER BY day',
['tz' => $tz]
)->rows();Pass user input safely
ClickHouse has no prepared statements over HTTP, but it has two safe paths, and the package gives you both.
The query builder escapes values you pass to where(), whereIn(), and friends. Raw fragments are your responsibility. Keep selectRaw, whereRaw, and orderByRaw for fixed SQL only:
// Safe: the builder quotes $userId.
Event::select(['event_time', 'url'])->where('user_id', $userId)->getRows();
// Unsafe: string interpolation into raw SQL.
Event::whereRaw("user_id = {$userId}")->getRows();For raw SQL, use the client bindings. Each :name is quoted and escaped by the client before the query is sent:
$db = DB::connection('clickhouse')->getClient();
$rows = $db->select(
'SELECT event_type, count() AS c
FROM events
WHERE user_id = :user_id AND event_time >= :since
GROUP BY event_type',
['user_id' => $userId, 'since' => $since->utc()->format('Y-m-d H:i:s')]
)->rows();ClickHouse also has server-side query parameters written as {name:Type} and sent as param_name on the HTTP request. They are typed and never touch the SQL text. Use them when you talk to the HTTP interface directly without the package:
SELECT event_type, count() AS c
FROM events
WHERE user_id = {user_id:UInt64} AND event_time >= {since:DateTime}
GROUP BY event_type;
-- HTTP: ?param_user_id=7¶m_since=2026-08-01%2000:00:00Identifiers cannot be bound. If a user chooses a column or a table, map the choice to an allow-list in PHP and never place the raw string into SQL.
Tips and tricks
- Store UTC. Convert at the edges only.
- Pass the zone to date functions, such as
toDate(ts, 'Asia/Tokyo'), instead of changing stored data. - Keep the ClickHouse server, MySQL, and Laravel on UTC to avoid surprises.
toDateandtoYYYYMMalso take a time zone argument.- Never interpolate user input into SQL. Use builder methods,
:nameclient bindings, or{name:Type}server parameters.
The next problem
Your main dashboard runs the same daily GROUP BY on every page load. As data grows, each load scans more rows. The dashboard is slow again, even though the query is correct.
Quiz
10 Stop recomputing
Where we are
Times are correct and stored in UTC. Your events table has good types, a good sort key, and monthly partitions. The dashboard runs the same aggregation on every load. The query is right but it repeats heavy work again and again.
The problem
SELECT toDate(event_time, 'UTC') AS day, count() AS views, uniq(user_id) AS users
FROM events
GROUP BY day;540 rows in set. Elapsed: 2.310 sec. Processed 412.00 million rowsEvery page load scans the raw events table. As the table grows, each load reads more and gets slower.
Why it happens
Raw aggregation repeats the same scan. You can compute the result once as data arrives, then read a small table. This is what materialized views do.
An incremental materialized view is a trigger that runs a query on each insert into the source table and writes the result into a target table. AggregatingMergeTree is an engine that stores partial aggregate states and combines them at merge time. The -State combinator writes the state. The -Merge combinator finalizes it at read time. SummingMergeTree is a simpler engine that sums numeric columns with the same key.
A refreshable materialized view runs a full query on a schedule and replaces its target in one step. It fits daily or hourly summaries and full-scan rollups.
A projection is an alternate ordering or pre-aggregation stored inside the same table. ClickHouse picks it for a matching query without a separate table.
The fix
Create the target and the incremental view:
CREATE TABLE events_daily
(
day Date,
views AggregateFunction(count),
users AggregateFunction(uniq, UInt64)
)
ENGINE = AggregatingMergeTree
ORDER BY (day);
CREATE MATERIALIZED VIEW events_daily_mv TO events_daily AS
SELECT
toDate(event_time, 'UTC') AS day,
countState() AS views,
uniqState(user_id) AS users
FROM events
GROUP BY day;uniq is an approximate count. It uses a fixed-size sketch, so its memory stays small, and its error is usually well under 2 percent. That is fine for a page-views chart. It is not fine when finance compares your number to theirs. Use uniqExact for exact numbers and accept that its state grows with the number of distinct values. uniqCombined64 sits between the two. The -State and -Merge combinators work with all of them:
-- exact, larger state
users AggregateFunction(uniqExact, UInt64)
-- write: uniqExactState(user_id) read: uniqExactMerge(users)Read with the -Merge combinator and a GROUP BY:
SELECT day, countMerge(views) AS views, uniqMerge(users) AS users
FROM events_daily
GROUP BY day
ORDER BY day;540 rows in set. Elapsed: 0.009 sec. Processed 1.62 thousand rowsThe view only sees rows inserted after it exists. You must backfill the old rows once. A naive INSERT ... SELECT from the whole table double counts every row that arrives while the backfill runs, because the view also sees those rows. Use a cutoff. Create the view so it only handles rows from the cutoff onward, then backfill everything before the cutoff:
-- 1. The view handles rows from the cutoff onward.
CREATE MATERIALIZED VIEW events_daily_mv TO events_daily AS
SELECT
toDate(event_time, 'UTC') AS day,
countState() AS views,
uniqState(user_id) AS users
FROM events
WHERE event_time >= '2026-09-01 00:00:00' -- cutoff, a little in the future
GROUP BY day;
-- 2. Backfill everything before the cutoff, once.
INSERT INTO events_daily
SELECT toDate(event_time, 'UTC') AS day, countState(), uniqState(user_id)
FROM events
WHERE event_time < '2026-09-01 00:00:00'
GROUP BY day;Pick a cutoff a few minutes in the future so no row can be on both sides. Once the cutoff has passed, the WHERE in the view costs nothing and you can leave it. If your ClickHouse version supports atomic POPULATE for a view with a TO table, that is the same idea done for you under a short lock. Check the docs for your version before you rely on it.
For a plain sum, SummingMergeTree is enough:
CREATE TABLE revenue_daily
(
day Date,
revenue Decimal(38, 2)
)
ENGINE = SummingMergeTree
ORDER BY (day);
CREATE MATERIALIZED VIEW revenue_daily_mv TO revenue_daily AS
SELECT toDate(event_time, 'UTC') AS day, sum(revenue) AS revenue
FROM events
GROUP BY day;Always GROUP BY and sum() at read time, because parts may not be merged yet.
For a scheduled full rollup, use a refreshable view:
CREATE MATERIALIZED VIEW events_weekly
REFRESH EVERY 1 DAY OFFSET 1 HOUR
ENGINE = MergeTree ORDER BY (week_start)
AS SELECT toMonday(day) AS week_start, countMerge(views) AS views
FROM events_daily
GROUP BY week_start;REFRESH EVERY sets the schedule. DEPENDS ON other_view makes one view wait for another. SYSTEM REFRESH VIEW events_weekly forces a refresh now.
Add a projection for a second common sort order:
ALTER TABLE events ADD PROJECTION by_type
(
SELECT event_type, event_time, user_id
ORDER BY (event_type, event_time)
);
ALTER TABLE events MATERIALIZE PROJECTION by_type;In Laravel, read the small rollup table:
<?php
namespace App\Models\Clickhouse;
use PhpClickHouseLaravel\BaseModel;
class EventDaily extends BaseModel
{
protected $table = 'events_daily';
}use App\Models\Clickhouse\EventDaily;
$daily = EventDaily::select(['day'])
->selectRaw('countMerge(views) AS views')
->selectRaw('uniqMerge(users) AS users')
->groupBy('day')
->orderBy('day')
->getRows();Tips and tricks
- Always fill
AggregatingMergeTreefrom a materialized view, not by hand, except for a one-time backfill with a cutoff. uniqis approximate. UseuniqExactwhere the number must match a ledger.- Always
GROUP BYthe key with-Mergeat read time, because parts may be unmerged. - Use an incremental view for running totals. Use a refreshable view for full-scan summaries.
- A materialized view sees only the rows of one insert block. It cannot join across inserts.
- Since 26.1, exactly-once delivery covers all materialized views attached to a table.
The next problem
You want dashboards even faster, so you add skip indexes to every column. Inserts slow down and query times do not improve. You added the wrong tool in the wrong order.
Quiz
11 Skip indexes are the last tool
Where we are
Rollups in events_daily and revenue_daily make dashboards fast. A projection covers queries by event_type. Hoping for more, you add skip indexes to every column of events. Inserts get slower and queries do not improve.
The problem
ALTER TABLE events ADD INDEX idx_url url TYPE bloom_filter GRANULARITY 1;
ALTER TABLE events ADD INDEX idx_country country TYPE set(100) GRANULARITY 1;
ALTER TABLE events ADD INDEX idx_revenue revenue TYPE minmax GRANULARITY 1;
-- ... one index per columnEXPLAIN indexes = 1
SELECT count() FROM events WHERE revenue > 100;Skip
Name: idx_revenue
Type: minmax
Parts: 24/24
Granules: 25102/25672The index skipped almost nothing. Each index still adds write work on every insert.
Why it happens
A skip index stores a small summary per block of granules, such as the lowest and highest value, a set of values, or a bloom filter. At query time ClickHouse reads the summary and skips blocks that cannot match.
A skip index helps only when the column correlates with the primary key. If the values are spread evenly across granules, most blocks can match, so nothing is skipped. revenue is spread across every user and every time. The minmax summary of each block covers the whole range.
Index types serve different filters. minmax fits range filters on columns that change slowly along the sort order. set(N) fits equality on a column with few values per block. bloom_filter fits equality and IN on a column with many distinct values.
The fix
Drop the indexes that do not prune:
ALTER TABLE events DROP INDEX idx_revenue;
ALTER TABLE events DROP INDEX idx_country;Keep a skip index only where it prunes granules. A filter on url for one user is a good case, because one user visits few pages:
ALTER TABLE events ADD INDEX idx_url url TYPE bloom_filter(0.01) GRANULARITY 4;
ALTER TABLE events MATERIALIZE INDEX idx_url;
EXPLAIN indexes = 1
SELECT count() FROM events WHERE user_id = 7 AND url = '/pricing';Skip
Name: idx_url
Type: bloom_filter
Parts: 3/3
Granules: 2/14Follow the tool order. Reach for tools in this sequence:
ORDER BYdesign. Fix the sort key first.- Projections. Add an alternate ordering or pre-aggregation.
- Materialized views. Pre-compute rollups.
- Skip indexes. Use them last, for a column that correlates with the key.
There is no app code change here. Put the index in a migration:
<?php
use PhpClickHouseLaravel\Migration;
return new class extends Migration
{
public function up(): void
{
static::write('ALTER TABLE events ADD INDEX idx_url url TYPE bloom_filter(0.01) GRANULARITY 4');
static::write('ALTER TABLE events MATERIALIZE INDEX idx_url');
}
public function down(): void
{
static::write('ALTER TABLE events DROP INDEX idx_url');
}
};Tips and tricks
- Always confirm with
EXPLAIN indexes = 1before you keep a skip index. - A bloom filter cannot prove absence for a negated filter, so ClickHouse does not use it for
!=orNOT IN. - After you add an index to existing data, run
MATERIALIZE INDEXto build it. - Use
ngrambf_v1ortokenbf_v1forLIKE '%word%'searches on text.
The next problem
The ClickHouse side is fast and correct. Over weeks, MySQL and ClickHouse drift apart. New orders in MySQL do not always reach orders_analytics. You need a reliable way to sync the OLTP data.
Quiz
12 Keeping MySQL and ClickHouse in sync
Where we are
ClickHouse is fast and your rollups are correct. MySQL stays the source of truth for users and orders. Your SyncOrderToClickHouse code runs from a model event. The two stores drift, because not every write reaches ClickHouse.
The problem
-- MySQL
SELECT count() FROM orders WHERE created_at >= '2026-08-01';
-- 184,203-- ClickHouse
SELECT count() FROM orders_analytics FINAL WHERE updated_at >= '2026-08-01';
-- 181,977A new order in MySQL sometimes never appears in orders_analytics. A refund updates MySQL but not ClickHouse. Your dashboards fall behind the real data.
Why it happens
Two databases need a defined sync path. There are three common options.
App-level events. Your Laravel code writes to MySQL and also queues a ClickHouse insert. This is simple and you already do it for events. It misses changes that bypass the app, such as manual SQL or admin tools.
ClickPipes MySQL CDC. CDC means change data capture. It reads the MySQL binary log and streams every change. ClickPipes is the managed ingestion service in ClickHouse Cloud. Its MySQL connector is in Beta and is built on PeerDB.
Debezium or PeerDB, self-hosted. These tools read the binlog and write to ClickHouse without ClickHouse Cloud. They cost more to run.
The old MaterializedMySQL database engine was removed in 24.12. Do not plan around it. One CDC gap matters. A cascading delete in MySQL does not appear as row deletes in the binlog, so a CDC stream can miss those deletes.
The fix
For an app that owns all writes, dual-write from an observer. Write to MySQL, then queue the ClickHouse insert after the transaction commits:
<?php
namespace App\Observers;
use App\Jobs\SyncOrderToClickHouse;
use App\Models\Order;
class OrderObserver
{
public bool $afterCommit = true;
public function saved(Order $order): void
{
SyncOrderToClickHouse::dispatch($order->id);
}
}<?php
namespace App\Jobs;
use App\Models\Clickhouse\OrderAnalytics;
use App\Models\Order;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
class SyncOrderToClickHouse implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable;
public int $tries = 3;
public function __construct(public int $orderId) {}
public function handle(): void
{
$o = Order::findOrFail($this->orderId);
OrderAnalytics::insertBulk([[
$o->id,
$o->user_id,
$o->status,
(string) $o->total,
$o->updated_at->utc()->format('Y-m-d H:i:s'),
(int) $o->updated_at->format('Uu'), // microseconds, no same-second ties
]], ['order_id', 'user_id', 'status', 'total', 'updated_at', 'version']);
}
}Model the target as ReplacingMergeTree(version), so retries and reorders resolve to the latest version. Add a nightly reconcile command that compares counts and re-sends missing ids:
<?php
namespace App\Console\Commands;
use App\Jobs\SyncOrderToClickHouse;
use App\Models\Clickhouse\OrderAnalytics;
use App\Models\Order;
use Illuminate\Console\Command;
class ReconcileOrders extends Command
{
protected $signature = 'clickhouse:reconcile-orders';
public function handle(): void
{
$mysqlIds = Order::where('updated_at', '>=', now()->subDay())->pluck('id');
$chIds = collect(OrderAnalytics::select(['order_id'])
->from('orders_analytics FINAL')
->whereIn('order_id', $mysqlIds->all())
->getRows())->pluck('order_id');
foreach ($mysqlIds->diff($chIds) as $id) {
SyncOrderToClickHouse::dispatch($id);
}
}
}For a hands-off pipeline on ClickHouse Cloud, use ClickPipes MySQL CDC. It handles the first full copy and then continuous replication from RDS, Aurora, Cloud SQL, or your own MySQL. It is Beta today. Enable binlog_format = ROW and GTID mode in MySQL first.
For self-hosted, run Debezium or PeerDB into ClickHouse. Both write into ReplacingMergeTree tables with a version column.
Handle cascading deletes yourself. Because they are not in the binlog, delete child rows in ClickHouse from application logic or the reconcile job.
Tips and tricks
- Use the MySQL
updated_attimestamp asversion, soReplacingMergeTreekeeps the newest state. Declare it asDATETIME(6)in MySQL and send microseconds so two writes in the same second cannot tie. - Make the sync job idempotent. A retried insert of the same version is safe.
- Set
binlog_row_image = FULLin MySQL, so CDC gets every column on update. - Reconcile counts between MySQL and ClickHouse on a schedule to catch drift and missed cascades.
The next problem
Everything works on your one Docker node. You want to ship. You must decide on Cloud or self-hosted, add replication and backups, and set up monitoring before real traffic arrives.
Quiz
13 Shipping to production
Where we are
Your system is correct and fast on one node. It syncs from MySQL with an observer and a reconcile job. Tables: events, orders_analytics, users_raw, users_dict, events_daily, revenue_daily, events_weekly. Now you must run it safely for real traffic.
The problem
A single node has no redundancy. If it fails, you lose queries and risk data. You have no backups and no alerts. You cannot ship like this.
Why it happens
Production needs more than a working query. You need copies of data, a coordination service, tested restores, and metrics you watch.
ReplicatedMergeTree is the engine family that keeps identical copies of each part on several servers. Every MergeTree variant has a Replicated form, such as ReplicatedReplacingMergeTree. Replication needs a coordination service. Keeper is the ClickHouse coordination service. It speaks the ZooKeeper protocol and ships with ClickHouse.
The fix
First, choose Cloud or self-hosted. ClickHouse Cloud runs the servers, storage, and Keeper for you and offers ClickPipes. Self-hosted gives full control at the cost of operations. In Cloud, replication is on by default and you use the SharedMergeTree family without extra setup.
For self-hosted redundancy, use ReplicatedMergeTree:
CREATE TABLE events ON CLUSTER my_cluster
(
event_time DateTime('UTC') CODEC(DoubleDelta, ZSTD(1)),
user_id UInt64 CODEC(Delta, LZ4),
event_type Enum8('page_view' = 1, 'click' = 2, 'purchase' = 3),
url String,
country LowCardinality(String),
plan LowCardinality(String),
revenue Decimal(12, 2),
tags Array(LowCardinality(String)),
attributes Map(String, String),
payload JSON
)
ENGINE = ReplicatedMergeTree(
'/clickhouse/tables/{shard}/events', '{replica}'
)
PARTITION BY toYYYYMM(event_time)
ORDER BY (user_id, event_time)
TTL event_time + INTERVAL 18 MONTH DELETE
SETTINGS ttl_only_drop_parts = 1;Run 3 Keeper nodes for a quorum. Run at least 2 replicas per shard.
Back up to object storage:
BACKUP DATABASE default TO S3(
'https://s3.amazonaws.com/my-bucket/clickhouse/2026-08-25', 'KEY', 'SECRET'
);Restore with RESTORE DATABASE default FROM S3(...). The open-source clickhouse-backup tool also works. Run a monthly restore drill on a staging server to confirm your recovery works.
Monitor with system tables:
SELECT table, absolute_delay FROM system.replicas WHERE absolute_delay > 300;
SELECT table, count() AS parts FROM system.parts WHERE active GROUP BY table;
SELECT query, query_duration_ms FROM system.query_log
WHERE type = 'ExceptionWhileProcessing' AND event_time >= now() - INTERVAL 1 HOUR;
SELECT name, formatReadableSize(free_space) FROM system.disks;Alert on replication delay, part counts, failed queries, and disk use.
Set common server settings in a profile:
CREATE SETTINGS PROFILE laravel_app SETTINGS
max_memory_usage = 8000000000,
max_execution_time = 30,
max_threads = 8;
CREATE USER laravel IDENTIFIED BY 'change-me' SETTINGS PROFILE 'laravel_app';
GRANT SELECT, INSERT ON default.* TO laravel;In Laravel, set retries, timeouts, and a health check:
CLICKHOUSE_HOST=ch.internal
CLICKHOUSE_PORT=8443
CLICKHOUSE_DATABASE=default
CLICKHOUSE_USERNAME=laravel
CLICKHOUSE_PASSWORD=change-me
CLICKHOUSE_HTTPS=true
CLICKHOUSE_TIMEOUT_CONNECT=2
CLICKHOUSE_TIMEOUT_QUERY=30
CLICKHOUSE_RETRIES=2CLICKHOUSE_RETRIES=2 retries a failed HTTP request twice. Retrying an insert is safe because ClickHouse deduplicates an identical block (section 03). Retrying a slow read can pile up load, so keep query timeouts short.
<?php
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Route;
Route::get('/health/clickhouse', function () {
try {
DB::connection('clickhouse')->getClient()->select('SELECT 1')->rows();
return response()->json(['ok' => true]);
} catch (\Throwable $e) {
return response()->json(['ok' => false], 503);
}
});Mark the flush job with $tries = 5 and a backoff, so a short outage does not lose events.
Test it in CI
Confidence comes from tests that hit a real ClickHouse. Mocks cannot tell you that a query scans the wrong granules or that a Map literal failed to parse. Run the same Docker image in CI that you run in production:
# .github/workflows/tests.yml (excerpt)
services:
clickhouse:
image: clickhouse/clickhouse-server:26.3
ports:
- 8123:8123
options: >-
--health-cmd "wget -qO- http://localhost:8123/ping || exit 1"
--health-interval 5s --health-retries 10
env:
CLICKHOUSE_HOST: localhost
CLICKHOUSE_DATABASE: testingGive tests their own database, run the migrations once, and truncate between tests. ClickHouse has no transactions to roll back, so RefreshDatabase does not apply:
<?php
namespace Tests\Concerns;
use Illuminate\Support\Facades\DB;
trait UsesClickHouse
{
protected function setUpUsesClickHouse(): void
{
$db = DB::connection('clickhouse')->getClient();
$db->write('CREATE DATABASE IF NOT EXISTS testing');
foreach (['events', 'orders_analytics', 'events_daily'] as $table) {
$db->write("TRUNCATE TABLE IF EXISTS testing.{$table}");
}
}
}<?php
namespace Tests\Feature;
use App\Models\Clickhouse\Event;
use App\Reports\DailyReport;
use Tests\Concerns\UsesClickHouse;
use Tests\TestCase;
class DailyReportTest extends TestCase
{
use UsesClickHouse;
public function test_it_groups_by_the_users_zone(): void
{
Event::insertBulk([
['2026-08-23 23:30:00', 7, 'purchase', '/checkout', 'FR', 'pro', '10.00'],
], ['event_time', 'user_id', 'event_type', 'url', 'country', 'plan', 'revenue']);
$rows = app(DailyReport::class)->for(userTz: 'Europe/Paris');
$this->assertSame('2026-08-24', substr($rows[0]['day'], 0, 10));
$this->assertSame('10.00', $rows[0]['revenue']);
}
}Run php artisan migrate --database=clickhouse once in the CI job before the tests. Keep one test that runs EXPLAIN indexes = 1 on your heaviest dashboard query and asserts the granule ratio, so a bad sort key fails the build instead of the dashboard.
Sharding
Everything in this tutorial runs on one shard with replicas, and that carries most SaaS analytics workloads far. Sharding splits a table across servers with a Distributed table on top. It adds a sharding key, cross-shard joins, and resharding pain. Do not start there. Add shards when one server cannot hold the data or the merges, and read the Distributed engine docs first.
Tips and tricks
- Use
Replicatedengines for every production table, not plainMergeTree. - Run 3 Keeper nodes and at least 2 replicas per shard.
- A backup you never restored is not a backup. Drill the restore.
- Cache configuration in Laravel, then rebuild the cache after you change ClickHouse env values.
- Watch
system.replicas.absolute_delay. Alert above 300 seconds. - Test against a real ClickHouse in CI. Truncate between tests, because there is no transaction to roll back.
- One shard plus replicas first. Shard only when one server cannot hold the data.
The next problem
None. You are live, replicated, backed up, and monitored. Return to the tips appendix as your system grows.
Quiz
Tips and tricks
The tips from every section, in one place. Each one is a rule an experienced ClickHouse user follows without thinking.
- Never insert one row per request. Buffer in Redis and flush thousands of rows from a
ShouldQueuejob. - Aim for a few thousand to tens of thousands of rows per insert. Each insert makes one part.
- Use
wait_for_async_insert = 1when you use async inserts and want durable acknowledgment. - Since 26.3 LTS, async inserts are on by default. Control batching in your own job anyway.
- The primary key is a sparse index, not a uniqueness constraint. ClickHouse accepts duplicate keys.
- Design the sort key from your
WHEREpatterns: most-filtered column first, then lower cardinality before higher among filtered columns. - To change a sort key, partition key, or column type, create a new table, copy in ranges, and
EXCHANGE TABLES. - Do not copy a MySQL primary key into
ORDER BY. Design the key from your filters. - Set
PRIMARY KEYas a prefix ofORDER BYto keep the index small. - Confirm granule pruning with
EXPLAIN indexes = 1before you trust a change. - Partition by month with
toYYYYMM. Keep the partition count under about 1,000. - Never partition by
user_id, a UUID, or a raw timestamp. - Align TTL with the partition key and set
ttl_only_drop_parts = 1, so ClickHouse drops whole parts. - Drop a partition to delete data in milliseconds instead of running
DELETE. - Put the small table on the right of a join, even though 24.12 and later reorder for you.
- Prefer dictionaries or denormalization over large joins. Mind the one-to-many trap with
dictGet. - Use
join_algorithm = 'grace_hash'when a large join must spill to disk. - Avoid
Nullable. Store a default value instead. - Use
LowCardinality(String)below about 10,000 distinct values. - Use
Enumfor fixed label sets andDecimalfor money. - Use the native
JSONtype. The oldObject('json')type is removed. - Add codecs.
DeltaandDoubleDeltafit ids and timestamps.Gorillafits floats.ZSTDfits general data. - Never write
SELECT *on a wide table. Name your columns. - Use
FINALonly when you need deduplicated state now. - Prefer
argMaxoverFINALfor deduplicated aggregations. - Store timestamps in UTC. Convert at the edges with a zone argument such as
toStartOfDay(ts, 'Europe/Paris'). - Pre-compute rollups with materialized views instead of scanning raw data on each page load.
- Fill
AggregatingMergeTreeonly from a materialized view, and read with-Mergeinside aGROUP BY. - Backfill a materialized view with a time cutoff: the view handles rows after it, the one-time insert handles rows before it.
uniqis approximate. UseuniqExactwhen the number must match a ledger.- Never interpolate user input into SQL. Use builder methods,
:namebindings, or{name:Type}server parameters. - Follow the tool order. Sort key first, then projections, then materialized views, then skip indexes.
- Add a skip index only where the column correlates with the primary key. Verify with
EXPLAIN. - In production use
Replicatedengines, 3 Keeper nodes, at least 2 replicas, and a tested restore. - Test against a real ClickHouse in CI and assert granule pruning on your heaviest query.
MySQL → ClickHouse cheat sheet
| Concept | MySQL | ClickHouse |
|---|---|---|
| Workload | OLTP (transactions) | OLAP (analytics) |
| Storage | Row storage | Column storage |
| Default engine | InnoDB | MergeTree |
| Primary key | Unique constraint | Sparse index, not unique |
| Insert style | One row per statement is fine | Batch thousands per statement |
| Update a row | UPDATE ... WHERE in place | Append a new version + ReplacingMergeTree. Lightweight UPDATE is Beta |
| Delete a row | DELETE ... WHERE | Lightweight DELETE (GA) or drop a partition |
| Upsert | INSERT ... ON DUPLICATE KEY UPDATE | ReplacingMergeTree(version) + FINAL or argMax |
| Speed up filters | Secondary B-tree index | Design ORDER BY, then projections, MVs, skip indexes |
| Partitioning | Optional, for large tables | For data management, PARTITION BY toYYYYMM(ts) |
| Join | Any side, optimizer decides | Right side held in memory. Put the small table right |
| Lookup table | Join | Dictionary with dictGet |
| Flexible column | JSON column | Native JSON type (GA since 25.3) |
| Nullable column | Common | Avoid. Use defaults |
| Low-cardinality text | ENUM or VARCHAR | LowCardinality(String) or Enum |
| Money | DECIMAL | Decimal(P, S) |
| Time zone | Session or server zone | Store DateTime('UTC'). Convert at the edges |
| Pre-aggregation | Summary tables by cron | Incremental or refreshable materialized views |
| Replication | Primary and replica via binlog | ReplicatedMergeTree + Keeper |
| Backup | mysqldump or snapshots | BACKUP ... TO S3(...) or clickhouse-backup |
| MySQL to ClickHouse sync | — | App events, ClickPipes CDC (Beta), Debezium or PeerDB. MaterializedMySQL is removed |
| Ports | 3306 | 8123 HTTP, 9000 native, 9004 MySQL wire |
| Laravel package | Built in | glushkovds/phpclickhouse-laravel or bavix/laravel-clickhouse |
Glossary
- OLAP
- Online analytical processing. Large scans and aggregations.
- OLTP
- Online transaction processing. Single-row reads and writes.
- Column storage
- Each column is stored apart, so a query reads only the columns it needs.
- Part
- One immutable folder of sorted, compressed column files, written per insert.
- Merge
- Background work that combines small parts into bigger parts.
- Granule
- The smallest block of rows the index tracks. 8,192 rows by default.
- Sparse index
- One index entry per granule, not per row. It is not unique.
- Cardinality
- The number of distinct values in a column.
- Mutation
- An
ALTER ... UPDATEorALTER ... DELETEthat rewrites whole parts. - Lightweight DELETE / UPDATE
- Statements that write small patch data.
DELETEis GA.UPDATEis Beta. - ReplacingMergeTree
- An engine that keeps the latest version per key during merges.
- FINAL
- A query modifier that deduplicates at read time.
- argMax
- An aggregate that returns the value tied to the highest version.
- Partition
- A group of parts that share one value of the partition expression.
- TTL
- Time to live. A rule that deletes rows after a set age.
- Dictionary
- An in-memory key-value store for fast one-value-per-key lookups.
- LowCardinality
- A type that stores repeated strings as small ids.
- Codec
- A per-column transform and compressor, such as
DeltaorZSTD. - Materialized view
- A view that writes results to a target table. Incremental on insert or refreshable on a schedule.
- AggregatingMergeTree
- An engine that stores partial aggregate states, used with
-Stateand-Merge. - SummingMergeTree
- An engine that sums numeric columns with the same key.
- Projection
- An alternate ordering or pre-aggregation stored inside a table.
- Skip index
- A per-block summary (minmax, set, bloom_filter) that lets ClickHouse skip granules.
- CDC
- Change data capture. Streaming database changes from a binlog.
- ClickPipes
- The managed ingestion service in ClickHouse Cloud.
- ReplicatedMergeTree
- An engine that copies parts across replicas.
- uniq / uniqExact
- Approximate and exact distinct counts.
uniqhas small fixed memory and a small error.uniqExactis exact and grows with the data. - EXCHANGE TABLES
- An atomic swap of two table names, used to replace a table after a copy.
- Query parameter
- A typed placeholder such as
{id:UInt64}that ClickHouse substitutes server-side, or a:namebinding the PHP client escapes. - Keeper
- The ClickHouse coordination service, a ZooKeeper-compatible replacement.