/* global React, Hero, SectionLabel, AntiPatterns, BestPractices, Takeaway, Panel */ const { useState, useEffect, useRef, useMemo } = React; /* ============================================================ * Ch1 · Ingest — Flink2Warehouse watermark simulator * Streaming events scatter on (event_time, session); user drags the * watermark line. Late events past the line get dropped. * ============================================================ */ function WatermarkSim() { const [watermark, setWatermark] = useState(720); const [running, setRunning] = useState(true); const [events, setEvents] = useState([]); const [lateness, setLateness] = useState(20); const stageRef = useRef(null); useEffect(() => { if (!running) return; const iv = setInterval(() => { setEvents(prev => { const next = [...prev]; const now = Date.now(); for (let i = 0; i < 3; i++) { const isLate = Math.random() * 100 < lateness; const eventTime = isLate ? 180 + Math.random() * 440 : 620 + Math.random() * 220; next.push({ id: now + i + Math.random(), x: eventTime, y: 60 + Math.random() * 280, born: Date.now(), late: isLate }); } return next.filter(e => Date.now() - e.born < 8000).slice(-90); }); }, 420); return () => clearInterval(iv); }, [running, lateness]); const onDragStart = () => { const rect = stageRef.current.getBoundingClientRect(); const move = (ev) => { const x = (ev.clientX || ev.touches?.[0]?.clientX) - rect.left; setWatermark(Math.max(140, Math.min(940, x * (1000 / rect.width)))); }; const up = () => { window.removeEventListener('mousemove', move); window.removeEventListener('mouseup', up); }; window.addEventListener('mousemove', move); window.addEventListener('mouseup', up); }; const included = events.filter(e => e.x < watermark).length; const dropped = events.filter(e => e.x >= watermark && e.late).length; const mode = watermark > 800 ? 'exact' : watermark > 520 ? 'balanced' : 'real-time'; return (
event time → session {[200,400,600,800].map(x => ( t−{Math.round((1000-x)/100)}m ))} {events.map(e => { const age = (Date.now() - e.born) / 8000; const inc = e.x < watermark; return ( ); })} DROPPED WATERMARK: drag
Included
{included}
before watermark
Late · dropped
{dropped}
excluded from window
Watermark
t−{((1000-watermark)/100).toFixed(1)}m
event time
Mode
{mode}
{mode === 'exact' ? 'low loss · high delay' : mode === 'balanced' ? 'default' : 'real-time · lossy'}
Watermark positiont−{((1000-watermark)/100).toFixed(1)}m
setWatermark(+e.target.value)} /> drag the slider or the blue line above
Network lateness{lateness}%
setLateness(+e.target.value)} /> % of events arriving late
); } /* ClickHouse vs Snowflake concept cards */ function IngestStreams() { return (
ClickHouse
Sampled · real-time
1-in-N rows. Sub-minute freshness. Perfect for "is something on fire?" Never sum raw sample counts and expect truth: always multiply by the sample rate.
Snowflake
Exact · batch
100% of rows, deterministic. Hours of delay. What you use for finance, policy, and anything a regulator might subpoena.
); } function Ch1_Ingest({ chapter }) { return ( <> ClickHouseLoggersCDC' }, { k: 'Sink', v: 'Snowflake · Iceberg tables' }, { k: 'Hard problem', v: 'late arrivals & clock skew' }, ]} />
Two clocks, one event

Event time vs processing time.

Every event carries two timestamps. Event time is when it happened: a tap on a phone, an ad impression rendered. Processing time is when your stream actually saw it. Mobile clients, retries, weak cell signal, and simple clock skew make these diverge. Any system that pretends they're the same ships the wrong numbers.

Modern logger tiers (Kafka + Flink CDC) emit events into ClickHouse within seconds of event time; Snowflake lands them as Parquet minutes to hours later. Between those two, the watermark decides which late events get to join the aggregate and which get dropped.

The compromise, visualized

When do you stop waiting?

Drag the blue line. Green dots are on-time events; amber dots arrived late. Anything past the watermark is dropped: gone from Snowflake. Too tight and you lose real data; too loose and dashboards lag by an hour. There is no free correct answer.

In production, watermarks are typically 15–60 minutes behind real-time: long enough to absorb mobile stragglers, short enough that dashboards feel live. Finance-critical pipelines push the watermark out to hours and accept the delay.

Two stores, two jobs

ClickHouse answers "now". Snowflake answers "exactly".

The rule is not "pick one." It's use both, and know which question each one answers. ClickHouse is for live debugging, oncall, and broad strokes. Snowflake is for contracts, finance, and any number that has to survive a regulator.

The canonical flink2warehouse SQL
flink2warehouse_events.sqlSpark
INSERT OVERWRITE TABLE events_daily PARTITION (ds='<DATEID>') SELECT user_id, event_name, event_time, COUNT(*) AS n FROM clickhouse_events WHERE event_time BETWEEN '<DATEID>' AND '<DATEID> 23:59:59' AND processing_time < '<DATEID+1> 00:30:00' -- watermark: 30m grace GROUP BY user_id, event_name, event_time;` }} />
Summing raw ClickHouse counts without the sample rate. A 1:1000 sample reports 1000× fewer impressions. Always multiply by sample_rate.", "Watermark = now. You'll drop every mobile event that round-trips through a cell tower. Give it at least 15 minutes of grace.", "Treating flink2warehouse as eventually consistent. It isn't. Once the window closes, late events are gone: no backfill, no retry will save them.", "Reading NOW() inside an ingest job. A backfill in May for last Tuesday becomes unreproducible. Use <DATEID>.", ]} /> both timestamps on every event: event_time (device) and processing_time (server). The gap between them is your watermark budget.", "Budget your watermark from the p99 network delay for mobile, not the median. 30 minutes is a sane starting point.", "Dashboards that demand real-time: read ClickHouse, annotate them \"sampled\". Anything cited in a deck: read Snowflake.", ]} /> event time and processing time. Late arrivals live in the gap between them.", "The watermark is the price you pay to close a window. Tighter = lossier. Looser = later.", "ClickHouse is sampled and fast; Snowflake is exact and slow. Use both: know which question each one answers.", ]} /> ); } window.Ch1_Ingest = Ch1_Ingest;