Demonstrate how you would implement windowing and watermarking in a streaming function, compared to simple grouping in a batch function.

Question

Anonymous · Accepted Answer

In a batch job, I already have the full dataset, so I can group records by key
and time period, then calculate the result once. A live stream is different:
events arrive continuously, sometimes late or out of order, so I have to decide
when a result is ready.

I’d assign each event to a fixed event-time window, say five minutes, and keep
the running count for each key and window. I’d use a watermark to represent how
far event time has progressed, allowing, for example, two minutes for late
arrivals. Once the watermark passes the end of a window, I can finalize its
result and clear its state. Events arriving after that need an explicit policy,
such as dropping them or sending them to a correction path.

So, the big difference is that batch grouping works on a complete dataset.
Streaming windowing groups an ongoing dataset, and watermarking controls how
long we wait for late data before calling a result final.

Demonstrate how you would implement windowing and watermarking in a streaming function, compared to simple grouping in a batch function.

Practice this question with AI

Go Premium

Community Answers

Unlock Full Access

Practice More Questions