// WAR STORY · WORKING DRAFT

Debugging a UTC vs EAT epoch boundary bug in a telecom campaign engine

2026-06-10 · Kafka · TypeScript · timezones

This post is a working draft — the structure is real, and the full investigation details are being written up.

Tanzania runs on East Africa Time, UTC+3, with no daylight saving. That should make time easy. It doesn't — it makes time quietly wrong, because UTC+3 means your midnight and the server's midnight disagree by exactly three hours, every single day, forever.

This is the story of a campaign engine that worked perfectly for 21 hours a day.

The symptom

A daily campaign behaved strangely in a specific window: customers who qualified late in the evening were treated as if they belonged to the next day's campaign run — daily counters reset under them, eligibility checks read the wrong bucket, and the numbers between two reporting systems refused to reconcile.

The giveaway, in hindsight: every anomaly clustered between 21:00 and 23:59 EAT. Exactly three hours. Exactly the UTC offset.

The investigation

The pipeline processes events through seven stages, and "what day is it?" gets asked more than once along the way — at ingestion, at eligibility, and again at reporting.

// Stage A — bucketing by server time (UTC on the cluster)
const day = new Date(event.timestamp).toISOString().slice(0, 10);
 
// Stage B — bucketing by local business day (EAT, UTC+3)
const day = formatInTimeZone(event.timestamp, "Africa/Dar_es_Salaam", "yyyy-MM-dd");

Same epoch in, different "day" out — but only for timestamps between 21:00:00 and 23:59:59 EAT, when the local date has already rolled over but the UTC date hasn't.

The root cause

Two stages bucketed the same event into different days. Each was internally correct. The system as a whole was wrong: a classic distributed-systems lesson wearing a timezone costume — correctness is a property of the whole pipeline, not of any one stage.

The fix

One rule, enforced everywhere: the business day is defined in EAT, and the conversion happens in exactly one place.

/** The ONLY function allowed to answer "which business day?" */
export function businessDay(epochMs: number): string {
  return formatInTimeZone(epochMs, "Africa/Dar_es_Salaam", "yyyy-MM-dd");
}

Everything downstream consumes the bucket; nothing recomputes it.

Lessons

  1. If anomalies cluster in a window whose width equals your UTC offset, you already know the bug.
  2. "What day is it?" is a business question, not a Date method. Answer it once, in one function.
  3. Store epochs, render zones. Every derived date is a cache of a decision — make the decision once.
  4. Boundary tests belong at 20:59, 21:00, and 23:59 EAT — not at noon, where everything passes.