Market-intelligence startup, co-founder
Data Observability: Feed Health at Scale
Every external feed, sweep, and scheduled job leaves evidence. A sampler judges each one every five minutes against its own freshness rule, draws 90 days of history, and emails once when a feed goes red and once when it recovers.
- Headline number
- 69 feeds judged every 5 minutes
- Role
- Architected and led
- Category
- Data Platforms
- Updated
Architecture
- Data fetchersBatch jobs and schedulers
- Feed events tableJob runs table
- Feed samplerevery 5 min, one leaderBrowser-style uptime probes4 regions
- 90-day health boardIncident emailone per episode
Key takeaways
- Every fetch and every job run writes an event, throttled and fire-and-forget so a slow database never slows a quote
- A single-leader sampler judges 69 feeds and jobs against per-feed rules every five minutes
- One outage produces one email. Recovery produces one more.
- Browser-identical uptime probes exist because the CDN once hung real browsers while curl passed
The situation
A market-data product can look healthy while silently serving stale numbers. Eight vendors, a dozen scheduled jobs, and feeds that are only read during market hours make "is the data fresh?" a hard question to answer without instrumentation.
What I built
Every fetch writes a feed event and every job run writes a run record. Writers are throttled and fire-and-forget by design. A sampler runs every five minutes under a database advisory lock, so exactly one replica judges each tick, and evaluates each of about 69 feeds and jobs against its own rule: how stale is too stale, and when should it be quiet because the market is closed.
Results draw a 90-day board of daily cells plus 24 hours of ticks per feed, and open and close incident episodes with one email each. Every 30 minutes the sampler actively probes feeds nobody may be reading, so "no reader" is never mistaken for "broken." Above that sit browser-style availability tests from four regions with body-content matching, because on one July morning the CDN hung real browsers while a plain health check returned 200.
The outcome
Feed problems are caught before members notice them, jobs that die on a weekend still go red, and the founders get one actionable message per incident instead of a flood.
Stack
Related work
Same discipline, different system.
Need this built inside your organization?
One person, architecture to deployment. Tell me what is stuck and I will tell you where I would start.