Market-intelligence startup, co-founder
Market Data Ingestion and Warehouse
Built the data backbone for a fintech platform: eight external feeds cached and health-monitored, four identity-scoped batch jobs, and a PostgreSQL warehouse with roughly 180 tables that no human credential can open.
- Headline number
- 186 migrations, zero passwords
- Role
- Architected and built
- Category
- Data Platforms
- Updated
Architecture
- Market data vendorquotes, bars, newsSEC EDGARfilings, XBRL, 13FMacro and rates feedsBLS, BEA, FRED, Treasury
- Cached data layerSWR, single-flightScheduled refresh jobs4 jobs, 4 identities
- PostgreSQL warehouse~180 tables, no passwords
- Web app and scannerMCP server
Key takeaways
- Eight sources: a real-time market-data vendor, SEC EDGAR, and five US government macro and rates feeds
- Whole-market sweeps run as isolated batch jobs, so serving traffic never competes with ingestion
- Roughly 180 tables, 186 additive migrations applied automatically on every deploy
- Twelve workload identities, each a database role. No password exists anywhere in the system.
The situation
A market-intelligence product lives or dies on its data layer. Quotes, bars, filings, financials, and macro prints come from eight different sources with eight different rate limits and failure modes. The product needed all of it fresh, cheap, and trustworthy, on a startup budget, with two founders.
What I built
Ingestion is a layer of about 87 modules with bounded in-process caches, stale-while-revalidate, and single-flight requests, so a thousand users looking at the same market snapshot cost one vendor call. Government feeds are timeout-capped so one slow agency degrades one tile rather than the page.
The expensive work moved out of the request path. Four scheduled jobs on Azure Container Apps refresh earnings, financials, liquidity, and institutional holdings. Each has its own managed identity with grants on only the tables it writes. A financials sweep covers 6,077 tickers in 27 seconds. Jobs fail loudly: a liquidity run exits non-zero if fewer than half the names produce a spread.
The warehouse is PostgreSQL 16 with two schemas: content and track record on one side, members and billing on the other. 186 additive-only migrations run from a migrate job during deploy, and a failed migration stops the rollout. Authentication is Entra-only with password auth disabled on the server. Developers never touch production; a snapshot job exports a scrubbed copy inside the cloud boundary, and the database client refuses to connect a dev process to the production host.
The outcome
The platform serves live market data to paying members with no stored credentials and a 35-day restore window that has been proven end to end. When the vendor retired an endpoint in October, the sweep had already been migrated.
Why it matters
The same pattern I built for a Fortune-level enterprise, source reality into a governed spine, works at startup scale. The constraint changes from politics to budget. The discipline does not.
Stack
Related work
Same discipline, different system.
Need this built inside your organization?
One person, architecture to deployment. Tell me what is stuck and I will tell you where I would start.