About the Role
Join a data and infrastructure team building a platform that transforms information from multiple sources into reliable, up-to-date company intelligence. You will own the platform end to end, from architecture and ingestion to production operations.
What You'll Do
Design a multi-source platform for concurrent ingestion, event-driven processing, and orchestrated batch workflows.
Build reliable intake from provider APIs, webhooks, files, and web scraping, with rate limits and robust failure handling.
Orchestrate scheduled workflows and historical backfills using tools such as Airflow, Dagster, or Prefect.
Develop event-driven systems with messaging and task queues, including retries, dead-letter queues, and controlled replay.
Store original payloads in cloud storage and transform them with dbt into raw, standardized, and curated layers.
Implement idempotent upserts, data contracts, automated validation, pipeline tests, and monitoring for freshness and failures.
Design and optimize PostgreSQL schemas and loading patterns, and integrate provider enrichment and structured extraction using LLMs.
At least 5 years building and operating production data platforms, including multi-source ingestion and ownership from architecture through ongoing operation.
Production experience with workflow orchestration, including dependencies, retries, and recovery. Airflow is preferred; Dagster, Prefect, Temporal, or comparable tools are relevant.
Hand