Misha Zubarav
March 26, 2025 at 5:00:00 PM
A recent engineering effort highlights how building resilient, API-compatible data connectors and modular dbt transformations can unlock significant value in large-scale ETL pipelines, far beyond simple data movement.
ETL is often seen as a solved problem—but at scale, and in the real world, it’s rarely straightforward. Recently, our team worked on a series of projects that reminded us how much value lies in crafting resilient, well-designed data pipelines that do more than just move data. We built and deployed multiple custom connectors designed to make external APIs compatible with Fivetran ingestions, among others.
These connectors didn’t just pass data through—they actively transformed the source format, managed state, handled pagination, dealt with API-specific quirks like error retrieval and rate limiting, and seamlessly interfaced with Fivetran’s expectations. Each connector was deployed as a stateless Cloud Run function, making them easy to scale and maintain.
In parallel, we designed a dbt-based transformation layer for a BigQuery dataset from a different data source. This let us modularize the logic, bring transparency to the data lineage, and align closely with the needs of downstream consumers—proof that dbt’s declarative model continues to be a force multiplier for data teams.