Lead DBT Engineer

Project overview

ABOUT THE ROLE

In this role, you will help transform a data platform that currently relies on Spark and Python, orchestrated with Airflow DAGs and stored in Apache Iceberg, into a more modular target architecture centered on dbt and Lakekeeper.

• Support the migration of the Agora data platform to a cloud-first architecture
• Develop and maintain data transformation logic using dbt as the central business logic layer
• Refactor and consolidate logic currently distributed across Spark, Python, and Airflow into a more standardized dbt-based approach
• Work with Apache Iceberg as the core storage layer in both current and target architecture
• Contribute to the implementation of an open and modular platform built around Lakekeeper
• Support data ingestion workflows across Starburst, Microsoft Fabric, and Snowflake
• Help build and maintain scheduling and orchestration processes using dbt and/or Microsoft Fabric
• Enable and improve data quality, lineage, documentation, and CI/CD practices across the platform
• Collaborate on the transition of the semantic layer from Dremio to Starburst
• Support semantic modeling for analytics using Iceberg views stored in Lakekeeper
• Work closely with cross-functional stakeholders to reduce operational complexity and improve platform maintainability
• Contribute to the development of a flexible data foundation prepared for future analytics and AI-driven use cases
• Strong experience in data engineering or analytics engineering
• Hands-on experience with dbt
• Solid knowledge of SQL and SQL-based transformation design
• Experience with modern data platform and warehouse/lakehouse concepts
• Practical experience with Apache Iceberg
• Experience with at least some of the following technologies: Spark, Python, Airflow, Starburst, Microsoft Fabric, Snowflake
• Understanding of data modeling, semantic layers, and analytics-focused data design
• Experience with CI/CD, testing, and data quality practices in data platforms
• Familiarity with lineage and technical documentation standards
• Ability to work in an open, modular architecture with separated storage and compute principles
• Understanding of cloud-first platform migration challenges and best practices
• Strong collaboration and communication skills

Project details

• Start: ASAP
• Location: Remote, PL