Required skills
Job description
Project overview
ABOUT THE ROLE
In this role, you will help transform a data platform that currently relies on Spark and Python, orchestrated with Airflow DAGs and stored in Apache Iceberg, into a more modular target architecture centered on dbt and Lakekeeper.
- Support the migration of the Agora data platform to a cloud-first architecture
- Develop and maintain data transformation logic using dbt as the central business logic layer
- Refactor and consolidate logic currently distributed across Spark, Python, and Airflow into a more standardized dbt-based approach
- Work with Apache Iceberg as the core storage layer in both current and target architecture
- Contribute to the implementation of an open and modular platform built around Lakekeeper
- Support data ingestion workflows across Starburst, Microsoft Fabric, and Snowflake
- Help build and maintain scheduling and orchestration processes using dbt and/or Microsoft Fabric
- Enable and improve data quality, lineage, documentation, and CI/CD practices across the platform
- Collaborate on the transition of the semantic layer from Dremio to Starburst
- Support semantic modeling for analytics using Iceberg views stored in Lakekeeper
- Work closely with cross-functional stakeholders to reduce operational complexity and improve platform maintainability
- Contribute to the development of a flexible data foundation prepared for future analytics and AI-driven use cases
- Strong experience in data engineering or analytics engineering
- Hands-on experience with dbt
- Solid knowledge of SQL and SQL-based transformation design
- Experience with modern data platform and warehouse/lakehouse concepts
- Practical experience with Apache Iceberg
- Experience with at least some of the following technologies: Spark, Python, Airflow, Starburst, Microsoft Fabric, Snowflake
- Understanding of data modeling, semantic layers, and analytics-focused data design
- Experience with CI/CD, testing, and data quality practices in data platforms
- Familiarity with lineage and technical documentation standards
- Ability to work in an open, modular architecture with separated storage and compute principles
- Understanding of cloud-first platform migration challenges and best practices
- Strong collaboration and communication skills
Project details
- Start: ASAP
- Location: Remote, PL
Indexed by 9am's meta-search. You are a job provider? Claim or manage this listing
Found on nofluffjobs.com