Geforderte Skills
Jobbeschreibung
- Develop and maintain data pipelines using Apache Airflow and Python. - Build data ingestion processes from APIs, databases, and file-based sources into AWS S3 and Snowflake. - Design and implement data transformations using dbt. - Optimize Snowflake queries, warehouse performance, and resource consumption. - Implement pipeline monitoring, error handling, a
Project overview
- Develop and maintain data pipelines using Apache Airflow and Python.
- Build data ingestion processes from APIs, databases, and file-based sources into AWS S3 and Snowflake.
- Design and implement data transformations using dbt.
- Optimize Snowflake queries, warehouse performance, and resource consumption.
- Implement pipeline monitoring, error handling, and alerting mechanisms.
- Ensure data quality, consistency, and reliability through automated testing and validation.
- Maintain technical documentation and follow data engineering best practices.
- 3+ years of professional experience in Data Engineering.
- Strong expertise in Apache Airflow, including DAG development, operators, sensors, hooks, TaskFlow API, and workflow optimization.
- Advanced Python skills, including writing clean, maintainable, and testable code.
- Advanced SQL knowledge, including complex queries, CTEs, window functions, and performance optimization.
- Hands-on experience with Snowflake, including warehouses, stages, external volumes, and data loading processes.
- Practical knowledge of dbt, including models, tests, macros, snapshots, and incremental transformations.
- Experience with AWS S3 and Data Lake architecture, including data partitioning, file formats, and access management.
- Familiarity with Git and CI/CD practices.
- English proficiency at B2 level or higher.
Nice to have
- Experience with Docker and Kubernetes.
- Knowledge of Apache Iceberg or Delta Lake.
- Familiarity with data quality and observability tools such as Great Expectations or Soda.
- Understanding of dimensional data modeling (Kimball methodology).
Project details
- Start: ASAP
- Location: Remote, Wrocław, Kraków, PL
- Rate: 110-120 EUR/h
Indexiert von der 9am-Meta-Suche. Du bist Job-Anbieter? Listing beanspruchen oder verwalten
Gefunden auf nofluffjobs.com