Senior Data Engineer

Project overview

Experience: 4 to 6 Years (Data Engineering) with Large-Scale Cloud Data Platform Migrations

Role Overview

We are seeking experienced Data Engineers to lead and support the migration of large-scale analytical and reporting workloads from Amazon Redshift to Databricks Lakehouse Platform. The ideal candidate will have hands-on experience in data warehouse modernization, ETL/ELT migration, performance optimization, cloud-native architectures, and big data processing.

Key Responsibilities

Required Skills

Data Engineering

Migration Experience

o Redshift to Databricks

o Teradata/Snowflake/Hadoop to Databricks (preferred)

Compliant working and operation of supported GxP systems.

• Lead the migration of data pipelines, data models, and reporting workloads from Redshift to Databricks.
• Perform assessment and discovery of existing Redshift environments, schemas, workloads, and dependencies.
• Design and implement scalable data solutions using Databricks, Delta Lake, and Spark.
• Convert SQL-based ETL/ELT processes into Databricks notebooks, workflows, and pipelines.
• Optimize data ingestion, transformation, and storage for performance and cost efficiency.
• Develop batch and streaming data pipelines using PySpark and Spark SQL.
• Implement data quality, reconciliation, and validation frameworks during migration.
• Collaborate with business stakeholders, architects, and analytics teams to ensure successful migration and adoption.
• Troubleshoot performance bottlenecks and tune Spark jobs, Delta tables, and SQL workloads.
• Establish CI/CD processes and infrastructure-as-code practices for Databricks environments.
• Document migration strategies, architecture, and operational procedures.
• Strong experience with large-scale data warehousing and data lake architectures.
• Hands-on experience with Amazon Redshift and Databricks.
• Expertise in PySpark, Spark SQL, and Python.
• Strong SQL development and query optimization skills.
• Experience with data modeling (Star Schema, Snowflake Schema, Dimensional Modeling).
• Prior experience migrating from:
• Data validation, reconciliation, and cutover planning.
• Performance tuning and workload modernization.

Project details

• Start: flexible
• Remote: remote
• Location: Remote Location, US
• Rate: 39-46 EUR/h
• Languages: English