Senior Azure Databricks Expert / Solution Architect for Building a Lakehouse Data Platform

Project overview

Starting Situation

The Data department is planning to use DataBricks (Azure Databricks) and is looking for an experienced DataBricks expert on a freelance basis to shorten the initial learning phase and build a suitable infrastructure from the start.

Tasks and Deliverables

Task: Analysis of the existing data processes of the Data department (data sources, current tools/architecture, workflows, pain points) and derivation of the business and technical requirements for the use of DataBricks. The data landscape will become significantly more diverse in the future – this is precisely what motivated the decision for DataBricks. The analysis therefore also considers foreseeable future data sources and processes.

Result: A requirements document that serves as a jointly agreed-upon basis for further implementation (including functional/non-functional requirements, prioritization, scope definition).

Task: Development of an implementation proposal for the DataBricks-related requirements (including workspace architecture, Unity Catalog / Governance, cluster or compute strategy, CI/CD approach, cost model) in close cooperation with the internal DevOps Engineers. All infrastructure topics will be accompanied and implemented by the DevOps Engineers; they require well-founded statements on the DataBricks architecture and the resulting infrastructure requirements.

Result: A design or architecture proposal that can be used as a template for the actual setup of the infrastructure by the internal DevOps Engineers.

Task: Joint, prototypical implementation of 1-2 technically challenging end-to-end use cases as a reference implementation based on Azure Databricks. The implementation will be carried out in close cooperation with the internal team and explicitly serves to build knowledge and enable the team to implement future use cases independently.

The reference implementations should map the complete data lifecycle, from data ingestion and processing to provision for downstream use cases, such as analytics/BI solutions, ML workloads, feature stores, or other serving endpoints.

The freelancer takes on the role of an experienced architect, coach, and technical sparring partner. Key architectural and implementation decisions will be developed jointly with the internal team. For relevant technical decisions, alternatives, advantages, and disadvantages, as well as the respective recommendation, must be presented comprehensibly.

The focus is on knowledge transfer and imparting practical best practices, not on isolated implementation by the freelancer.

• Current State Analysis & Requirements Catalog
• Implementation and Architecture Proposal
• Prototypical Implementation

Project details

• Start: 02.11.2026
• Duration: 2 Monate mit Verlängerung
• Location: DE
• Languages: German