Senior Data & AI Engineer – Azure & Databricks

Project overview

Senior Data & AI Engineer – Azure & Databricks

For one of our clients, we are looking for an experienced Senior Data & AI Engineer to develop and productively implement modern data and AI solutions based on Microsoft Azure and Databricks.

The goal of the project is to reliably connect internal data sources and systems, efficiently process data, and build productive LLM applications based on this. The focus is on Azure OpenAI, RAG, AI agents, as well as the automation and secure operation of data and AI processes.

Responsibilities

What you bring to the project:

Framework conditions:

• Connecting internal data sources, systems, and REST APIs
• Reading, cleaning, and processing data with Python, SQL, and Apache Spark/PySpark
• Developing LLM applications for the evaluation and assessment of processed data with Azure OpenAI
• Implementing RAG solutions including embeddings, vector search, tool calling, and AI agents
• Automating data and AI processes with Databricks Workflows
• Transferring data pipelines, models, and AI applications into production
• Using MLflow and Mosaic AI for the management and operation of AI solutions
• Building and further developing CI/CD and MLOps processes for versioning, testing, evaluation, deployment, and monitoring
• Ensuring traceable and secure operation through Unity Catalog, Azure Key Vault, Managed Identities, RBAC, and Private Endpoints
• Implementing logging and alerting mechanisms
• Independently taking responsibility for the entire development process – from data integration to stable operation
• Close collaboration with technical and business stakeholders in a small development team
• Profound practical experience in Data Engineering with Azure Databricks, Python, SQL, and Apache Spark/PySpark
• Good knowledge of Delta Lake and Lakehouse architectures
• Experience in developing GenAI and LLM applications with Azure OpenAI
• Knowledge of RAG, embeddings, vector search, and structured outputs, and ideally of tool calling and agentic workflows
• Experience with Microsoft Azure, particularly Azure Data Factory, Azure Functions, and Azure Storage/ADLS
• Knowledge of Databricks Workflows, MLflow, and Mosaic AI
• Experience in building CI/CD and MLOps processes
• Confident use of Git and Azure DevOps or GitHub Actions
• Knowledge of Infrastructure as Code, for example with Terraform or Bicep
• Experience with security and governance concepts for productive data and AI applications
• Independent and solution-oriented approach, as well as willingness to take technical responsibility from development to operation
• Very good German language skills and experience in collaborating with technical and business stakeholders

Project details

2027

06 months with option for extension

Utilization: 80%

100% remote

Project language: German

Are you open to new challenges and want to connect modern data platforms with productive AI applications? Then send me your CV as a Word document, including your possible start date and hourly rate, to [contact removed].

We operate as a personnel consultancy. None of the technical terms mentioned are intended to be discriminatory with regard to age or experience. We confirm that we welcome all applications from candidates of all ages and experiences.

• Start: ab sofort
• Duration: 6 Monate
• Location: DE
• Languages: German