Senior Data Engineer (Databricks / Microsoft Azure)

Project overview

Senior Data Engineer (Databricks / Microsoft Azure) – Business Intelligence Team

Job Description

You play a central role within the Business Intelligence Team of BZ.

Foreign Affairs has the ambition to grow into an information-driven, data-driven organization. To achieve this, a diversity of business intelligence & analytics (BIA) and data science projects are underway, which are largely intertwined. Development and management are carried out by two development and management teams under the direction of two project leaders.

The various agile-scrum teams work closely with product owners and clients from different departments. Different stakeholders, different interests, all with the goal of working more and more based on information for an ever-larger user group.

Your activities will focus on further developing the data warehouse based on the Databricks platform. Extensive knowledge of Python and SQL is important for this.

Key Responsibilities

Responsibilities

Background of the Assignment

As a Data Engineer, you know better than anyone that the rise of information-driven work is inevitable. Data is all around us. Applying data can make BZ more efficient and smarter. The goal of the Data Competence Centre is to collect (big) data (internal/external, structured/unstructured) and arrange it in such a way that the data can be used to gain insights to support BZ's processes. A business intelligence & analytics and data science team has been established for its development.

You will work in a multidisciplinary scrum team for the business intelligence products, consisting of data analysts, developers, and business representatives, among others. In collaboration with representatives from various departments, work is being done to structurally unlock BZ's most important systems and to house this data in an enterprise data warehouse and associated analytical environments. For the facilitation of this team, BZ is looking for a passionate Senior Data Engineer with relevant experience on the Microsoft Azure platform.

The data engineer plays a central role in strategically positioning and making data and analysis initiatives applicable (making available, automating, loading, and preparing) for (policy) departments and posts. The data engineer spends most of the time building, managing, and optimizing data pipelines and effectively moving these data pipelines to production.

In this, the Data Engineer works closely with the functional DevOps team of Infra Engineers, analysts, and Tableau developers. This sometimes requires a good dose of flexibility and creativity towards colleagues. The suitable candidate knows how to get things done, can handle a wide variety of tasks during the workdays, and is a true team player. Moreover, the Data Engineer is the specialist in unlocking data from sources, data transformations, and data modeling.

Job Requirements (knock-out)

Wishes (preferred)

Other Wishes

Competencies

• Develops, maintains, and optimizes data pipelines for specific information products. The Databricks platform is used for this.
• Supports, advises, and coordinates with data analysts, infra engineers, visualizers, data scientists, and BI consultants (and information managers) regarding information planning and information strategy.
• Works closely with BI consultants, data users, data scientists, etc. on the aforementioned techniques, enabling them to integrate and use the data they need.
• Ensures that workloads are managed (deployment via DevOps pipelines) to development and production. This process is continuously further developed by the data engineer.
• The Data Engineer is responsible for monitoring and signaling the latest developments in innovative and modern tools, techniques, and architecture for data development and management.
• The Data Engineer ensures the complete or partial automation of loading and processing procedures as much as possible.
• Together with the data steward, among others, ensures that compliance requirements and data governance are incorporated into the technical solution, so that users (BI consultants, data scientists, data users) use the data responsibly.
• Thinks independently and uses technological knowledge to integrate different concepts to create unique solutions (Innovate).
• Experience with networks, specifically in relation to Databricks and Microsoft Azure.
• Aware of the latest insights in setting up and securing Azure resources such as Databricks.
• Not suitable for ZZP (self-employed professionals).
• Experience with CI/CD methodology and Azure DevOps.
• Experience with Azure tools such as Azure DevOps, Data Factory, and similar (streaming & batch).
• Familiar with enterprise data warehouse and business intelligence-oriented architectures. Experience with PowerShell.
• Person who is interested in and always up-to-date with the latest technologies in Data Engineering, Data Analytics, and data warehousing.
• Experience managing cloud services.
• Experience managing visualization programs (such as Tableau).
• Knowledge of and experience with visualization tools like Tableau is a plus.
• Knowledge of financial processes and/or HR processes, preferably within the Central Government.
• Broad product knowledge of Business Intelligence and Data Warehouse tools.
• Experience with supplier management.
• Knowledge and experience with SAP architecture.
• Knowledge and experience with Data management and governance.
• Good command of the Dutch language.
• Ability to gain support for the boundaries of data engineering.
• Skill in developing, communicating, and defending policy, business operations products, services, policies, and processes, new ideas, and concepts.
• Skill in creating, implementing, and executing products, services, policies, and processes.
• System building
• System integration
• Implementing data solutions
• Testing
• Supporting changes
• User-driven design
• Privacy, security, and ethics
• Information and knowledge management

Project details

• Start: flexible
• Duration: 6 months
• Location: Den Haag, NL
• Languages: Dutch