Technical Expert Kafka

Project overview

Our client in the insurance sector is looking for a Technical Expert Kafka H/F

Mission Description:

Context of the need

As part of a project to implement a data exchange foundation based on the open-source technology Apache Kafka, using the Confluent distribution, we are looking for a Kafka expert to strengthen the project team.

The objective is to establish an enterprise Kafka foundation intended to support different use cases for data exchange, replication, and integration.

Subject of the service

To strengthen the project team, this specific contract aims to find Kafka technical expertise for the implementation of a new Confluent Kafka foundation.

The missions will cover the different phases of the project that have been identified:

Phase 1 - Implementation of the pilot foundation:

This first step consists of setting up a pilot foundation to feed a first use case of the Data Platform type. The data made available by this foundation will be produced by the various technical foundations of the Agir-Arrco IS, namely: Management IS (PostgreSQL), Hadoop (Cloudera), SID-RC (Datastage) + Teradata.

Responsibilities

· Installation and configuration of the Kafka / Confluent cluster.

· Definition and implementation of operating standards and norms.

· Setup of administration and operating tools;

· Implementation of security mechanisms and authorization management by integrating into the Agirc-Arrco IS ecosystem;

· Platform performance optimization.

· Construction and industrialization of deliverable creation and integration processes.

· Implementation of integration and deployment workflows.

· Installation and configuration of "standard" and "Confluent-specific" Kafka producer and consumer connectors necessary for the first use cases: PostgreSQL, Hadoop (HDFS), Datastage, Teradata, other connectors necessary for the pilot scope..

Particular attention will be paid to the industrialization of connectors, including:

· Declaration of new producers and consumers,

· Configuration management,

· Initialization and refresh of consumer data,

· Automated deployment of subscriptions,

· Consideration of data schema evolutions by Kafka Topics,

· Automation of recurring operations,

It will also be necessary to implement all the foundation's operating procedures:

· Shutdown and restart procedures;

· Management of technical services;

· Supervision and monitoring;

· Setup of dashboards;

· Operating documentation.

· Connectors to implement

Phase 2: Foundation Extension:

After validation and delivery of the pilot foundation, the project will enter an extension phase to build a target foundation with greater capacity and able to accommodate new use cases.

The main use cases envisioned are:

· Use case #1: Data Platform Extension + Extension of the initial pilot to all databases and uses of the Data Platform.

· Use case #2: Replacement of IBM CDC tool + Migration of flows currently managed by IBM Change Data Capture (CDC) to Kafka.

· Use case #3: Replacement of batch extractors + Progressive replacement of batch extraction mechanisms feeding decision-making platforms with real-time flows based on Kafka.

· Use case #4: PostgreSQL database replication + Implementation of replication and synchronization mechanisms for PostgreSQL databases via Kafka.

· Use case #5: WebMethods foundation migration + Study and implementation of the migration of exchanges currently managed by the WebMethods foundation to Kafka.

Information regarding teleworking:

Yes — 2 days/week

Offices in Bordeaux

Project details

• Start: flexible
• Remote: hybrid
• Location: Bordeaux, France, FR
• Rate: 72-72 EUR/h
• Languages: French