Expert Kafka / Kafka Confluent / IBM / Kafka Strimzi

Project overview

Context

We are looking for an Expert in Kafka / Kafka Confluent / IBM / Kafka Strimzi for one of our clients.

The objective is to implement an enterprise Kafka foundation intended to support different use cases for data exchange, replication, and integration.

Missions

The missions will cover the different phases of the project that have been identified:

Phase 1 - Implementation of the pilot foundation:

This first step consists of implementing a pilot foundation to feed a first use case of the Data Platform type. The data made available to this foundation will be produced by the different technical foundations of the client's IS, namely: Management IS (PostgreSQL), Hadoop (Cloudera), SID (Datastage) + Teradata.

Responsibilities

It will also be necessary to implement all the operating procedures for the foundation:

Phase 2: Foundation Extension:

After validation and delivery of the pilot foundation, the project will enter an extension phase to build a target foundation with greater capacity and capable of hosting new use cases.

The main use cases envisaged are:

Tools & Environment

Working Conditions

• Installation and configuration of the Kafka / Confluent cluster.
• Definition and implementation of operating standards and norms.
• Implementation of administration and operation tools.
• Implementation of security mechanisms and authorization management by integrating into the client's IS ecosystem.
• Optimization of platform performance.
• Construction and industrialization of deliverable creation and integration processes.
• Implementation of integration and deployment workflows.
• Installation and configuration of "standard" and "Confluent-specific" Kafka producer and consumer connectors required for the first use cases: PostgreSQL, Hadoop (HDFS), Datastage, Teradata, other connectors required for the pilot scope.
• Industrialization of connectors, including:
• Declaration of new producers and consumers,
• Configuration management,
• Initialization and refresh of consumer data,
• Automated deployment of subscriptions,
• Consideration of data schema evolutions by Kafka Topics,
• Automation of recurring operations.
• Shutdown and restart procedures;
• Management of technical services;
• Supervision and monitoring;
• Implementation of dashboards;
• Operating documentation.
• Use case #1: Data Platform Extension + Extension of the initial pilot to all databases and uses of the Data Platform.
• Use case #2: Replacement of IBM CDC tool + Migration of flows currently managed by IBM Change Data Capture (CDC) to Kafka.
• Use case #3: Replacement of batch extractors + Progressive replacement of batch extraction mechanisms feeding decision-making platforms with real-time flows based on Kafka.
• Use case #4: PostgreSQL database replication + Implementation of replication and synchronization mechanisms for PostgreSQL databases via Kafka.
• Use case #5: WebMethods foundation migration + Study and implementation of the migration of exchanges currently managed by the WebMethods foundation to Kafka.
• "Kafka Confluent / IBM" distribution: component of the Kafka cluster (Broker, Kraft, Cwork)
• Schema registry, connectors (JDBC, Debezium PgSQL, HDFS, ...)
• Other distributions: "Kafka Strimzi", proprietary connectors (example: "Datastage") or complementary tooling
• Scripting and development languages: Shell, Python, Json
• Infrastructure and operating system: Linux, Kubernetes, Ansible
• Related tools: ETL Datastage, Power-BI, SAS, Teradata, PostgreSQL, Data streaming, Hadoop
• Mission location: Paris
• Long-term mission
• 2 days of remote work per week
• Start date: ASAP
• Professional English mandatory
• Required seniority: Minimum 8 years of experience

Project details

• Start: flexible
• Remote: hybrid
• Location: Paris, France, FR
• Languages: French