Expert KAFKA/Confluent

Project overview

Context

As part of a project to implement a data exchange foundation based on the open-source technology Apache Kafka, using the Confluent distribution, we are looking for a Kafka expert to strengthen the project team.

MISSIONS

To strengthen the project team, we are seeking Kafka technical expertise for the implementation of a new Confluent Kafka foundation. The objective is to establish an enterprise Kafka foundation to support various data exchange, replication, and integration use cases.

The missions will cover the different phases of the project that have been identified:

Phase 1 - Implementation of the pilot foundation:

This first step involves setting up a pilot foundation to support a first use case of a Data Platform. The data made available by this foundation will be produced by the various IT system foundations, namely: Management IT (PostgreSQL), Hadoop (Cloudera), SID-RC (Datastage) + Teradata.

Responsibilities

It will also be necessary to implement all the foundation's operational procedures:

Phase 2: Foundation Extension:

After validation and delivery of the pilot foundation, the project will enter an extension phase to build a target foundation with greater capacity, capable of accommodating new use cases.

The main envisaged use cases are:

Profile sought:

We are looking for a Kafka / Confluent expert capable of participating in all phases of the project and supporting the team on the following topics:

The expert will join the project team to contribute to the design, implementation, and industrialization of the company's future data exchange foundation.

Technical - Technological Environment

Kafka Foundation Environment:

Other technical environments:

Technical Skills

"Kafka Confluent / IBM" Distribution: Kafka cluster components (Broker, Kraft, Cwork), Schema Registry, Connectors (JDBC, Debezium PgSQL, HDFS, ...)

Other Distributions: "Kafka Strimzi", proprietary connectors (e.g., "Datastage") or complementary tools

Scripting and development languages: Shell, Python, Json

Infrastructure and Operating System: Linux, Kubernetes, Ansible

Related tools: ETL Datastage, Power-BI, SAS, Teradata, PostgreSQL, Data streaming, Hadoop

Know-how and soft skills

Writing skills: technical documentation, specifications, communication and reporting

Communication and interpersonal skills (listening and dialogue skills, meeting facilitation, negotiation)

Autonomy

Ability to work in a team

Analytical and synthesis skills

Experienced Profile (profile with 3 to 7 years of experience in the same service category.

Expected deliverables

Work environment

As part of a project to implement a data exchange foundation based on the open-source technology Apache Kafka, using the Confluent distribution, we are looking for a Kafka expert to strengthen the project team.

• Installation and configuration of the Kafka / Confluent cluster.
• Definition and implementation of operational standards and norms.
• Setup of administration and operational tools;
• Implementation of security mechanisms and authorization management, integrating into the IT system ecosystem;
• Optimization of platform performance.
• Construction and industrialization of deliverable production and integration processes.
• Implementation of integration and deployment workflows.
• Installation and configuration of "standard" and "Confluent-specific" Kafka producer and consumer connectors required for the first use cases: PostgreSQL, Hadoop (HDFS), Datastage, Teradata, other connectors necessary for the pilot scope. Particular attention will be paid to the industrialization of connectors, including:
• Declaration of new producers and consumers,
• Configuration management,
• Initialization and refresh of consumer data,
• Automated deployment of subscriptions,
• Consideration of data schema evolutions for Kafka Topics,
• Automation of recurring operations,
• Stop and restart procedures;
• Management of technical services;
• Supervision and monitoring;
• Setup of dashboards;
• Operational documentation.
• Connectors to be implemented
• Use case #1: Data Platform Extension + Extension of the initial pilot to all databases and uses of the Data Platform.
• Use case #2: Replacement of IBM CDC tool + Migration of flows currently managed by IBM Change Data Capture (CDC) to Kafka.
• Use case #3: Replacement of batch extractors + Progressive replacement of batch extraction mechanisms feeding decision-making platforms with real-time flows based on Kafka.
• Use case #4: PostgreSQL database replication + Implementation of PostgreSQL database replication and synchronization mechanisms via Kafka.
• Use case #5: WebMethods foundation migration + Study and implementation of the migration of exchanges currently managed by the WebMethods foundation to Kafka.
• Kafka and Confluent Architecture;
• Installation and administration of Kafka platforms;
• Securing exchanges;
• Performance and tuning;
• Implementation of Kafka Connect connectors;
• Industrialization and automation of deployments;
• Supervision and operations;
• Support for project teams and skills transfer.
• Writing architecture documents, standards and best practices, operational documents, etc.;
• Confluent and Strimzy Kafka Distribution
• Scripting and development languages: Shell, Python, Json,
• Operating system: Linux Redhat
• Database: PostgreSQL
• Hadoop: Cloudera
• Data Streaming: IBM Infosphere CdC
• Infrastructure foundation: Kubernetes, Ansible, Linux RedHat, Vmware
• Operations: scheduler IWS, HDPS backup, Prometheus / Grafana supervision,
• Weekly reporting
• Monthly activity monitoring dashboard
• Technical documentation: Standards, principles, architecture document, administration document, operational document, operating procedures
• Knowledge transfer: organization of knowledge transfer sessions and creation of support materials.
• Request management: Incident / request / change tickets duly completed
• Contribute to service continuity within the team

Project details

• Start: ASAP
• Duration: 18 mois
• Remote: hybrid
• Location: Gradignan, Nouvelle-Aquitaine, FR
• Languages: French