Expert Senior Kafka Consultant

Project overview

Context

The objective is to implement an enterprise Kafka foundation to support various data exchange, replication, and integration use cases.

The missions will cover the different project phases that have been identified:

This first step consists of setting up a pilot foundation to feed a first Data Platform type use case. The data made available to the foundation will be produced by the various IT systems, namely: Management IT (PostgreSQL), Hadoop (Cloudera), SID-RC (Datastage) + Teradata.

After validation and delivery of the pilot foundation, the project will enter an extension phase to build a more capable target foundation capable of accommodating new use cases.

Missions

During Phase 1 of foundation implementation, the identified tasks to be carried out are as follows:

Particular attention will be paid to the industrialization of connectors, including:

It will also be necessary to implement all the foundation's operational procedures:

Connectors to be implemented

During Phase 2 of foundation extension, the main identified use cases are as follows:

The client is looking for a Kafka / Confluent expert capable of participating in all phases of the project and supporting the team on the following topics:

The expert will join the project team to contribute to the design, implementation, and industrialization of the company's future data exchange foundation.

Technical Environment

Examples of expected deliverables:

Technical skills:

Confluent and Strimzi Kafka Distribution.

Scripting and development languages: Shell, Python, Json.

Operating system: Linux Redhat.

Databases: PostgreSQL.

Hadoop: Cloudera.

Data Streaming: IBM Infosphere CDC.

Infrastructure platform: Kubernetes, Ansible, Linux RedHat, Vmware.

Operations: IWS scheduler, HDPS backup, Prometheus / Grafana monitoring.

The consultant must demonstrate the following skills:

• Phase 1
• Implementation of the pilot foundation:
• Phase 2: Foundation Extension:
• Installation and configuration of the Kafka / Confluent cluster.
• Definition and implementation of operational standards and norms.
• Setup of administration and operational tools.
• Implementation of security mechanisms and authorization management by integrating into the IT ecosystem.
• Platform performance optimization.
• Construction and industrialization of deliverable creation and integration processes.
• Implementation of integration and deployment workflows.
• Installation and configuration of "standard" and "Confluent specific" Kafka producer and consumer connectors required for the first use cases: PostgreSQL, Hadoop (HDFS), Datastage, Teradata, other connectors required for the pilot scope.
• Declaration of new producers and consumers.
• Configuration management.
• Initialization and refresh of consumer data.
• Automated deployment of subscriptions.
• Consideration of data schema evolutions by Kafka Topics.
• Automation of recurring operations.
• Shutdown and restart procedures.
• Management of technical services.
• Supervision and monitoring.
• Setup of dashboards.
• Operational documentation.
• Use case n°1: Data Platform Extension + extension of the initial pilot to all databases and uses of the Data Platform.
• Use case n°2: Replacement of IBM CDC tool + migration of flows currently managed by IBM Change Data Capture (CDC) to Kafka.
• Use case n°3: Replacement of batch extractors + progressive replacement of batch extraction mechanisms feeding decision-making platforms with real-time flows based on Kafka.
• Use case n°4: PostgreSQL database replication + implementation of PostgreSQL database replication and synchronization mechanisms via Kafka.
• Use case n°5: WebMethods foundation migration + study and implementation of the migration of exchanges currently managed by the WebMethods foundation to Kafka.
• Kafka and Confluent Architecture.
• Installation and administration of Kafka platforms.
• Security of exchanges.
• Performance and tuning.
• Implementation of Kafka Connect connectors.
• Industrialization and automation of deployments.
• Supervision and operations.
• Support for project teams and skills transfer.
• Writing architecture documents, standards and best practices, operational documents, etc.
• Weekly reporting.
• Monthly activity tracking dashboard.
• Technical documentation: standards, principles, architecture document, administration document, operational document, operating procedures.
• Knowledge transfer: organization of knowledge transfer sessions and preparation of materials.
• Request management: fully completed incident/request/change tickets.
• Contribution to service continuity within the team.
• Kafka Foundation Environment:
• Other technical environments:
• "Kafka Confluent / IBM" Distribution: Kafka cluster components (Broker, Kraft, Cwork), Schema Registry, Connectors (JDBC, Debezium PgSQL, HDFS, ...).
• Other Distributions: "Kafka Strimzi", proprietary connectors (e.g., "Datastage") or complementary tools.
• Scripting and development languages: Shell, Python, Json.
• Infrastructure and operating system: Linux, Kubernetes, Ansible.
• Related tools: Datastage ETL, Power-BI, SAS, Teradata, PostgreSQL, Data streaming, Hadoop.

Project details

• Start: flexible
• Remote: hybrid
• Location: Bordeaux, France, FR
• Rate: 69-69 EUR/h
• Languages: French