Data Engineer Kafka / Confluent – Deployment & Industrialization @ Bordeaux

Responsibilities

The expert will be responsible for the installation, configuration, and industrialization of Kafka producer and consumer connectors, standard or specific to the Confluent distribution.

Connectors and systems concerned:

Particular attention will be paid to the industrialization of connectors and their operation.

Expected work:

The expert will contribute to the definition and implementation of all procedures necessary for the operation of the Kafka foundation.

Scope:

After validation and delivery of the pilot foundation, the project will enter an extension phase aimed at building a more scalable target platform.

Objectives:

The expert will also be responsible for producing the deliverables necessary for the sustainability of the foundation and its adoption by internal teams.

Expected deliverables:

Technical Stack

Kafka & Data Streaming

Infrastructure & systems

Development & scripting

Databases & Data ecosystem

Operations & supervision

Mission modalities

Mission objective

Contribute to the implementation of a robust, industrialized, secure, supervised, and scalable Apache Kafka / Confluent foundation, capable of sustainably supporting the exchanges and data flows of a complex enterprise IT system.

• Install and configure the Apache Kafka / Confluent cluster.
• Define and implement operational standards and norms.
• Install and configure administration and operational tools.
• Implement security mechanisms and authorization management, ensuring their integration into the existing IT system ecosystem.
• Optimize platform performance.
• Build and industrialize the processes for creating and integrating deliverables.
• Define and implement integration and deployment workflows.
• Integration and industrialization of Kafka connectors
• PostgreSQL.
• Hadoop / HDFS.
• IBM DataStage.
• Teradata.
• Other connectors necessary for the pilot scope.
• Declare new producers and consumers.
• Manage and maintain connector configurations.
• Ensure data initialization and refresh on the consumer side.
• Automate subscription deployment.
• Account for data schema evolutions at the Kafka topic level.
• Automate recurring technical operations.
• Guarantee the maintainability and operability of deployed connectors.
• Implementation of operations and supervision
• Definition of stop and restart procedures.
• Management of technical services.
• Implementation of supervision and monitoring mechanisms.
• Construction of operational dashboards.
• Writing and maintaining operational documentation.
• Management of procedures related to deployed connectors.
• Contribution to service continuity within the team.
• Extension and capacity increase of the Kafka foundation
• Support the evolution of the pilot foundation towards an enterprise Kafka platform.
• Enable the integration of new use cases.
• Extend the initial scope of the Data Platform to all its databases and uses.
• Contribute to increasing the infrastructure's capacity to support future data exchange, replication, and integration needs.
• Documentation, reporting, and skills transfer
• Weekly activity reporting.
• Monthly activity monitoring dashboard.
• Technical documentation: standards, principles, architecture documents, administration documents, operational documents, and operating procedures.
• Organization of knowledge transfer sessions.
• Writing of skills transfer materials.
• Completion and tracking of incident, request, and change tickets.
• Apache Kafka.
• Confluent Platform.
• Kafka Strimzi.
• Kafka Brokers.
• KRaft.
• Kafka / Confluent Components, including Cwork (project specification name).
• Schema Registry.
• Kafka Connect.
• JDBC Connectors.
• Debezium PostgreSQL.
• HDFS Connectors.
• Proprietary connectors, notably IBM DataStage.
• IBM InfoSphere CDC.
• Linux Red Hat.
• Kubernetes.
• Ansible.
• VMware.
• Shell.
• Python.
• JSON.
• Hadoop / Cloudera.
• HDFS.
• SAS.
• Power BI.
• Data Streaming Technologies.
• Prometheus.
• Grafana.
• IWS Scheduler.
• HDPS Backup.
• Kafka administration, supervision, and operational tools.
• Main location: Bordeaux (Gironde).
• Alternative locations: Paris or Lyon, subject to strong autonomy and regular travel to Bordeaux.
• Initial duration: 1 firm year.
• Renewal: Up to two possible renewals.
• Initial estimated start: November / December
• Organization: Intervention within a project team dedicated to the implementation and industrialization of an enterprise Kafka foundation.

Project details

Bordeaux (33) – Paris or Lyon are possible subject to strong autonomy and regular travel to Bordeaux

12 firm months, with the possibility of two renewals

Profile: Kafka / Confluent Technical Expert – 3 to 7 years of experience in the relevant service family, with client references

Environment: Enterprise IT System, Data Streaming, data integration and replication

Context and challenges

As part of a strategic IT system modernization project, a large organization is looking for an Apache Kafka / Confluent Expert to support the design, deployment, and industrialization of a new enterprise data exchange foundation.

This platform, based on the open-source Apache Kafka technology and the Confluent distribution, is intended to support various use cases for data exchange, replication, and integration between IT system components.

The project is structured around two main phases:

The expert will work within the project team, with a strong involvement in technical architecture, integration, security, automation, operations, and performance topics.

Detailed missions

The first objective is to set up a Kafka / Confluent infrastructure to feed an initial Data Platform use case.

Data will be produced by various IT system technical foundations:

• Phase 1: Construction and production deployment of a pilot Kafka foundation to feed an initial Data Platform.
• Phase 2: Extension of the pilot foundation towards a more scalable, industrialized target architecture capable of hosting new use cases.
• Construction and deployment of the pilot Kafka foundation
• Management IT system based on PostgreSQL.
• Hadoop / Cloudera Big Data environment.
• Business intelligence system using IBM DataStage.
• Teradata data platform.
• Start: flexible
• Remote: hybrid
• Location: Bordeaux, France, FR
• Languages: French