Expert Kafka / Confluent - Data Exchange Foundation

Project overview

Our client, a social protection organization (supplementary retirement), is building its enterprise Kafka foundation on the Confluent distribution. This foundation will handle data exchanges, replication, and integration for the entire IT system. It must also progressively replace IBM CDC, batch extractions, and the WebMethods foundation. You will join the project team to design, build, and industrialize this foundation.

The mission is based in Gradignan, near Bordeaux. A profile based in Paris or Lyon is possible if they are highly autonomous and can travel regularly to the site.

Missions

Profile

• Design the Kafka / Confluent architecture and evolve it from a pilot foundation to a target platform that is scalable and multi-use case
• Install, configure, and administer Kafka clusters: brokers in KRaft mode, Schema Registry, Kafka Connect workers, on RedHat and Kubernetes (Confluent / Strimzi)
• Implement and industrialize Kafka Connect connectors: JDBC, Debezium PostgreSQL, HDFS, Datastage, Teradata. This covers declaring producers and consumers, managing configurations and subscriptions, initializing data, and evolving schemas
• Replace existing CDC and batch flows with real-time flows, and implement PostgreSQL replication via Kafka
• Secure exchanges and integrate authorization management with the existing IT ecosystem
• Tune and optimize platform performance
• Automate deployment and recurring operations with Ansible, Shell, Python, and CI/CD pipelines
• Implement platform operations: Prometheus / Grafana monitoring, dashboards, shutdown and restart procedures, incident, request, and change management
• Write architecture, administration, and operations documentation, as well as standards and best practices
• Train teams and ensure knowledge transfer

Requirements

Appreciated

Soft skills

• Minimum 5 years of experience with Kafka, including experience administering production platforms: complex incidents, tuning, performance
• Expertise in Confluent Platform: brokers and KRaft, Schema Registry, Kafka Connect and its connectors (JDBC, Debezium, HDFS)
• Experience in Kafka architecture design, not just operations
• Expert level Linux RedHat, Kubernetes, and Ansible
• Shell and Python scripting, proficiency in JSON
• Strimzi
• Proprietary connectors (Datastage)
• Data ecosystem: PostgreSQL, Hadoop Cloudera, Teradata, CDC
• Referent posture capable of training
• Writing rigor
• Complete autonomy

Project details

• Start: ASAP
• Duration: 12 mois renouvelable
• Remote: hybrid
• Location: Gradignan, Nouvelle-Aquitaine, FR
• Rate: 88-94 EUR/h
• Languages: French