AI Data Engineer (m/f/d) focus Web Crawling | Information Extraction

Project overview

Leistungsbeschreibung

Design, development and implementation of a solution for the automated retrieval, extraction, structuring and provision of technical asset data from publicly accessible internet sources

Requirements

Design, development and implementation of a solution for the automated retrieval, extraction, structuring and provision of technical asset data from publicly accessible internet sources, in particular from:

Über den Auftraggeber

• Manufacturers’ websites and digital product catalogues
• Online directories, portals and databases
• Publications by professional and engineering associations
• Technical data sheets and other product-related web documents. The aim is to build structured product and model catalogues containing manufacturers, model designations and technical specifications. The results are to be provided as CSV files so that they can be technically reviewed and further processed to populate the existing EAM structure
• Analysis and identification of suitable web sources The contractor shall identify and evaluate suitable sources with regard to:
• Relevance to defined asset classes and types of equipment
• Information content and coverage of technical characteristics
• Data quality, timeliness and traceability
• Degree of structuring of the content and available interfaces
• Coverage of manufacturers, products and types
• Access conditions and technical accessibility
• HTML, table, PDF and download structures
• Legal and technical framework for data collection
• Design of the crawling and extraction solution; development of a technical target architecture for:
• Web crawling and web scraping
• API and download interfaces
• Extraction from HTML pages, tables and technical documents
• AI-supported information extraction and normalisation
• Rules for data validation and duplicate detection
• Traceability of sources and quality checks
• Development of the crawling, extraction and AI components:
• Entwicklung und Parametrisierung modularer Komponenten zur Quellenermittlung, Datenerhebung, Extraktion, Zuordnung und Normalisierung von Herstellern, Produktreihen, Typen und technischen Merkmalen
• Unterschiedliche Webseitenstrukturen und Dokumentformate sind über erweiterbare Konnektoren und Parser zu berücksichtigen
• Data structuring and provision: Conversion of the extracted information into:
• Technically defined asset, manufacturer, product and type objects
• Attribute structures that can be mapped onto the existing EAM structure
• Standardised CSV files
• Traceable data records specifying the respective source
• Quality assurance and technical validation: Implementation of procedures and a Databricks app to:
• Completeness and plausibility checks
• Technical validation of manufacturers, models and technical specifications by electrical engineers
• Assessment of data extraction reliability and flagging of unreliable values
• Correction, approval and error classification by the business user
• Pilot scheme: Conducting a pilot for selected asset classes, manufacturers and web sources. The specific selection will be determined jointly within the project. Result:
• Demonstration of the solution’s technical and professional suitability
• Measurement of extraction and validation quality
• Identification of opportunities for optimisation and scaling requirements
• Documentation and handover: Preparation of:
• Technical documentation for the crawlers, connectors and extraction logic
• Operational and configuration documentation
• Architecture overview, including the Databricks app
• Data model and CSV interface specifications
• Handover documentation, including source code and deployment configuration
• Web-Crawling & Web-Scraping
• Experience with Scrapy, Beautiful Soup or similar frameworks
• HTML, DOM and table analysis
• Working with dynamic web pages and download structures
• Development of robust, extensible crawlers and parsers Information Extraction & Generative AI
• Large Language Models and Natural Language Processing
• Prompt engineering and structured output formats
• Extraction and normalisation of technical features
• Classification, mapping and duplicate detection
• Interface and document processing
• REST APIs and file-based data sources
• PDF and datasheet extraction
• Table extraction and schema mapping
• Source and metadata management
• Error handling and monitoring of extraction runs Data Engineering & Databricks
• Python
• Pandas and PySpark
• Databricks, Delta Lake and development of Databricks Apps Desirable additional skills
• MLOps and CI/CD for data and AI components
• Design of crawling solutions that comply with data protection and legal requirements
• Selenium, Playwright or comparable browser automation
• Azure services and Databricks deployment
• Experience with product, asset or type catalogues
• Ontologies, semantic models and technical attribute catalogues
• EAM and SAP attribute models
• Understanding of the electrical engineering domain

Project details

2 November 2026

End: 31 December 2026; + option to extend

approx. 100%

On-site hours: 0

Off-site hours: 280

Contract type: Contracting | Only nearshore resources can be considered.

• Start: flexible
• Remote: remote
• Location: DE, Deutschland, DE