Python / AI Data Engineer – Web Scraping, LLM & Databricks | Remote - NEARSHORE

Project overview

Contact person for this request

Kerstin Ufer

Tel.: [contact removed]

E-Mail: [contact removed]

For the following activity, we are looking for 1 applicant with the following qualifications - perhaps you would be available for this or can make a recommendation:

By using the masculine form of the word "consultant / developer etc.", all genders female/male/diverse are meant.

With this request, we are looking for a Python / AI Data Engineer – Web Scraping, LLM & Databricks

DEADLINE FOR OFFERS: 28.10.2026 - 10:00 AM

ATTENTION: We can only offer NEARSHORE (EU citizens) for this!

Activity:

Project Goal

Conception, development, and implementation of a solution for automated research, extraction, structuring, and provision of technical asset data from publicly accessible internet sources, particularly from:

The goal is to build structured product and type catalogs with manufacturers, type designations, and technical characteristics. The results are to be provided as CSV files so that they can be technically reviewed and further processed for populating the existing EAM structure.

Scope of Services

The contractor identifies and evaluates suitable sources regarding:

Development of a technical target architecture for:

Development and parameterization of modular components for source identification, data collection, extraction, assignment, and normalization of manufacturers, product lines, types, and technical characteristics. Different website structures and document formats are to be considered via expandable connectors and parsers.

Transformation of the extracted information into:

Implementation of procedures and a Databricks App for:

Execution of a pilot for selected asset classes, manufacturers, and web sources. The specific selection will be determined jointly in the project.

Result:

Creation of:

Mandatory Competencies

Web Crawling & Web Scraping

Information Extraction & Generative AI

Interface and Document Processing

Data Engineering & Databricks

Desirable Additional Competencies

Utilization:

280 hours remote

Fee:

max. € 60.00 / hour for remote

Payment Terms:

30 days

Other

• Manufacturer websites and digital product catalogs
• Online directories, portals, and databases
• Publications by technical and engineering associations
• Technical data sheets and other product-related web documents
• Analysis and development of suitable web sources
• Relevance for defined asset classes and equipment types
• Information content and coverage of technical characteristics
• Data quality, timeliness, and traceability
• Degree of structuring of content and available interfaces
• Manufacturer, product, and type coverage
• Access conditions and technical availability
• HTML, table, PDF, and download structures
• Legal and technical framework conditions for data collection
• Conception of the crawling and extraction solution
• Web crawling and web scraping
• API and download interfaces
• Extraction from HTML pages, tables, and technical documents
• AI-supported information extraction and normalization
• Rules for data validation and duplicate detection
• Traceability of sources and quality checks
• Development of crawling, extraction, and AI components
• Data structuring and provision
• Technically defined asset, manufacturer, product, and type objects
• Characteristic structures that can be mapped to the existing EAM structure
• Standardized CSV files
• Traceable data records with source indication
• Quality assurance and technical validation
• Completeness and plausibility checks
• Technical validation of manufacturers, types, and technical characteristics by electrical engineers
• Assessment of extraction reliability and marking of uncertain values
• Correction, release, and error classification by the specialist user
• Piloting
• Proof of the technical and functional suitability of the solution
• Measurement of extraction and validation quality
• Identification of optimization potentials and scaling needs
• Documentation and handover
• Technical documentation of the crawlers, connectors, and extraction logic
• Operating and configuration documentation
• Architecture overview including Databricks App
• Data model and CSV interface description
• Handover documentation including source code and deployment configuration
• Experience with Scrapy, Beautiful Soup, or comparable frameworks
• HTML, DOM, and table analysis
• Handling dynamic websites and download structures
• Development of robust, extensible crawlers and parsers
• Large Language Models and Natural Language Processing
• Prompt Engineering and structured output formats
• Extraction and normalization of technical characteristics
• Classification, mapping, and duplicate detection
• REST APIs and file-based data sources
• PDF and data sheet extraction
• Table extraction and schema mapping
• Source and metadata management
• Error handling and monitoring of extraction runs
• Python
• Pandas and PySpark
• Databricks, Delta Lake, and development of Databricks Apps
• MLOps and CI/CD for data and AI components
• Data protection and legally compliant design of crawling solutions
• Selenium, Playwright, or comparable browser automation
• Azure services and Databricks deployment
• Experience with product, asset, or type catalogs
• Ontologies, semantic models, and technical characteristic catalogs
• EAM and SAP attribute models
• Electrical engineering domain understanding

Project details

01.11.2026

Duration / End

31.12.2026

Start / End

01.11.2026 - 31.12.2026

Work Location

Payment Terms

30 days

Fee

Please state your hourly rate on an all-in basis

Other

If you can submit a qualified offer for this request, please send us a meaningful profile, including a concrete statement on availability and conditions. Please note that we will use the data you provide exclusively for the purpose of mediating project assignments in accordance with §5 BDSG. Disclosure to third parties within this scope will only occur after obtaining your consent. Thank you in advance for your support.

• Start: 01.11.2026
• Duration: bis 31.12.2026
• Remote: remote
• Location: Remote, DE
• Languages: German