Data Engineer – Unified Data Layer (UDL) (f/m/div.)
Prefer a full account with a password? Sign Up to Apply
Job Description
Platform & Pipeline Engineering
• Develop, automate, and maintain resilient data pipelines connecting on-premises
• laboratory equipment, high throughput test benches, and edge systems to Microsoft Azure.
• Investigate, unpack, and parse complex, non-standard, binary, proprietary, and undocumented file formats where standard connectors do not exist.
• Implement parsers and normalization routines capable of ingesting diverse technical data streams, for example:
• High-frequency sensor streams, time series, and bus telemetry.
• Multi-dimensional scientific arrays, spatial grids, and spectral data.
• - Hierarchical, nested, and unstructured binary logs or custom ASCII exports.
• Event logs, operational metadata, and intermittent experimental batch runs.
• Clean, validate, and transform fragmented or schema-less raw data into optimized, partitioned Lakehouse tables (Delta Lake/Parquet) ready for analytics and ML model training.
• Deploy, monitor, and maintain pipeline resources on Azure using Terraform, Databricks, and containerized workflows following established team templates.
Collaboration & Lab Support
• Work directly alongside experimental scientists, lab engineers, and technicians to decipher raw measurement files, clarify domain-specific semantics, and automate recurring data uploads.
• Collaborate closely with solution architects and senior engineers to implement data ingestion patterns, cataloguing schemas, and governance standards.
• Maintain clear technical documentation, code comments, and onboarding walkthroughs to help research teams connect their equipment seamlessly.
• Support proof-of-concept initiatives exploring metadata integration with corporate AI assistants, data discovery catalogs, and modern data tools.
Requirements
Function: Engineering
Experience Level: Mid-Senior Level