Group Manager- Data Engineering | Azure Databricks | Snowflake SQL | Team Management

Data Engineering Lead

We are looking for a Data Engineering Lead with 8 to 12 years of experience to lead the build of data pipelines for an enterprise finance data platform on Azure, Databricks and Snowflake. You will run the data engineering team, own delivery from source systems through to certified Gold data products, and set the engineering standards the team works to.

KEY RESPONSIBILITIES

Lead a team of data engineers building ingestion and transformation pipelines

Deliver ingestion from finance source systems using change data capture, Azure Data Factory batch loads, API and event-driven feeds, and governed manual uploads

Build the Bronze, Silver and Gold layers in Databricks, including conformed dimensions and Gold data domains

Build Gold data domains for financial control, FP&A;, tax, recharges and project accounting with the Data Architect

Publish certified data products from Databricks to Snowflake for reporting and analysis

Build data quality, reconciliation and matching checks into pipelines with the data quality team

Move recurring Dataiku analyst workflows into engineered Databricks pipelines

Set coding standards, branching, code review and CI/CD practices in Azure DevOps

Estimate work, plan sprints and report progress to the Engagement Manager

Tune pipeline performance and manage compute usage

Support production issues and handover to the support team

TECHNICAL SKILLS

Must-have

Databricks, including PySpark, Spark SQL, Delta Lake, Workflows and Unity Catalog

Snowflake SQL, data loading and performance tuning

Azure Data Factory

Python and advanced SQL

Dimensional modelling

Git and CI/CD with Azure DevOps

Leading engineering teams in agile delivery

Good-to-have

Informatica CDC and PowerCenter

API Management, Service Bus and Functions

Delta Live Tables

Dataiku

Microsoft Purview

DOMAIN KNOWLEDGE

Banking finance data, including general ledger, sub-ledgers, intra-group and recharges

Finance systems such as SAP ECC, BW and BPC, TM1, Alfa and other lending platforms

Treasury and banking data from systems such as TMS, T24 and bank statement feeds

CERTIFICATIONS (GOOD TO HAVE)

Databricks Certified Data Engineer Professional

SnowPro Advanced: Data Engineer

EXPERIENCE AND QUALIFICATIONS

8 to 12 years in data engineering, with at least 3 years leading teams

Delivered at least one cloud data platform on Databricks or Snowflake into production

Experience on data programmes in banking or financial services

Degree in engineering or computer a

Function: Research
Experience Level: Mid-Senior Level