Lead Azure Databricks Engineer

Azure Databricks Engineer
Lloyd's of London Market

Hybrid 2 days in London Office

We are seeking an outstanding Lead Azure Databricks Engineer to own the design, build, optimisation and governance of our enterprise cloud data platforms. This is a senior, fully hands-on engineering position requiring deep technical expertise, strong delivery capability and experience operating within the Lloyd's of London market and its regulatory demands.

You will shape the future of our cloud data strategy, enabling advanced analytics and building secure, scalable and highly performant data solutions across the organisation.

. Lead the development, optimisation and governance of large-scale data platforms using Azure Data Factory, Data Lake, Key Vault, Azure Functions, Databricks, Delta Lake, PySpark and Unity Catalog
. Partner closely with underwriting, actuarial, delegated authority, bordereaux, exposure management, reinsurance, finance, Solvency II and risk teams to deliver solutions aligned to Lloyd's of London regulatory and operational requirements
. Translate complex business needs into scalable cloud data architectures in partnership with architects, SMEs and product owners
. Resolve complex technical challenges rapidly and decisively, consistently delivering high-quality outcomes
. Champion engineering best practices including design patterns, data life cycle management, CI/CD automation and cloud-native methodologies
. Provide hands-on technical leadership, mentorship and uplift engineering capability across the team
. Drive continuous improvement across performance, cost optimisation, reliability and resilience
. Oversee end-to-end data engineering delivery ensuring quality, observability, robustness and compliance

. Expert-level knowledge of Azure Data Services and Databricks within enterprise environments
. Deep understanding of Delta Lake, Medallion architecture, distributed compute, Lakehouse patterns, data modelling and performance optimisation
. Strong Python, PySpark and SQL experience, with hands-on implementation of CI/CD for data solutions using Azure DevOps
. Strong understanding of data governance, lineage, access management, FinOps and secure cloud engineering
. Excellent communication skills with the ability to confidently engage senior stakeholders and simplify complex technical concepts
. Proven ability to take ownership, deliver at pace and operate effectively in high-pressure, high-expectation environments
. Experience working within or delivering solutions for the Lloyd's of London market or similarly regulated environments

Please send me a copy if your CV if you're interested