Azure Data Engineer

The Opportunity:

We are currently seeking an experienced Data Engineer to join a high-profile data transformation programme, helping to build and enhance a modern cloud-based data platform using Azure technologies, who combines strong Azure Databricks expertise with modern data engineering practices and a passion for solving complex data challenges.

This role will focus on designing and developing scalable data engineering solutions within Azure Databricks, supporting the creation of robust data pipelines and contributing to the evolution of a modern Lakehouse architecture. The successful candidate will also work extensively with geospatial datasets, making this an excellent opportunity for engineers with experience in spatial data processing and geospatial analytics.

Working within a collaborative Agile delivery environment, you will partner closely with stakeholders, product owners and technical teams to deliver high-quality, scalable and maintainable data solutions that support critical business and analytical requirements.

If you're an experienced Data Engineer with strong Azure Databricks expertise and experience working with geospatial data (ideally within the Public Sector), we'd love to hear from you.

Skills and Experience:

• Strong commercial experience working as a Data Engineer within Azure cloud environments.
• Proven hands-on expertise with Azure Databricks in enterprise-scale production environments.
• Strong programming skills using: Python / SQL / PySpark / Spark SQL.
• Experience designing and building scalable data pipelines and data processing frameworks.
• Strong understanding of modern Lakehouse architecture principles.
• Experience working with: Unity Catalog / Databricks Workflows / Delta Live Tables / Databricks SQL.
• Experience integrating data from APIs and external data sources.
• Strong understanding of data modelling, data quality and performance optimisation techniques.
• Experience working within Agile software delivery environments.

Geospatial Experience

Candidates should have experience working with geospatial datasets and technologies, including:

• Shapefiles
• GeoParquet
• GeoPackage
• GeoTIFF
• Spatial data processing and transformation techniques
• Geospatial analytics and data engineering workflows

Experience with Apache Sedona or similar geospatial processing frameworks is highly desirable.

Role and Responsibilities:

• Collaborate with stakeholders, product owners and technical teams to understand data requirements and translate them into scalable engineering solutions.
• Design, develop and maintain high-performance data pipelines using Azure Databricks, PySpark, Spark SQL and Python.
• Contribute to the design and implementation of Lakehouse architecture using Delta Lake and Databricks best practices.
• Develop and manage data workflows using Databricks Workflows and Delta Live Tables.
• Utilise Unity Catalog to support effective governance, security and data discovery.
• Develop analytical and reporting datasets using Databricks SQL.
• Ingest, transform and process data from APIs and a variety of structured, semi-structured and geospatial data sources.
• Apply geospatial processing techniques to support data integration, analysis and business requirements.
• Write clean, maintainable and well-tested code in line with engineering best practices.
• Participate in Agile ceremonies, technical design discussions, code reviews and sprint planning activities.
• Support data quality, performance optimisation and operational monitoring across data pipelines.

NB: The successful candidate will be required to undergo a basic level of security clearance before undertaking the assignment (around two weeks lead-time).

Applications:

To learn more about this newly created opportunity consulting as an Azure-focused Data Engineer for an initial 6 month engagement, operating on a remote-working basis; please call and speak with Edward Laing here at ISR Recruitment or please send through a copy of your latest online profile and/or CV for an immediate call back in the strictest of confidence.