Pyspark Developer

PySpark + Fabric Developer (Contract) London Office-Based

Location: London (Office-based)
Contract: 6 months (potential extension)
Start: ASAP
Rate: Market rate - Inside IR35

We re looking for experienced PySpark + Fabric Developers to join a major transformation programme with a leading global financial data and infrastructure organisation. This is an exciting opportunity to work on cutting-edge data engineering solutions, driving innovation and performance at scale.

Key Responsibilities

• Design, build, and optimise data pipelines for both batch and streaming workloads.
• Develop and manage dataflows and semantic models to support analytics and reporting.
• Implement complex data transformations, aggregations, and joins with a focus on performance and reliability.
• Apply robust data validation, cleansing, and profiling techniques to maintain accuracy.
• Enforce role-based access, data masking, and compliance standards.
• Tune and optimise workloads to minimise latency and enhance throughput.
• Collaborate with analysts and stakeholders to translate business needs into technical solutions.
• Maintain clear documentation and contribute to internal best practices.

Requirements

• Strong hands-on experience with PySpark (RDDs, DataFrames, Spark SQL).
• Proven ability to build and optimise ETL pipelines and dataflows.
• Familiar with Microsoft Fabric or similar lakehouse/data platform environments.
• Experience with Git, CI/CD pipelines, and automated deployment.
• Knowledge of market data, transactional systems, or financial datasets.
• Excellent communication skills and collaborative mindset.

Desirable

• Experience with Azure Data Lake, OneLake, or distributed computing environments.
• Understanding of data security and compliance (e.g., GDPR, SOX).
• Exposure to preparing datasets for Power BI.