Principal Azure Data Engineer (Databricks)

Responsibilities

We are representing a consulting firm working with financial services organizations to modernize their data capabilities through cloud-native platforms. As a Principal Azure Data Engineer, you will architect and lead the delivery of enterprise-grade data solutions using Databricks within the Azure ecosystem. You'll design and implement streaming and batch data pipelines, working with cross-functional teams to guide clients on technical strategy and best practices while ensuring quality, security, and governance standards are maintained across deployments.

Your responsibilities will include:

• Architecting and leading end-to-end delivery of Databricks-based data platforms across Azure environments
• Designing secure, scalable data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog, and Azure-native services, applying AI-enabled engineering approaches where appropriate
• Establishing and promoting engineering best practices across the data lifecycle, ensuring robust governance, quality, and operational excellence
• Partnering with clients to define technical strategy, data architecture, and governance frameworks that align with business objectives
• Mentoring engineering teams, developing internal capability, and encouraging the responsible use of approved AI tools and automation to enhance delivery and collaboration

What We're Looking For

Our client seeks a principal-level engineer with proven expertise in modern data platforms and cloud architecture:

• Proven expertise with the Databricks platform, including Unity Catalog, Delta Lake, orchestration, and enterprise-scale Azure deployments
• Strong technical skills in Python, PySpark, and distributed data processing frameworks, with experience building production-grade data platforms
• Extensive experience implementing CI/CD pipelines using Azure DevOps, Jenkins, GitHub Actions, or equivalent DevOps tooling
• Deep understanding of Data Lakehouse architecture, data modelling, security, and GDPR-compliant solution design
• Experience using AI-enabled tools responsibly to improve engineering efficiency while maintaining appropriate human oversight, governance, and delivery quality

Additional valued experience:

• Strong client-facing consulting and commercial experience supporting pre-sales, solution shaping, and RFP responses
• Experience coaching and mentoring engineering teams while fostering engineering excellence
• Development experience with Scala or Java
• Knowledge of sensitive data management, PII handling, and regulatory frameworks within financial services
• Active contribution to technical communities, thought leadership, and the exploration of responsible AI and automation to improve engineering practices

How We Work with You

Candidates apply directly through Get A Job.ai. Our talent team will screen qualified applicants and coordinate introductions with our client. We manage the process end-to-end and serve as your advocate throughout. Please do not contact the client directly—all communication should flow through Get A Job.ai to ensure a smooth, professional experience.

Pay

Compensation details will be discussed with qualified candidates during the screening process.

Equal Opportunity: Get A Job.ai is committed to inclusive hiring practices. We welcome applications from candidates of all backgrounds and will ensure fair consideration throughout our process.