Databricks Solution Architect

About This Opportunity

We are representing a confidential digital transformation consultancy seeking a Lead Databricks Solution Architect to design, build, and scale their cloud-based lakehouse platform. This is a remote opportunity based in the United States.

In this role, you will own the end-to-end architecture of the data ecosystem on Databricks, partner with data science and analytics teams to productionize ML and analytical workloads, and set the technical direction for ingestion, transformation, governance, and performance optimization across petabyte-scale datasets. You will be a hands-on technical leader: writing production code, mentoring engineers, and shaping standards that the broader data organization will adopt.

Responsibilities

• Architect and lead the implementation of an enterprise lakehouse on Databricks (Delta Lake, Unity Catalog, Photon, Workflows) across one or more major clouds (AWS, Azure, or GCP)
• Design scalable batch and streaming data pipelines using PySpark, Spark SQL, Structured Streaming, and Delta Live Tables; establish patterns for ingestion from operational systems, event streams, and third-party APIs
• Define and enforce platform standards for data modeling (medallion architecture), CI/CD, code quality, testing, observability, and cost optimization
• Lead the governance strategy using Unity Catalog — fine-grained access control, data lineage, audit, and PII handling — in partnership with security and compliance teams
• Optimize Spark workloads for performance and cost: cluster sizing, Photon, autoscaling, file layout, Z-ordering, caching, and query tuning
• Partner with ML engineers and data scientists to operationalize models using MLflow, feature stores, and model serving on Databricks
• Own the cloud infrastructure footprint for the platform: networking, IAM, secrets, encryption, and Terraform/IaC for Databricks workspaces and supporting services
• Mentor a team of data engineers; lead architecture reviews, code reviews, and technical design sessions; raise the bar on engineering practices
• Engage with stakeholders across analytics, product, and finance to translate business needs into a roadmap for the data platform
• Promote and enforce information security best practices, including data protection, access controls, and compliance with GDPR, CCPA, and other privacy standards

What We're Looking For

• 8+ years of data engineering experience, with 4+ years building production workloads on Databricks
• Deep expertise in Apache Spark (PySpark and Spark SQL) — including performance tuning, partitioning strategy, and the Catalyst/Photon execution model
• Strong hands-on experience with Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables
• Production experience on at least one major cloud platform (AWS, Azure, or GCP), including networking, IAM, storage (S3/ADLS/GCS), and compute primitives
• Proficiency in Python and SQL; comfort with Scala is a plus
• Experience designing medallion (bronze/silver/gold) architectures and dimensional models for analytics
• Strong CI/CD and DevOps practice: Git, Terraform, Databricks Asset Bundles or dbx, automated testing of data pipelines
• Track record of leading technical projects end-to-end and mentoring engineers
• Excellent written and verbal communication skills; able to drive alignment with both engineering and business stakeholders

How We Work With You

Our talent team at Get A Job.ai partners with leading organizations to connect exceptional candidates with career-defining opportunities. When you apply through our platform, a dedicated recruiter will screen your profile and arrange an introduction to our client. We manage the entire process to ensure a smooth experience. Please note: do not contact the client directly, as all communication is coordinated through Get A Job.ai.

Pay

Compensation details will be discussed during the screening process based on experience and qualifications.

Equal Opportunity: Get A Job.ai is committed to inclusive hiring practices and welcomes candidates from all backgrounds. We evaluate applicants without regard to race, religion, color, sex, gender identity, sexual orientation, age, disability, national origin, veteran status, or any other protected characteristic.