Sr Data Engineer

About This Opportunity

Our talent team is representing a confidential insurance technology organization that is building production data infrastructure on a modern cloud lakehouse platform. This is a hands-on engineering role where you'll own end-to-end data pipelines—from ingestion through transformation to trusted consumption-ready datasets—using Databricks, Delta Lake, Python, SQL, and orchestration tooling on Azure.

You'll work embedded on a delivery pod that owns its data products from design through production operation, not servicing tickets from a queue. You'll write production-grade code, enforce data contracts, implement quality as code, and help the organization treat data as a product with named owners and real consumers.

Responsibilities

• Design, develop, and maintain production data pipelines on Databricks using Python, SQL, Apache Spark, and Delta Lake across a medallion architecture (Bronze/Silver/Gold layers).
• Build Bronze-layer ingestion from APIs, relational databases, flat files, cloud storage, and SaaS platforms using dlt (dltHub) and Databricks-native patterns—including incremental loading, pagination, watermarking, state management, and replay after failure.
• Develop Silver-layer transformations in dbt and Python over Delta Lake that cleanse, standardize, validate, deduplicate, conform, and enrich data for reuse across domains.
• Create Gold-layer data products: dimensional models, slowly changing dimensions, fact and bridge tables, aggregates, and serving tables aligned to consumption patterns.
• Produce curated datasets for ML engineering: versioned, reproducible feature and training tables.
• Author and maintain data contracts using the Open Data Contract Standard (ODCS)—schema with semantics, named owner, known consumers, quality rules, and freshness expectations—and assess backward compatibility before changes.
• Implement data quality as code: uniqueness and not-null on keys, plus referential, accepted-value, freshness, and custom business-rule tests surfaced to producers and consumers.
• Orchestrate ingestion and transformation as assets in Dagster deployed to Dagster Cloud across development, branch, and production environments—schedules, sensors, dependencies, backfills, and observability.
• Apply governance through Unity Catalog—catalogs, schemas, external locations, grants, row- and column-level security, and lineage—and manage credentials through Azure Key Vault.
• Implement incremental and merge-based processing with Delta Lake (MERGE, schema evolution, time travel, OPTIMIZE) and tune Spark jobs, table layouts, and compute for performance and cost.
• Troubleshoot production failures, data quality issues, source-system changes, and late-arriving or duplicate data—including backfills and recovery—and participate in on-call rotation with root-cause analysis.
• Build and maintain CI/CD for data assets in Azure DevOps—automated tests and CI checks on dlt, dbt, and Dagster changes, promotion through environments, and auditable releases.
• Instrument pipelines for observability: freshness, volume, quality, latency, and cost, with alerting tied to SLAs and SLOs.
• Work within platform control expectations—least-privilege access, secrets in Azure Key Vault, change management through pull request and pipeline, and audit evidence built into deployment.
• Participate in code review and document architecture, runbooks, and data products for discoverability and reuse.
• Collaborate with data architects, analysts, product owners, and business stakeholders to translate requirements into maintainable solutions.

What We're Looking For

Required:

• Bachelor's degree in Computer Science, Information Systems, Engineering, or related field; equivalent practical experience considered.
• 3+ years building and supporting production data pipelines in a cloud data platform environment.
• Strong hands-on Python and SQL—both used daily.
• Hands-on experience with Databricks or comparable Spark-based lakehouse, including Delta Lake tables, MERGE, and incremental load patterns.
• Practical understanding of medallion/multi-layer lakehouse design and judgment on what belongs in Bronze versus Silver versus Gold.
• Experience ingesting data from APIs, relational databases, files, or SaaS applications, including incremental and state-management challenges.
• Working knowledge of dimensional modeling—grain, keys, facts and dimensions, slowly changing dimensions—and ELT design patterns and data quality practice.
• Experience with orchestration and scheduling using Dagster, Databricks Workflows, Airflow, Azure Data Factory, or similar.
• Git-based source control, pull request review, automated testing, and CI/CD as normal practice—Azure DevOps or comparable.
• Experience troubleshooting production data failures, performance bottlenecks, and source-system changes.
• Experience with pipeline monitoring and alerting, and understanding of freshness or quality SLAs with real consumers.
• Ability to explain technical designs and trade-offs to both technical and non-technical partners.

Preferred:

• Databricks certification (Data Engineer Associate or Professional) or equivalent demonstrated depth.
• Unity Catalog experience: catalogs, schemas, volumes, external locations, storage credentials, permissions, and lineage.
• dbt on Databricks or another transformation framework used alongside Spark.
• Python-based modeling frameworks over Delta Lake, and experience implementing Type 2 history, surrogate keys, and merge strategies in code.
• Experience with a declarative Python ingestion framework such as dlt (dltHub), Airbyte, Meltano, or Fivetran.
• Dagster experience specifically, including assets, asset checks, sensors, schedules, and branch deployments.

How We Work With You

When you apply through Get A Job.ai, one of our recruiters will screen your background and technical fit. If there's alignment, we'll submit your profile to our client for consideration. We ask that you do not contact the client directly—we manage the process end to end and advocate on your behalf throughout interviews, offer negotiation, and onboarding.

Pay

Pay details will be discussed during the screening process and are based on experience, location, and client budget.

Equal Opportunity: Get A Job.ai is an equal opportunity recruiter. We welcome applicants of all backgrounds and do not discriminate on the basis of race, color, religion, sex, national origin, age, disability, veteran status, or any other protected characteristic.