Azure Databricks Engineer

Plateau GRP is seeking a highly skilled Azure Databricks Engineer to support a large-scale data modernization initiative for the Federal Deposit Insurance Corporation (FDIC). The selected candidate will design develop and optimize enterprise-grade data lakehouse solutions using Azure Databricks Apache Spark and related Azure data services.

This position requires deep technical expertise in data engineering advanced analytics enablement and large-scale data pipeline optimization. The ideal candidate will collaborate with cross-functional teams to deliver secure scalable and high-performing data infrastructure within a modernized cloud environment.

Key Responsibilities

Data Pipeline Development & Management

• Design build and maintain scalable ETL/ELT pipelines using Azure Databricks Delta Lake and Apache Spark to process large volumes of structured and unstructured data.
• Integrate data from multiple sources using Azure Data Factory Azure Data Lake Storage and Azure Synapse Analytics.

Data Architecture & Optimization

• Architect and optimize data lakehouse solutions leveraging Delta Lake format and partitioning strategies.
• Implement data modeling best practices for efficient data storage retrieval and analytical performance.

Collaborative Analytics Platform Management

• Configure and manage Databricks workspaces clusters and compute resources.
• Implement security controls access management and cost optimization strategies aligned with FDIC standards.

Real-time & Batch Processing

• Develop streaming data solutions using Structured Streaming and implement batch processing jobs for transformation aggregation and integration.
• Ensure performance scalability and reliability of both real-time and batch data workflows.

MLOps & Data Science Support

• Collaborate with data scientists to deploy machine learning models and integrate them into data pipelines.
• Implement MLflow for model lifecycle management and automate ML pipelines for training and inference.

Performance Monitoring & Troubleshooting

• Monitor pipeline performance and system health across Databricks and Azure environments.
• Troubleshoot job failures implement data quality checks and optimize Spark jobs for improved efficiency and reduced processing cost.

Requirements

Required Qualifications

• BA/BS/MS in Computer Science Engineering Data Science or equivalent experience.
• Over 5 years of experience as a Data Engineer with 3 years specifically in Azure Databricks and related Azure data services.
• This position requires US Citizenship and the ability to pass/obtain any required government background check or security clearance. An active security is preferred.
• Expertise in Apache Spark Delta Lake Python SQL and Scala.
• Hands-on experience designing developing and optimizing large-scale data pipelines in cloud environments.
• Technical proficiency with:
• Azure Data Factory Azure Data Lake Storage and Azure Synapse Analytics
• ETL/ELT processes data modeling and big data architectures
• Strong understanding of data governance data security and cloud operations in enterprise environments.
• Experience working within Agile delivery frameworks.

Preferred Qualifications

• Microsoft Certified: Azure Data Engineer Associate (DP-203) or Databricks Certified Data Engineer certification.
• Experience supporting federal data modernization or enterprise cloud transformation programs.
• Familiarity with DevOps/CI-CD pipelines GitHub Enterprise and Infrastructure as Code (Terraform).
• Experience supporting AI/ML initiatives and integrating data science workflows within Azure Databricks.

Plateau does not accept unsolicited resumes from third-party recruiters without a signed third-party agreement. Any unsolicited third-party resumes forwarded by recruiters to Plateau or to any of our managers or employees will be considered public information may be treated as a direct application from the person identified in the resume and will not be eligible for placement fee payment to the agency.

EEO Statement:

Plateau is an equal opportunity/affirmative action employer; all qualified applicants will receive consideration for employment without regard to age gender gender identification sex sexual orientation color race creed national origin religion marital status parental status citizenship status ancestry physical or mental disability genetic information veteran status military status or any other classification protected by federal state or local laws.

Plateau is committed to providing access and reasonable accommodations for individuals with disabilities. If you require reasonable accommodation please email the Plateau HR Department at

Plateau is a proud Equal Opportunity Employer! (M/F/Disability/Vets)