Senior Data Engineer
We are seeking a Senior Data Engineer to lead the development and optimization of data pipelines, data movement tools, and data preparation services to support advanced analytics and enterprise integration. This role requires deep expertise in handling both structured and unstructured data, designing scalable data architectures, and developing reliable ETL solutions. The ideal candidate has significant experience with both cloud-native and on-premise technologies, and thrives in collaborative, mission-driven environments.
Clearance Requirement: TS/SCI with Polygraph
Key Responsibilities:
Design, implement, and maintain scalable data pipelines using tools such as Spark, Apache Iceberg, Trino, OpenSearch, EMR, NiFi, and Kubernetes
Perform data ingestion, extraction, formatting, and transformation for a wide range of data types and formats
Clean, preprocess, and format data for exploration, analysis, and operational use
Develop ETL tools and scripts and enhance existing pipelines using coding best practices
Work with Data Scientists, analysts, and mission partners to align data pipelines with mission goals
Collaborate with software engineers to configure and maintain data services and back-end components
Ensure data quality, standardization, and provenance through collaboration with testing and data quality teams
Produce comprehensive documentation including ETL mappings, data lineage, code usage guides, and access instructions
Participate in enterprise working groups to define and promote data standards across systems
Support both one-time and ongoing data extraction and transformation efforts across various repositories
Required Qualifications:
10+ years of experience in data lifecycle engineering
Deep expertise in ETL development, pipeline design, and data architecture
Strong coding skills in Python, SQL, Spark, and other data engineering tools
Experience with COTS and open-source tools including ElasticSearch and Apache NiFi
Strong understanding of cloud and on-premise data storage and processing solutions
Experience working in Agile development environments
Proven track record of integrating diverse data sources into enterprise systems
Preferred Qualifications:
Experience deploying solutions on cloud platforms (e.g., AWS EMR, S3, RDS, EKS)
Familiarity with data governance, data lineage tracking, and security requirements
Hands-on experience with container orchestration using Kubernetes
Knowledge of data lake architectures and streaming data processing
Excellent communication skills and experience collaborating with cross-functional teams
Tech Stack & Tools:
Languages & Frameworks: Python, SQL, Spark
Data Tools: Apache NiFi, Apache Iceberg, Trino, ElasticSearch, OpenSearch
Cloud & Infrastructure: AWS EMR, Kubernetes, Docker, On-Prem Systems
DevOps & CI/CD: Git, Agile methodologies, automated testing frameworks
Why Join M1?
Tackle large-scale data engineering challenges with national impact
Work with cutting-edge cloud and open-source data tools
Collaborate with a multidisciplinary team of engineers and analysts
Flexible, mission-driven environment with opportunities for leadership
M1 Technology is an equal opportunity employer and values diversity. We do not discriminate in hiring on the basis of race, color, religion, sex, national origin, age, disability, veteran status, or any other characteristic protected by federal, state, or local law.
Recommended Jobs
Principal Member of Technical Staff - Oracle Health
**Job Description** Oracle Health is dedicated to creating people-centric healthcare experiences powered by unified global data. We act with the speed and attitude of a start-up, with the scale and cu…
Full Stack Engineer (Typescript/Node.js)
*Candidates must be local to the Washington D.C. metro area and must possess work authorization which allows them to be hired as a full-time Infinitive employee without the need for Visa sponsorship. …
Assistant Store Manager PT
About Our Company Lids Sports Group is the largest licensed sports retailer in North America, selling fan and fashion-oriented headwear and apparel across the US, Canada, Mexico, Europe, and Austr…
Egg Processing Associate
Want to build a stronger, more sustainable future and cultivate your career? Join Cargill's global team of 160,000 employees who are committed to safe, responsible and sustainable ways to nourish th…
Software Engineer, AI
WHAT IS AIS? Anduril Intelligence Systems (AIS) is focused on positioning Anduril as a lead provider of specialized engineering and products for Intelligence Community (IC) customers. We work wit…
Senior Enterprise Analysis Lead
Title: Senior Enterprise Analysis Lead KBR's National Security Solutions team provides high-end engineering and advanced technology solutions to our customers in the intelligence and national s…
Medical Sales Representative
Medical Sales Representative – Commission Only Type: 1099 Independent Contractor | Commission-Only About the Role We are seeking motivated Medical Sales Representatives to introduce provider…
Cybersecurity Specialist - Advanced
JRC is actively recruiting for a Cybersecurity Specialist-Advanced to assist in the assessment of the DoD current cyber policies, the coverage of those policies, and areas of risks. You will evaluat…
Transit Operator - Part Time - CDL required
TRANSIT OPERATORS – PART TIME - CDL REQUIRED STARTING HOURLY RATE: $19.99 Responsible for transporting passengers on designated routes within the transit system’s service area; including, the…
US Tech - Technical Business Analyst Manager Save for Later Remove job
At PwC, our people in software and product innovation focus on developing cutting-edge software solutions and driving product innovation to meet the evolving needs of clients. These individuals com…