Data Engineer
BT-66 – Data Engineer
Skill Level: Senior
Location: Herndon/Chantilly (fully on-site, no remote option)
Develop new tools, code, and services to execute data engineering activities involving data of varying types and in varying conditions. Activities include the following tasks: Movement of structure and unstructured data using approved methods. Execute data ingestion activities for storing data in a local or enterprise level location. Develop code to format data that supports exploration. Analyze source data formats and work with Data Scientists and partners to determine the formats and transforms that best meet objectives. Develop code and tools to provide one-time and on-going data extraction from various repositories, formatting and transformations into enterprise or standalone data models. Develop new ETL and perform O&M and enhancements on existing ETL code using best practices/standards. Develop and deliver documentation for each project including ETL mappings, code use guide, code location and access instructions.
- Design and optimize Data Pipelines using tools such as Spark, Apache Iceberg, Trino, OpenSearch, EMR cloud services, NiFi and Kubernetes containers
- Ensure the pedigree and provenance of the data is maintained such that the access to data is protected
- Clean and preprocess data to enable access for advanced analytics
- Collaborate with enterprise working groups to advance the state of data standards
- Collaborate with the engineering team, data stewards, and partners to aid in getting actionable value out of the data holdings
- Collaborate with software engineers to update, configure, and maintain data services based on the requirements
- Ensure data quality by working with the testing and data quality team to enhance standardization of data conditioning pipelines
- Experience adapting to various types and formats of data, and working with development teams to integrate new data processing platforms
10+ years' experience with:
- Data lifecycle engineering
- Development and maintenance of extract, transform and load (ETL) tools and services
- Cloud and on-prem data storage and processing solutions
- Python, SQL, Spark and other data engineering programming
- COTS and open source data engineering tools such as ElasticSearch and NiFi
- Processing data within the Agile Lifecycle
Recommended Jobs
Unit Secretary - Emergency Department - Harbour View Medical Center
At Bon Secours Mercy Health, we are dedicated to continually improving health care quality, safety and cost effectiveness. Our hospitals, care sites and clinicians are recognized for clinical and oper…
Office Admin
Our client, a leading organization in the energy services industry, is seeking a dedicated Office Admin to join their team. As an Office Admin, you will be part of the administrative support team, ass…
Emergency Veterinarian - Winchester, VA
&##128680; Emergency Veterinarians – Let’s Talk Flexibility, Culture & Mountains of Perks! &##128062; Valley Veterinary Emergency Hospital (VVERC) is on the lookout for experienced and …
Associate Veterinarian - Alexandria Animal Hospital of Fort Hunt
Overview: At Hollin Hall Animal Hospital , we're on a mission to build something truly special, and we want you to be part of it! We’re looking for an experienced Associate Veterinarian who’s eage…
Graphic Designer
Who Are We? At Parry Labs we believe the modern combat advantage relies on the speed and scale of open systems. We’re redefining the edge of the increasingly ever-changing battlespace by deliverin…
Cloud Infrastructure DevOps Engineer
Who We’re Looking For (Position Overview): Spry Methods is on the search for a DevOps Engineer to join our team in Herndon, VA. You will work in a highly critical production environment where your…
School Psychologist
The School Psychologist works to support students through avariety of mental health and developmental issues and aids in qualifyingstudents for additional support services. The School Psychologist is …
Data Scientist
SteerBridge Strategies is a CVE-Verified Service-Disabled, Veteran-Owned Small Business (SDVOSB) delivering a broad spectrum of professional services to the U.S. Government and private sector. Backed…
Diabetes Educator
Full job description Bayview Physicians Group is a rapidly growing outpatient multi-specialty medical group, that strongly believes the doctor-patient relationship is the cornerstone of quality he…
R
Veterinary Practice Manager at Banfield Pet Hospital Practice Managers play a pivotal role in our hospitals. Your medical practice management skills will be put to use to drive business results,…