Data Engineer
BT-66 – Data Engineer
Skill Level: Senior
Location: Herndon/Chantilly (fully on-site, no remote option)
Develop new tools, code, and services to execute data engineering activities involving data of varying types and in varying conditions. Activities include the following tasks: Movement of structure and unstructured data using approved methods. Execute data ingestion activities for storing data in a local or enterprise level location. Develop code to format data that supports exploration. Analyze source data formats and work with Data Scientists and partners to determine the formats and transforms that best meet objectives. Develop code and tools to provide one-time and on-going data extraction from various repositories, formatting and transformations into enterprise or standalone data models. Develop new ETL and perform O&M and enhancements on existing ETL code using best practices/standards. Develop and deliver documentation for each project including ETL mappings, code use guide, code location and access instructions.
- Design and optimize Data Pipelines using tools such as Spark, Apache Iceberg, Trino, OpenSearch, EMR cloud services, NiFi and Kubernetes containers
- Ensure the pedigree and provenance of the data is maintained such that the access to data is protected
- Clean and preprocess data to enable access for advanced analytics
- Collaborate with enterprise working groups to advance the state of data standards
- Collaborate with the engineering team, data stewards, and partners to aid in getting actionable value out of the data holdings
- Collaborate with software engineers to update, configure, and maintain data services based on the requirements
- Ensure data quality by working with the testing and data quality team to enhance standardization of data conditioning pipelines
- Experience adapting to various types and formats of data, and working with development teams to integrate new data processing platforms
10+ years' experience with:
- Data lifecycle engineering
- Development and maintenance of extract, transform and load (ETL) tools and services
- Cloud and on-prem data storage and processing solutions
- Python, SQL, Spark and other data engineering programming
- COTS and open source data engineering tools such as ElasticSearch and NiFi
- Processing data within the Agile Lifecycle
Recommended Jobs
System Engineer - Acquisition
Title: System Engineer - Acquisition System Engineer - Acquisition Belong. Connect. Grow. with KBR! KBR's National Security Solutions team provides high-end engineering and advanced techn…
Application Developer
Required Skills * Demonstrated experience using JIRA. * Demonstrated experience working on multi-disciplinary application development teams. * Collectively, the team possesses demonstrated experience …
Sales Associate
Requisition ID: 901910 Store # : 004413 Sunglass Hut Position: Part-Time Total Rewards: Benefits/Incentive Information At Sunglass Hut, we're always in the sun. You’ll find a dynamic…
Vocational Counselor I - Sterling, VA
Are you ready for an Accredited Career? Can you be the Change your community needs? Omega Health Services is a CARF Accredited leading provider in Vocational Rehabilitation for persons with disabilit…
Perm Placement - CT Tech OOJ - 33547
Job Description A CT technologist, or computed tomography technologist, uses a CT scanner to take images of the inside of a patient's body. They work in hospitals, clinics, and other medical facil…
Tax Manager
Job Description Job Description Tax Manager Tax Manager --Hiring for multiple offices throughout Virginia-- As a Top 100 firm, we deliver assurance, tax, and business advisory services to a …
Oral Surgery Assistant
Job Description: We are seeking a highly skilled and dedicated Oral Surgery Assistant to join our dynamic team. The ideal candidate will have experience assisting oral surgeons in a clinical setting…
Environmental data analyst
The South Florida Water Management District (SFWMD) is a State of Florida regional agency responsible for the water resources of a large region, from Orlando in central Florida to the Keys. The agen…
Direct Sales
Job Description Covington Promotions, a leader in direct sales, is excited to announce a job opening for a Direct Sales position. This part-time opportunity is ideal for individuals eager to join …
Remote Life Insurance Agent
We are seeking a Life Insurance Agent to join our fast-growing, family-operated insurance agency. We are looking for purpose-driven professionals who are ready to take control of their future, whethe…