Senior Data Engineer

Description

In this important role as a PySpark Senior /Lead Data Engineer you will provide engineering and support for migration from SAS Viya to AWS, Databricks, and all data-related services, including managing design, implementation, operation, and monitoring on systems provided by the Federal Government. The PySpark Senior /Lead Data Engineer reports to the Director of Information Systems.

The PySpark Senior /Lead Data Engineer reports to the Director of Information Systems. Responsibilities include, but not limited to:

Responsibilities include, but not limited to:

• Develop a plan for migration of programs and data developed in SAS Viya to AWS Data Lake, Databricks PySpark, and Databricks SQL.

• Develop and optimize ETL processes, including extracting data from source systems, transforming it into a usable format, and loading it into the target storage.

• Create and maintain data pipelines that facilitate the flow of data from various sources to Data Lake or other storage systems.

• Implement processes to ensure data quality and integrity, including validating and cleansing data to maintain its quality and integrity.

• Work closely with cross-functional teams, including SMEs, quality assurance, report developers, administrators, and stakeholders to identify and address data-related issues.

• Troubleshoot and resolve issues with Data Lake, Pipelines, Reports, and related infrastructure; escalate to other technology support resources as needed.

• Schedule, communicate, perform, and monitor both regular and ad hoc pipeline activities in accordance with defined policies and procedures.

• Participate in the creation (or modification) of documentation activities related to system design, configuration, operation, support policies, and compliance procedures.

• Design and implement scalable data architectures, ensuring efficient data flow and storage.

Qualifications

• Demonstrate expertise in software development using PySpark and Databricks in AWS Cloud Services.

• Minimum of five years of experience specifically in software development in PySpark with Databricks.

• Preferably, two years of experience using SAS Viya.

• Experience using version control software, preferably GitHub.

• Excellent written and verbal communication skills in English.

All work must be performed in the United States of America.

Skills

pyspark, Databricks, etl, big data, sas

Top Skills Details

pyspark,Databricks,etl,big data,sas

Additional Skills & Qualifications:

Communication is a TOP priority for this role. It will require a high level of communication to the existing staff/team as well as key stakeholders. This role will require someone that is more than just a "do-er", they need someone to teach the team along the way and guide them until they can get up to speed. PySpark is something new to this environment and they need someone that is willing and excited to not only share their knowledge, but make sure they are setting the team up for success.

Experience Level

Expert Level

About TEKsystems:

We're partners in transformation. We help clients activate ideas and solutions to take advantage of a new world of opportunity. We are a team of 80,000 strong, working with over 6,000 clients, including 80% of the Fortune 500, across North America, Europe and Asia. As an industry leader in Full-Stack Technology Services, Talent Services, and real-world application, we work with progressive leaders to drive change. That's the power of true partnership. TEKsystems is an Allegis Group company.

The company is an equal opportunity employer and will consider all applications without regards to race, sex, age, color, religion, national origin, veteran status, disability, sexual orientation, gender identity, genetic information or any characteristic protected by law.