Python / PySpark DATA Engineer

RESPONSIBILITIES: Kforce has a client that is seeking a Data Engineer in New Brunswick, NJ. Summary: The Data Lake will be built on the Azu...

RESPONSIBILITIES:
Kforce has a client that is seeking a Data Engineer in New Brunswick, NJ.

Summary:
The Data Lake will be built on the Azure Cloud, with pipelines implemented in a mixture of PySpark and SparkSQL triggered from a REST API. All infrastructure will deployed using scripted processes developed with Agile methodologies. Software Engineers will be responsible for building and maintaining both the Data Lake infrastructure and multiple ETL pipelines that run on it.

Essential Duties of the Data Engineer:
* Data Engineer will work with Software Engineers and business experts to clarify requirements for components and features of the Data Lake
* Design, document and implement solutions to problems involving data ingestion, transformation, storage, and security using Azure tools and best practices
* Data Engineer will follow and enforce strict standards for code quality, automated testing, infrastructure-as-code and code maintainability

REQUIREMENTS:
* Data Engineer with 5-6 years of experience
* This will require someone who has hands-on experience developing systems that leverage multiple AWS services including CloudFormation, Lambda, Glue and API Gateway for data management and analysis
* Experience migrating legacy systems into an Enterprise Data Lake is a plus
* Big Data ETL/ELT experience
* REST API design ( Python - Flask)
* Azure Databricks experience ( 1-2 years minimum)
* Data Engineer with PyTest / PyLint experience ( 2-4 years)
* Azure Data Lake store ( 1-2 years)
* Kubernetes experience will be a plus
* Data Interoperability experience will be a plus

Kforce is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status. - provided by Dice