Python / PySpark DATA Engineer

Company: Kforce Technology
Location: New Brunswick, New Jersey, United States
Type: Full-time
Posted: 11.AUG.2021
< >

Summary

RESPONSIBILITIES: Kforce has a client that is seeking a Data Engineer in New Brunswick, NJ. Summary: The Data Lake will be built on the Azu...

Description

RESPONSIBILITIES:

Kforce has a client that is seeking a Data Engineer in New Brunswick, NJ. Summary: The Data Lake will be built on the Azure Cloud, with pipelines implemented in a mixture of PySpark and SparkSQL triggered from a REST API. All infrastructure will deployed using scripted processes developed with Agile methodologies. Software Engineers will be responsible for building and maintaining both the Data Lake infrastructure and multiple ETL pipelines that run on it. Essential Duties of the Data Engineer:
  • Data Engineer will work with Software Engineers and business experts to clarify requirements for components and features of the Data Lake
  • Design, document and implement solutions to problems involving data ingestion, transformation, storage, and security using Azure tools and best practices
  • Data Engineer will follow and enforce strict standards for code quality, automated testing, infrastructure-as-code and code maintainability


Job Requirements:

REQUIREMENTS:

  • Data Engineer with 5-6 years of experience
  • This will require someone who has hands-on experience developing systems that leverage multiple AWS services including CloudFormation, Lambda, Glue and API Gateway for data management and analysis
  • Experience migrating legacy systems into an Enterprise Data Lake is a plus
  • Big Data ETL/ELT experience
  • REST API design ( Python - Flask)
  • Azure Databricks experience ( 1-2 years minimum)
  • Data Engineer with PyTest / PyLint experience ( 2-4 years)
  • Azure Data Lake store ( 1-2 years)
  • Kubernetes experience will be a plus
  • Data Interoperability experience will be a plus
Kforce is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.

 
Apply Now

Share

Flash-bkgn
Loader2 Processing ...