Architect (AWS, Python, Spark)

Company: Computer Enterprises, Inc.
Location: Philadelphia, Pennsylvania, United States
Type: Full-time
Posted: 13.MAR.2021

Summary

Architect (AWS, Python, Spark) Location: Philadelphia, PA 19103 **open to remote** Summary This role will be the technical lead for the ...

Description

Architect (AWS, Python, Spark)

Location: Philadelphia, PA 19103

**open to remote**

Summary

This role will be the technical lead for the ROCI project, which is a graph representation of the access network. As part of this project, they would be responsible for extracting data from various disparate data sources through ETL scripts and storing them in S3 and eventually importing them into AWS Neptune database. They would then also be responsible for developing APIs to expose to consume data from the Neptune DB. They will be responsible for design and architecting the E2E solution and providing guidance to junior data engineers to get the work done. They should be very proficient in Python, especially using libraries like Pandas, etc. They should have experience working with Databricks. They should be very familiar using AWS services.

Responsibilities:

  • Developing large scale data pipelines exposing data sources within the company's team of data analysts and data scientists.
  • Developing REST APIs utilizing AWS lambda and API Gateway.
  • Developing Spark streaming and batch jobs to clean and transform data.
  • Writing build automation to deploy and manage cloud resources.
  • Writing unit and integration tests.

Job Requirements:

  • 10 years of experience working as a software engineer
  • 5 years of experience working within an enterprise data lake/warehouse environment or big data architecture
  • Experience working as a lead on data engineering projects
  • Extensive experience with Python using Pandas
  • Extensive experience with Cloud Computing
  • Strong experience with AWS (Lambdas, S3, EC2)
  • Strong experience with Spark (AWS EMR, Databricks)
  • Soft Skills: Can do attitude!

Nice to Have Skills:

  • Scala
  • AWS Neptune
  • QuickSight, Tableau

Some of the specific technologies we use:

  • Programming Languages (Python, Scala, Golang, Node.js)
  • Build Environment: GitHub Enterprise, Concourse CI, Jira, Serverless, SAM
  • Cloud Computing (AWS Lambda, EC2, ECS)
  • Spark? (AWS EMR, Databricks)
  • Stream Data Platforms: Kinesis, Kafka

- provided by Dice

 
Apply Now

Share

Flash-bkgn
Loader2 Processing ...