Sr. Data Engineer with Python

Company: Wavicle Data Solutions
Location: Knoxville, Tennessee, United States
Type: Full-time
Posted: 10.FEB.2021

Summary

About Wavicle Wavicle Data Solutions is built upon the principle that modern organizations rely on data to drive their businesses, yet not ...

Description

About Wavicle

Wavicle Data Solutions is built upon the principle that modern organizations rely on data to drive their businesses, yet not every company has the time, staff, or knowledge to leverage this valuable resource. Wavicle's driving purpose is to help these organizations craft custom data solutions focused on Big Data, Cloud and Analytics. Solutions that help companies gain deep insights from their data that position them to make today's business decisions and gives them the knowledge to predict future trends.

Role

  • At least 5 years of experience with AWS and Python programming, experience with Python frameworks (e.g. Django, Flask, Bottle)
  • Experience with AWS Cloud on data integration with Apache Spark, EMR, Glue, Kafka, Kinesis, and Lambda in S3, Redshift, RDS, MongoDB/DynamoDB ecosystems
  • Strong real-life experience in python development especially in pySpark in AWS Cloud environment.
  • Design, develop test, deploy, maintain and improve data integration pipeline.
  • Working experience on ETL pipeline implementation using AWS services such as Glue,Lambda,EMR, Athena, S3,SNS,Kinesis, Data-Pipelines, Pyspark
  • Hands-on Experience using programming language Scala, python, R, or Java
  • Develop pipeline objects using Apache Spark / Pyspark /Python or Scala
  • Design and develop data pipeline architectures using Hadoop, Spark and related AWS services.
  • Build the infrastructure required for optimal extraction, transformation and loading of data from a wide variety of sources Hadoop, Spark, AWS Lambda, etc.
  • Knowledge or experience in architectural best practices in building data lakes.
  • Load and performance test data pipelines built using the above-mentioned technologies
  • Expert-level knowledge of using SQL to write complex, highly-optimized queries across large volumes of data.
  • Strong problem solving and troubleshooting skills with the ability to exercise mature judgment.


Requirements
  • Experience with Big Data technologies and ETL tools
  • Hands on Experience with Python and Spark is must
  • Good understanding of the AWS environment
  • 2+ years of experience with UNIX/Linux including basic commands and shell scripting.
  • At this time, we will not sponsor a new applicant for employment authorization for this position

 
Apply Now

Share

Flash-bkgn
Loader2 Processing ...