Data Engineer (Python/Spark/PySpark)

Company: Nielsen Holdings
Location: Chicago, Illinois, United States
Type: Full-time
Posted: 03.SEP.2021
< >

Summary

Data Engineer (Python/Spark/PySpark) - 86561 Technology and Engineering - USA Chicago, IllinoisIf you are a DevOps and/or Cloud-based system...

Description

Data Engineer (Python/Spark/PySpark) - 86561 Technology and Engineering - USA Chicago, IllinoisIf you are a DevOps and/or Cloud-based systems enthusiast who likes to work on cutting-edge technologies in an environment that gives you the freedom to explore, play, and learn, then Nielsen is the right place for you. Join a close-knit, passionate team that collaborates in an agile environment to develop great technology.In this role, you will be working with devops engineers, data engineers, and data scientists to build/improve the data platform on AWS to ingest data from both internal and external sources, perform ETL and most importantly enable the process for the entire company through the Media Data Lake Ecosystem.The role requires you to have experience working with large datasets with complex schemas, a can-do approach towards automation, treat data with due care and compliance.RESPONSIBILITIES: Develop and enhance data-processing, orchestration, monitoring, and more by leveraging popular open-source software, AWS, and GitLab automation. Collaborate with product and technology teams to design and validate the capabilities of the data platform Identify, design, and implement process improvements: automating manual processes, optimizing for usability, re-designing for greater scalability Provide technical support and usage guidance to the users of our platform's services. Drive the creation and refinement of metrics, monitoring, and alerting mechanisms to give us the visibility we need into our production services. Design, implement and simplify our system for 24x7 availability QUALIFICATIONS: Experience building and optimizing data pipelines in a distributed environment Experience supporting and working with cross-functional teams Proficiency working in Linux environment 4+ years of advanced working knowledge of SQL, Python, and Spark 2+ years of experience with using a broad range of AWS technologies - S3, Lambda, Glue, Athena, IAM, SQS, CloudWatch, CloudFormation Experience using tools such as: Git/Bitbucket, Jenkins/CodeBuild, CodePipeline Experience with platform monitoring and alerts tools Desired Skills: Experience with GCP / Azure Experience with docker and kubernetes. Experience with Presto, Airflow, Prometheus, ELK, or similar technologies. History of working with external data providers. History of open source contributions. ABOUT NIELSEN As the arbiter of truth, Nielsen Global Media fuels the media industry with unbiased, reliable data about what people watch and listen to. To discover what's true, we measure across all channels and platforms-from podcasts to streaming TV to social media. And when companies and advertisers are armed with the truth, they have a deeper understanding of their audiences and can accelerate growth. Do you want to move the industry forward with Nielsen? Our people are the driving force. Your thoughts, ideas and expertise can propel us forward. Whether you have fresh thinking around maximizing a new technology or you see a gap in the market, we are here to listen and take action. Our team is made strong by a diversity of thoughts, experiences, skills, and backgrounds. You'll enjoy working with smart, fun, curious colleagues, who are passionate about their work. Come be part of a team that motivates you to do your best work! Nielsen is committed to hiring and retaining a diverse workforce. We are proud to be an Equal Opportunity/Affirmative Action-Employer, making decisions without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability status, age, marital status, protected veteran status or any other protected class. Job Type: Regular Primary Location: Chicago , Illinois Secondary Locations: IL - Evanston, IL - Schaumburg, , Travel: No

 
Apply Now

Share

Flash-bkgn
Loader2 Processing ...