Hadoop PySpark Developer Job at Purple Drive, Pittsburgh, PA

TzRRSGRWanEybmpnWDY0Z1FrN0xEWDlP
  • Purple Drive
  • Pittsburgh, PA

Job Description

Job Title: Hadoop PySpark Developer
Location: Pittsburgh, PA (Onsite)
Experience Level: 8+ years preferred

We're hiring a Hadoop PySpark Developer to work onsite in Pittsburgh. You'll be using tools like Spark, Python, Hive, and Kafka to build and manage data pipelines on the Cloudera Data Platform. The role involves working with large datasets, building ETL solutions, and making sure everything runs smoothly and efficiently. You'll also work with other teams to understand data needs and help deliver reliable solutions.

Roles & Responsibilities

  • Build ETL pipelines using Spark, Python, Hive, and Kafka
  • Understand business requirements and turn them into technical tasks
  • Work with data analysts and business users to gather data needs
  • Create design documents and source-to-target mappings
  • Make sure data pipelines run on time and handle large volumes
  • Improve performance of Spark jobs and existing pipelines
  • Set up data governance and ingestion frameworks
  • Help with production deployments and fix issues when needed
  • Support data migration and rebuilding history data
  • Review code and test cases, and help solve technical problems
  • Work independently and coordinate with global teams
  • Bring in ideas and best practices to improve solutions

Key Skills

  • Apache Spark (PySpark)
  • Python
  • Hive
  • Kafka
  • Cloudera Data Platform
  • Linux/Unix
  • Impala
  • Big Data tools
  • Cloud technologies
  • Performance tuning
  • Data governance
  • Good communication and teamwork

Job Tags

Full time,

Similar Jobs

Loma Linda University Faculty Medical Group (LLUFMG)

Intensivist Job at Loma Linda University Faculty Medical Group (LLUFMG)

Join to apply for the Intensivist role at Loma Linda University Faculty Medical Group (LLUFMG) .6 months ago Be among the first 25 applicants.Loma Linda University Faculty Medical Group, Division of Critical Care, is seeking a board eligible or board certified Intensivist... 

Alvarez & Marsal

Director, Global Transaction Tax (Federal) (Washington) Job at Alvarez & Marsal

 ...Director, Global Transaction Tax (Federal) Join to apply for the Director, Global Transaction Tax (Federal) role at Alvarez & Marsal...  ...Review and comment on tax aspects of purchase agreements; Coordinate with state and local and international tax team members, as well... 

Agape Care Group

Hospice Chaplain Job at Agape Care Group

 ...meaningful impact and help create positive experiences for patients and families? We are looking for spiritual care coordinators or chaplains to join our team who are committed to guiding each patient with dignity and compassion. As a spiritual care coordinator, you will... 

S Universe

Salesforce Admin Consultant Job at S Universe

 ...Were seeking a skilled Salesforce Admin Consultant to support the design, implementation, and optimization of Salesforce solutions for our business needs. You will serve as the go-to expert for translating requirements into efficient system configurations, helping teams... 

Providence

RN - ER / Acute Care *Per Diem* - Valdez Job at Providence

 ...You will thrive in our culture of patient-focused, whole-person care built on understanding, commitment, and mutual respect. Your voice...  ...world-class health with human connection and make a difference every day through your extraordinary care. \n \n \n \n\n \n \n \n...