| Job Description: |
Job Description:
We are looking for an experienced Apache Spark Developer with strong expertise in large-scale data processing and performance optimization. The role involves designing, building, and optimizing scalable ETL pipelines using Spark in cloud environments.
Key Skills
Strong hands-on expertise in Apache Spark 3.5.x or later
Solid experience with Spark DataFrames, Datasets, and RDDs
Strong proficiency in Spark SQL
Experience working in cloud environments (AWS and/or Azure)
Hands-on experience with cloud storage solutions such as Amazon S3 and Azure Data Lake Storage (ADLS)
Strong understanding of ETL pipeline design and data modeling
Proven experience in Spark job performance tuning and query optimization
Preferred / Good to Have
Experience with Snowflake for data warehousing and analytics
Exposure to real-time/streaming data processing
Familiarity with distributed systems and large-scale data architectures
Scala Programming Language
Roles & Responsibilities:
Design, develop, and maintain large-scale batch and real-time data processing solutions using Apache Spark (v3.5.x or later)
Work extensively with Spark DataFrames, Datasets, and RDDs for high-volume data processing
Develop and optimize queries using Spark SQL
Build and manage ETL pipelines to support analytics and downstream data consumption
Perform Spark job tuning and performance optimization to improve efficiency and scalability
Collaborate with data engineers, architects, and analytics teams to deliver end-to-end data solutions
Ensure data quality, reliability, and adherence to best practices across pipelines
Role Descriptions: Pyspark and Scala Developer
Essential Skills: Pyspark and Scala Developer
Desirable Skills:
Keyword:
Skills: Informatica Powercentre~Digital : BigData and Hadoop Ecosystem - MapR~Digital : NOSQL~Digital : Scala~Digital : PySpark
Experience Required: 8-10 |
