This is a Data Engineer position focused on building and maintaining big data pipelines using the Hadoop ecosystem (HDFS, YARN, Hive) and Spark/PySpark. The role sits at the intersection of data infrastructure and analytics — the engineer will design ETL/ELT pipelines that move and transform large volumes of structured and unstructured data, while ensuring performance, scalability, and data quality across distributed systems.

• Design, build, and maintain scalable ETL/ELT data pipelines using Hadoop, Hive, HDFS, and Spark/PySpark • Develop and optimize complex SQL queries and Hive scripts for large-scale data processing • Write clean, efficient, and reusable Python code for data transformation and automation • Work with structured and unstructured data across distributed storage systems (HDFS) • Optimize Spark/PySpark jobs for performance, scalability, and resource efficiency • Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements • Ensure data quality, consistency, and integrity across pipelines • Troubleshoot and resolve issues related to data pipeline failures, performance bottlenecks, and cluster resource management • Participate in code reviews and follow best practices for data engineering and version control • Document technical designs, data flows, and pipeline architecture

• 6 –10 years of hands-on experience in Data Engineering • Strong working knowledge of Hadoop ecosystem (HDFS, YARN, MapReduce concepts) • Proficiency in Hive for data warehousing and query optimization • Solid experience with Spark/PySpark for distributed data processing • Strong programming skills in Python • Advanced SQL skills — query optimization, joins, window functions, performance tuning