Apply Now

Job Summary:
We are seeking a skilled Data Engineer to design, build, and maintain high-quality, scalable data pipelines with a strong focus on Silver and Gold layer transformations, data quality automation, and CI/CD-driven deployments. The ideal candidate will have hands-on experience with Python, PySpark, Git-based CI/CD pipelines, ARM templates, and data quality libraries such as Great Expectations.

Key Responsibilities:
       Design and implement Silver and Gold layer data models following medallion architecture best practices.
       Perform data cleansing, standardization, enrichment, and aggregation to support analytics and reporting.
       Build optimized PySpark-based transformations for large-scale data processing.
       Ensure data consistency, performance, and scalability across datasets.
       Build and maintain CI/CD pipelines using Git-based workflows (Azure DevOps / GitHub).
       Use ARM templates (or IaC equivalents) for automated infrastructure provisioning.
       Enable automated deployment of data pipelines, notebooks, and configurations.
       Follow DevOps best practices for version control, branching, and release management.
       Develop reusable Python-based automation scripts for pipeline orchestration, validation, and monitoring.
       Create modular, maintainable, and testable Python code.
       Support automation of metadata, logging, alerting, and operational tasks.
       Implement data quality libraries such as Great Expectations.
       Define and automate data quality rules (completeness, accuracy, freshness, consistency).
       Integrate data quality checks into data pipelines and CI/CD workflows.
       Monitor, log, and troubleshoot data quality issues proactively.
       Work closely with data architects, analysts, QA, and business stakeholders.
       Translate business and analytical requirements into robust data engineering solutions.
       Participate in Agile ceremonies and support sprint-based delivery.
       Provide production support, performance tuning, and troubleshooting

Required Skills and Qualifications:
5–10+ years of experience in data engineering or analytics engineering.
Strong hands-on experience with Silver & Gold layer development, Python (automation and data processing) and PySpark. Also with Great Expectations (data quality framework) and Experience with CI/CD pipelines using Git-based tools. Hands-on experience with ARM templates or infrastructure-as-code concepts. Strong understanding of data modeling and medallion architecture. Experience working with large datasets in distributed environments.

Good to Have:
Microsoft Certified: Azure Data Engineer Associate or Azure Enterprise Data Analyst Associate.
Wast management or oil and gas domain knowledge