About the role
Role Overview
We are seeking an experienced Senior Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The role covers data ingestion, processing, storage, orchestration, and delivery to Analytics and BI teams within an AWS-based data ecosystem.
Responsibilities
Design and maintain batch and real-time data pipelines.
Integrate data from APIs, databases, files, and external sources using Apache Kafka and AWS services.
Develop and orchestrate workflows with Apache Airflow, including monitoring, retry mechanisms, and alerting.
Build and optimize distributed data processing solutions using Apache Spark, PySpark, Spark SQL, and AWS EMR.
Manage Data Lake and Data Warehouse environments using Amazon S3 and Amazon Redshift.
Design analytical data models, including Star and Snowflake schemas.
Prepare and optimize datasets for Analytics and Business Intelligence use cases, particularly for Qlik.
Develop high-quality solutions using Python and SQL, following best practices for Git, code reviews, testing, and documentation.
Ensure data quality, security, governance, and compliance, including AWS IAM, encryption, and access control.
Monitor production data pipelines and proactively address operational and performance issues.
Mandatory Requirements
Proven experience as a Senior Data Engineer or in a similar role.
Advanced proficiency in Python and SQL.
Hands-on experience with:
Apache Spark / PySpark
Apache Kafka
Apache Airflow
Strong AWS experience, particularly with:
Amazon S3
Amazon Redshift
AWS EMR
Experience with Data Lakes, Data Warehouses, ETL/ELT processes, and data modeling.
Strong understanding of batch and streaming architectures.
Experience with performance optimization, Git, testing practices, and production support.
Technical Stack
Python
SQL
Apache Spark / PySpark
Apache Kafka
Apache Airflow
AWS EMR
Amazon S3
Amazon Redshift
Git
Qlik
Originally posted on Himalayas