Senior Data Engineer

Role Overview

We are seeking an experienced Senior Data Engineer to design, build, and maintain scalable data pipelines and cloud-based data solutions. The role covers data ingestion, processing, storage, orchestration, and delivery to Analytics and BI teams within an AWS-based data ecosystem.

Responsibilities

Design and maintain batch and real-time data pipelines.

Integrate data from APIs, databases, files, and external sources using Apache Kafka and AWS services.

Develop and orchestrate workflows with Apache Airflow, including monitoring, retry mechanisms, and alerting.

Build and optimize distributed data processing solutions using Apache Spark, PySpark, Spark SQL, and AWS EMR.

Manage Data Lake and Data Warehouse environments using Amazon S3 and Amazon Redshift.

Design analytical data models, including Star and Snowflake schemas.

Prepare and optimize datasets for Analytics and Business Intelligence use cases, particularly for Qlik.

Develop high-quality solutions using Python and SQL, following best practices for Git, code reviews, testing, and documentation.

Ensure data quality, security, governance, and compliance, including AWS IAM, encryption, and access control.

Monitor production data pipelines and proactively address operational and performance issues.

Mandatory Requirements

Proven experience as a Senior Data Engineer or in a similar role.

Advanced proficiency in Python and SQL.

Hands-on experience with:

Apache Spark / PySpark

Apache Kafka

Apache Airflow

Strong AWS experience, particularly with:

Amazon S3

Amazon Redshift

AWS EMR

Experience with Data Lakes, Data Warehouses, ETL/ELT processes, and data modeling.

Strong understanding of batch and streaming architectures.

Experience with performance optimization, Git, testing practices, and production support.

Technical Stack

Python

SQL

Apache Spark / PySpark

Apache Kafka

Apache Airflow

AWS EMR

Amazon S3

Amazon Redshift

Git

Qlik

Originally posted on Himalayas