The Data Engineer (AWS Stack) is responsible for architecting and maintaining the end-to-end data lifecycle. Your primary mission is to transform raw, fragmented data into "gold-standard" datasets within Amazon Redshift to power Business Intelligence (BI) and Machine Learning (ML). You act as the bridge between raw infrastructure and data-driven decision-making.
1. Pipeline Development & ETL Orchestration PySpark Processing: Design and develop distributed processing jobs using PySpark on AWS Glue or EMR to handle large-scale data transformation. Serverless ETL: Build and manage AWS Glue jobs (Python/Spark), Crawlers, and Data Catalogs to automate schema discovery and metadata management. Workflow Automation: Orchestrate complex data workflows using AWS Step Functions or Apache Airflow (MWAA) to ensure dependencies are met and failures are handled gracefully. 2. Data Warehousing & Modeling Redshift Optimization: Design and maintain Amazon Redshift schemas (Star/Snowflake). You are responsible for performance tuning via Distribution Keys, Sort Keys, and WLM (Workload Management). Advanced SQL: Write complex, high-performance SQL for data aggregation, window functions, and views to support downstream analytics. Data Lake Integration: Implement "Lakehouse" architectures using Redshift Spectrum to query cold data directly from S3 without loading it into the warehouse. 3. Data Quality & Governance Validation Frameworks: Implement automated data quality checks (e.g., using AWS Glue Data Quality or Great Expectations) to catch anomalies before they reach production. Security: Manage data encryption (KMS), IAM roles, and VPC security groups to ensure data remains secure at rest and in transit.
Mandatory Skills
Search your career opportunity with our categories
Discover similar job openings and career opportunities that match your skills and experience. Apply Now!
