Job Description

Lead AWS Data Engineer with 12+ Years exp -AWS,S3,Glue,Spark,Terraform,SQL,Lambda,Python,pyspark
Toronto

Must Have Technical/Functional Skills

Client is looking for engineers with the following skills:

Strong Experience with AWS Services: S3, Lake Formation, Glue, EMR, EC2 Athena, Lambda, EventBridge, SNS, SQS

Strong Experience in SQL, Python and Spark for Data Engineering tasks

Expertise in Terraform

Experience in designing and implementing Data Pipeline on AWS using native and configurable AWS services

Excellent design and troubleshooting skills

Proficiency in Apache Spark for distributed data processing

Experience with Amazon Neptune DB for graph-based metadata management

Strong understanding of Data lake architecture, data governance and security best practices

Strengthen Entitlement capability

Launch self-serve data subscription and sharing of data products through Data Portal

Improve data discovery by displaying composite data products only, simplifying catalog organization, Low Latency Solutions

Roles & Responsibilities

Build, design and implement scalable data lake architecture using AWS S3 and LakeFormation

Build and optimize ETL Pipelines using AWS Glue, EMR and Spark

Implement Event-Driven workflows using EventBridge, SNS and SQS

Design and query datasets using Athena

Manage metadata and data lineage using Amazon Neptune DB

Expose APIs for data subscription and sharing using API Gateway and AWS Lambda

Automate infrastructure provisioning using Terraform and CloudFormation

Ensure data security and compliance by implementing robust IAM policies and access controls.

Develop and maintain a self-serve data portal using Angular and integrate it with backend services

Nice to have:

Experience with RestfulAPI

Generic Managerial Skills,

If any

Strong Communication skills Experience in Leading Teams

Key Words: AWS, S3, Glue, Spark, Terraform, SQL, Lambda