Job Description

Hybrid onsite 2 days/week or 8 days/month
1 interview will be onsite (2 interview process)
Databricks is a must
AWS or Azure
Must be able to do code reviews of Python code
Create designs and validate code
"own" the architecture
As the Data Engineering Manager, you will be responsible for architecting, implementing, and optimizing end-to-end data solutions on Databricks while integrating with core AWS services. You will lead a technical team of data engineers, ensuring best practices in performance, security, and scalability. This role requires a deep, hands-on understanding of Databricks internals and a track record of delivering large-scale data platforms in a cloud environment.
Lead a team of data engineers in the architecture and maintenance of Databricks Lakehouse platform, ensuring optimal platform performance and efficient data versioning using Delta Lake
Manage and optimize Databricks infrastructure including cluster lifecycle, cost optimization, and integration with AWS services (S3, Glue, Lambda)
Design and implement scalable ETL/ELT frameworks and data pipelines using Spark (Python/Scala), incorporating streaming capabilities where needed
Drive technical excellence through advanced performance tuning of Spark jobs, cluster configurations, and I/O optimization for large-scale data processing
Implement robust security and governance frameworks using Unity Catalog, ensuring compliance with industry standards and internal policies
Lead and mentor data engineering teams, conduct code reviews, and champion Agile development practices while serving as technical liaison across departments
Establish and maintain comprehensive monitoring solutions for data pipeline reliability, including SLAs, KPIs, and alerting mechanisms
Configure and manage end-to-end CI/CD workflows using source control, automated testing, and version control
Qualifications
Your Role:
You have a Bachelor's Degree in Engineering, Computer Science or equivalent.
5+ years of hands-on experience with Databricks and Apache Spark, demonstrating expertise in building and maintaining a production-grade data pipelines
Proven experience leading and mentoring data engineering teams in complex, fast paced environments
Extensive experience with AWS cloud services (S3, EC2, Glue, EMR, Lambda, Step Functions)
Strong programming proficiency in Python (PySpark) or Scala, and advanced SQL skills for analytics and data modeling
Demonstrated expertise in infrastructure as code using Terraform or AWS CloudFormation for cloud resource management
Strong background in data warehousing concepts, dimensional modeling, and experience with RDBMS systems (e.g., Postgres, Redshift)
Proficiency with version control systems (Git) and CI/CD pipelines, including automated testing and deployment workflows
Excellent communication and stakeholder management skills, with demonstrated ability to translate complex technical concepts into business terms
Has demonstrated the use of AI in the development lifecycle
Some travel may be required to the US
Knowledge of financial industry will be preferred

Skills Required