Job Description

Title: ETL Developer/Specialized IT Consultant - Senior
Location: 5775 YONGE STREET FLOOR 4 TORONTO ONTARIO M2M 4J1
Duration: 248 days
Skills
Experience and Skill Set Requirements
Technical Skills - 70 points

• Experience in developing and managing ETL pipelines, jobs, and workflows in Databricks.
• Deep understanding of Delta Lake for building data lakes and managing ACID transactions, schema evolution, and data versioning.
• Experience automating ETL pipelines using Delta Live Tables, including handling Change Data Capture (CDC) for incremental data loads.
• Proficient in structuring data pipelines with the Medallion Architecture to scale data pipelines and ensure data quality.
• Hands-on experience developing streaming tables in Databricks using Structured Streaming and readStream to handle real-time data.
• Expertise in integrating CDC tools like GoldenGate or Debezium for processing incremental updates and managing real-time data ingestion.
• Experience using Unity Catalog to manage data governance, access control, and ensure compliance.
• Skilled in managing clusters, jobs, autoscaling, monitoring, and performance optimization in Databricks environments.
• Knowledge of using Databricks Autoloader for efficient batch and real-time data ingestion.
• Experience with data governance best practices, including implementing security policies, access control, and auditing with Unity Catalog.
• Proficient in creating and managing Databricks Workflows to orchestrate job dependencies and schedule tasks.
• Strong knowledge of Python, PySpark, and SQL for data manipulation and transformation.
• Experience integrating Databricks with cloud storage solutions such as Azure Blob Storage, AWS S3, or Google Cloud Storage.
• Familiarity with external orchestration tools like Azure Data Factory
• Implementing logical and physical data models
• Knowledge of FHIR is an asset

Design Documentation and Analysis Skills - 20 points

• Demonstrated experience in creating design documentation such as:
• Schema definitions
• Error handling and logging
• ETL Process Documentation
• Job Scheduling and Dependency Management
• Data Quality and Validation Checks
• Performance Optimization and Scalability Plans
• Troubleshooting Guides
• Data Lineage
• Security and Access Control Policies applied within ETL
• Experience in Fit-Gap analysis, system use case reviews, requirements reviews, coding exercises and reviews.
• Participate in defect fixing, testing support and development activities for ETL
• Analyze and document solution complexity and interdependencies including providing support for data validation.
• Strong analytical skills for troubleshooting, problem-solving, and ensuring data quality.

Communication and Leadership Skills - 10 points

• Ability to collaborate effectively with cross-functional teams and communicate complex technical concepts to non-technical stakeholders.
• Strong problem-solving skills and experience working in an Agile or Scrum environment.
• Ability to provide technical guidance and support to other team members on Databricks best practices.
• Must have previous work experience in conducting Knowledge Transfer sessions, ensuring the resources will receive the required knowledge to support the system.
• Must develop documentation and materials as part of a review and knowledge transfer to other members.

Must Have Skills

• 7+ years using ETL tools such as Microsoft SSIS, stored procedures, T-SQL
• 2+ Delta Lake, Databricks and Azure Databricks pipelines
• Strong knowledge of Delta Lake for data management and optimization.
• Familiarity with Databricks Workflows for scheduling and orchestrating tasks.
• 2+ years Python and PySpark
• Solid understanding of the Medallion Architecture (Bronze, Silver, Gold) and experience implementing it in production environments.
• Hands-on experience with CDC tools (e.g., GoldenGate) for managing real-time data.
• SQL Server, Oracle

Skills Required