Job Description

General Responsibilities

• Design, develop and implement ingestion framework from Oracle source to Azure Data Lake - initial load and incremental ETL. Used tools are:

• Azure Data Factory (expert knowledge) to maintain pipeline from Oracle to Azure Data Lake
• Azure Synapse to build stored procedures and read data from data lake
• Review the requirements, database tables, and database relationships - Identify gaps and inefficiencies in current production reporting environment and provide recommendations address them in the new platform.
• Continue to evolve and design ingesting framework and CDC

• Prepare design artifacts
• Analysis of data - physical model mapping from data source to reporting destination.

• Understand the requirements. Recommend changes to the Physical model.
• Develop the scripts of physical model, and create DB.
• Access Oracle DB environments, use SSIS, SQL Server and other development tools for developing solution.
• Proactively communicate with business on any changes required to conceptual, logical and Physical models, communicate and review dependencies and risks.
• Development of ETL strategy and solution based on different set of modules

• Understand the Tables and Relationships.
• Create low level design documents and unit test cases.
• Create the workflows of package design
• Development and testing of data with Incremental and Full Load.

• \xef\xbb\xbfDevelop high quality ETL mappings/scripts/jobs
• ETL data from Applications to Data Warehouse
• ETL data from Data Warehouse to Data Mart
• Perform unit tests.
• Performance Review, data Consistency checks

• Troubleshoot performance issues, ETL Load issues, log activity for each Individual package and transformation.
• Review Performance of ETL Overall.
• End to end Integrated testing for Full Load and Incremental Load
• Plan for Go Live, Production Deployment.

• Create production deployment steps.
• Configure parameters, scripts for go live. Test and review the instructions.
• Create release documents and help build and deploy code across servers.
• Go Live Support and Review after Go Live.

• Review existing ETL process, tools and provide recommendation on improving performance and reduce ETL timelines.
• Review Infrastructure and any pain points for overall process improvement
• Knowledge Transfer to Ministry staff, development of documentation on the work completed.

• Document share and work on the ETL end to end working knowledge, Troubleshooting steps, configuration and scripts review.
• Transfer documents, scripts and review of documents

• 7+ years in ETL tools such as Microsoft SSIS, stored procedures (Must Have)
• 2+ Azure Data Lake and Data Warehouse, and building Azure Data Factory pipelines (Must Have)
• 2+ years Python (nice to have)
• Databricks
• Synapse (nice to have)
• SQL Server
• Oracle
• Ability to present technical requirements to the business

emergiTEL