Role : Azure Data Engineer (Databricks Expert) Location : Toronto, ON Hire Type : Full-Time with Eviden
Job Description: About the Role: We are seeking a highly skilled and experienced Senior Azure Data Engineer to join team and tackle critical performance challenges within Azure data platform. You will play a pivotal role in optimizing and re-architecting key components of data infrastructure, directly impacting the performance and scalability of core data marts. This is not a maintenance role; this is a chance to make a significant impact by redesigning and implementing solutions for a high-visibility project. Responsibilities: Performance Optimization: Lead the performance tuning and optimization of Azure-based data pipelines, specifically addressing the current bottleneck where processing 1000 records takes 40 minutes. This requires a deep understanding of Azure Data Factory (ADF), Databricks (Spark), and data processing best practices. Architecture Review and Redesign: Evaluate the existing Azure data platform architecture, identify bottlenecks and design flaws, and propose and implement solutions for a more efficient and scalable system. This will involve working with Delta tables and optimizing data storage and retrieval. Databricks Expertise: Utilize your advanced PySpark skills, particularly with structured streaming, to re-engineer data transformation and processing logic within Databricks. The ideal candidate will have a proven track record of optimizing Spark jobs for performance. Data Integration: Work with Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Azure Database (DB), and file exchange processes (batch mode) to ensure seamless data flow and integration. MDM Integration (Desirable): Contribute to the ongoing development and optimization of Master Data Management (MDM) system, which follows a hub-and-spoke architecture. Experience with large-scale MDM implementations is highly desirable. Collaboration: Work closely with other engineers, business stakeholders, and vendors to understand requirements, communicate progress, and ensure successful project delivery. You will be expected to mentor and guide other team members. Qualifications: Minimum experience of implementing 2 projects on Azure data platform Minimum experience of around 8-12 years "Ninja" Level Expertise: A proven track record of success in optimizing and scaling Azure data platforms, particularly with Databricks and Spark. We are looking for an expert who can quickly diagnose and resolve complex performance issues. Deep Azure Knowledge: Extensive experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure Synapse, Event hub, and other relevant Azure services. Experience with Azure SaaS offerings is a plus. Databricks Mastery: Advanced proficiency in PySpark and structured streaming. Demonstrated ability to write and optimize complex Spark queries for performance. Data Warehousing and MDM: Solid understanding of data warehousing principles and experience with Master Data Management (MDM) implementations, preferably with a hub-and-spoke architecture. Problem-Solving Skills: Exceptional analytical and problem-solving skills, with the ability to identify root causes of performance bottlenecks and design effective solutions. Communication Skills: Excellent communication and collaboration skills, with the ability to work effectively 1 with both technical and non-technical 2 stakeholders. Desirable: Experience with AI Search and Cosmos DB. Experience with Informatica MDM in an Azure environment.
Job title: Cloud SRE Specialist Locations: Montreal, Canada onsite Type: Fulltime Job Details: Experience with Infrastructure as Code Experience with CI/CD pipelines Sound knowledge of server infrastructure and cloud computing Good knowledge of security (SAML, OAuth, OpenID, Kerberos, Policies, entitlements etc.) Experience with architecting and maintaining high availability production systems Strong development skills in Python Experience in software installation, configuration and patching Hands on experience in playbook and infrastructure automation (Ansible, Terraform) Implementing open source observability tools (Prometheus, Grafana, or Open Telemetry) Experience with Agile and DevOps methodologies Developing monitoring architecture and implementing monitoring agents, dashboards, escalations and alerts Ability to communicate technical issues and ideas to colleagues and customers with claity Experience creating technical architecture documentation
Desired Skills Knowledge of security controls for the Public cloud (encryption of data in motion/rest and key management) Hands-on experience with Azure and/or AWS design and implementation Knowledge of Linux and Windows containers Experience with Open Source Cloud and configuration management tools (Puppet, Chef) Bachelor degree in a related field
Role: Azure Data Engineer Location: Toronto, ON Fulltime with Eviden Job Description: Must Have – Azure Data Bricks, Azure Data Factory, Spark SQL with analytical knowledge. 4-5 years of development experience in data engineering skills. Strong experience in Spark. Understand complex data system by working closely with engineering and product teams. Develop scalable and maintainable applications to extract, transform, and load data in various formats to SQL Server, Hadoop Data Lake or other data storage locations.
