Senior Data Engineer Lead

Vista previa de la oferta

Senior Data Engineer Lead

senior · Technology / Software Development

ID de la solicitud: 271471

Programa de Referido de Empleados – Probable Pago: $400.000,00

Estamos comprometidos a invertir en nuestros colaboradores y ayudarles a continuar su carrera profesional en ScotiaTech.

 

Purpose 

The Senior Data Engineer (Level 8) designs, builds, and operates scalable data products and pipelines on Microsoft Azure and Azure Databricks, with a strategic focus on International Banking initiatives. This role is pivotal in driving data-driven decision-making by delivering robust, efficient, and secure lakehouse-based data solutions. Reporting to the Director, Data Engineering, the incumbent is the hands-on technical authority for the solutions in scope while leading a small team of Data Engineers: owning design and implementation end-to-end, setting engineering standards, developing and mentoring the team, and ensuring data solutions comply with enterprise architecture, security, and regulatory requirements across International Banking markets. 

Accountabilities  

• Design, build, test, and optimize data pipelines and lakehouse solutions on Azure and Azure Databricks (Delta Lake, medallion architecture) to support scalable data products for International Banking.  

• Act as the hands-on technical lead for assigned initiatives: own the end-to-end solution design, produce the technical documentation, and deliver production-grade code.  

• Lead, mentor, and develop a team of Data Engineers, including work assignment, coaching, performance feedback, and participation in hiring; lead code and design reviews and promote a culture of engineering excellence, collaboration, and continuous improvement.  

• Provide effort estimates, technical risks, and dependencies to support delivery planning, allocate work across the team, and keep the Director informed of progress and blockers.  

• Collaborate with cross-functional partners (Data Scientists, Product Owners, Architecture, Platform Engineering, and Software Engineering) to translate business requirements into data products aligned with international business goals.  

• Apply and help enforce best practices for data governance, security, and compliance within Azure environments, including Unity Catalog, role-based access control, data lineage, encryption, and key management.  

• Monitor and maintain data pipelines to ensure high availability, data quality, cost efficiency (FinOps), and efficient processing across international markets.  

• Champion the adoption of modern data engineering tools and methodologies, including ETL/ELT with Azure Data Factory and Databricks Workflows, dimensional and Data Vault modelling, DevOps/DataOps practices, and orchestration at scale.  

• Troubleshoot and resolve complex data pipeline and platform incidents, performing root-cause analysis and implementing permanent fixes to meet agreed service levels.  

• Support the migration and rationalization of legacy workloads (including existing Google Cloud Platform estates) onto the Azure Databricks target platform, minimizing business and regulatory risk.  

• Evaluate and pilot new Azure and Databricks services and third-party tools, recommending adoption where they enhance data capabilities and support international expansion. 

Education / Experience / Other Information

Education 

• A Master’s degree is considered an asset. 

Experience 

• 5+ years of progressive hands-on experience in data engineering, including at least 3 years delivering production data solutions on Microsoft Azure and/or Azure Databricks. 

• Demonstrated experience as a technical lead on complex data initiatives, including leading design and code reviews and directly supervising, coaching, and developing data engineers. 

• Expertise in designing and implementing data pipelines using Azure Databricks (Spark, Delta Lake, Unity Catalog, Workflows), Azure Data Factory, Azure Data Lake Storage Gen2, Event Hubs, and Azure Synapse or equivalent serving layers. 

• Advanced proficiency in Python, PySpark, and SQL (Scala or Java an asset) for large-scale batch and streaming data processing, including performance tuning and cost optimization of Spark workloads. 

• Hands-on experience with data modelling, schema design, lakehouse/medallion patterns, and data warehousing concepts. 

• Strong working knowledge of CI/CD pipelines (Azure DevOps or GitHub Actions), version control (Git), Infrastructure as Code (Terraform or Bicep), automated testing, and Databricks Asset Bundles or equivalent deployment tooling. 

• Knowledge of data governance, security, and compliance standards in cloud environments (e.g., GDPR, local data residency and privacy regulations), including Microsoft Entra ID, RBAC, and cloud key management. 

• Experience migrating workloads from another public cloud or on-premises platform (e.g., GCP BigQuery/Dataflow to Azure Databricks) is a strong asset; Azure or Databricks certifications (e.g., Databricks Certified Data Engineer Professional, Azure DP-203) are an asset. 

• Excellent problem-solving, influencing, and communication skills, with the ability to present complex technical concepts to non-technical stakeholders; Spanish language ability is an asset given the International Banking footprint. 

Soft Skills: 

• Strong leadership, stakeholder communication, and problem-solving abilities. Candidates must demonstrate adaptability and a growth mindset. 

• Results-oriented mindset with a strong focus on execution, quality, and delivery. 

• Strong communication skills (written & verbal), interacting with Product, Design and Back-end Teams.  

• Strong sense of ownership, accountability, and commitment to achieving business outcomes. 

• Great communication skills.   

• Proactive approach to identifying risks, removing obstacles, and driving continuous improvement. 

• Ability to manage multiple priorities while maintaining a sense of urgency and attention to detail. 

• Ability to work effectively in collaborative, distributed, and multicultural teams. 

• Strong verbal and written English communication skills (B2+ or higher). 

Working Conditions 

• Work in a standard office-based environment, with standard hours.    

​ Ubicación(s):  Colombia : Bogota : Bogota ScotiaTech es una unidad de negocio de ScotiaGBS, un grupo de empresas de Scotiabank, ubicado en Bogotá, Colombia. ScotiaTech fue creado para apoyar diversos sistemas y procesos tecnológicos del Banco. Ofrecemos un entorno de trabajo inclusivo y positivo, además de ventajas competitivas. En ScotiaTech, valoramos las habilidades y experiencias únicas que cada persona aporta y nos hemos comprometido a crear y mantener un entorno inclusivo y accesible para todos. Los candidatos deben postularse directamente en línea si desean ser tomados en cuenta para este puesto. Agradecemos a todos los candidatos por su interés en esta oportunidad profesional en ScotiaTech; sin embargo, solo contactaremos a quienes hayan sido seleccionados para una entrevista. ​