Job Overview
We are looking for talented and motivated professionals to join our Technology & Transformation team as Consultant /Senior Consultant - Databricks. The ideal candidate will have strong experience in data engineering, cloud technologies, and Databricks, with the ability to design, develop, and implement scalable data solutions that support business transformation and data-driven decision-making.
Key Responsibilities:
- Design, develop, and implement scalable data pipelines and data engineering solutions using Databricks, Apache Spark, Python, and SQL
- Build and optimise data processing workflows using PySpark, Spark SQL, and Delta Lake
- Develop and maintain modern Lakehouse solutions using architectural patterns such as the Medallion Architecture
- Implement data ingestion, transformation, and integration solutions across multiple data sources
- Work with cloud platforms such as Microsoft Azure, AWS, or Google Cloud to develop and deploy data solutions
- Support the migration and modernisation of legacy data platforms to modern cloud-based and Lakehouse architectures
- Optimise data pipelines and queries to improve performance, scalability, and cost efficiency
- Apply data engineering best practices to ensure data quality, reliability, security, and governance
- Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions
- Participate in technical discussions, solution design, code reviews, and troubleshooting activities
- Lead, mentor, and support junior data engineers, contributing to capability development and team growth within the practice
- Develop and maintain CI/CD pipelines and DataOps practices to support automated, reliable, and consistent deployments
- Participate in client workshops, requirements-gathering sessions, and solution design activities
- Deliver data engineering and modernisation solutions for clients across multiple industries, working within diverse business and technology environments
Requirements:
- Bachelor’s or Master’s degree in Computer Science, Data & AI, Software Engineering, Data Engineering, or a related field
- Relevant Databricks certifications, including Databricks Certified Data Engineer Associate, will be considered an added advantage
- Consultant: 4 - 6 years of relevant experience in data engineering and related technologies
- Senior Consultant: 6 - 8 years of relevant experience in data engineering and related technologies
- Hands-on experience with Databricks and Apache Spark
- Strong programming skills in Python and SQL, including PySpark and Spark SQL
- Experience with Delta Lake, data pipelines, ETL/ELT processes, and modern data architectures
- Experience working with at least one major cloud platform: Microsoft Azure, AWS, or Google Cloud
- Hands-on experience in dimensional modelling, data warehousing, and designing scalable data models for analytics and reporting solutions
- Familiarity with data quality, performance optimisation, and data governance practices
- Strong analytical, problem-solving, communication, and stakeholder management skills
- Ability to work collaboratively in a team environment and manage multiple priorities effectively
- Experience leading technical workstreams, managing delivery teams, and driving end-to-end solution delivery
- Ability to engage with stakeholders, translate business requirements into technical designs, and provide technical guidance to junior team members
- Hands-on experience with Databricks Unity Catalog, Databricks Workflows/Jobs, and Git-based development practices, including Azure DevOps or similar CI/CD tools