Vista previa de la oferta
Software Engineer III - Data Engineer
senior · Technology / Software Development
As a Senior Associate Data Engineer within the Corporate Technology Risk organization, you will contribute to the development and modernization of the Consumer and Community Banking Risk Feature Engineering Platform. You will be expected to apply strong software engineering and data engineering practices while contributing to the platform's strategic direction through technical execution, innovation, automation, and continuous improvement.
Job Responsibilities
- Design, develop, and support scalable feature engineering solutions on Databricks that enable risk analytics, fraud detection, machine learning, and enterprise data products.
- Build and maintain reusable batch and real-time feature pipelines, including feature onboarding, versioning, testing, monitoring, and lifecycle management within the Risk Feature Store ecosystem.
- Implement modern data engineering solutions using Databricks, Apache Spark, PySpark, Delta Lake, Lakeflow, and declarative pipeline patterns, ensuring scalability, resiliency, and maintainability.
- Drive platform modernization initiatives by migrating legacy Spark and EMR workloads to Databricks-native architectures and adopting cloud-native engineering practices.
- Apply software engineering best practices including CI/CD, automated testing, code reviews, observability, release management, and production support to deliver high-quality, reliable solutions.
- Leverage enterprise-approved AI-assisted engineering tools such as GitHub Copilot and LLM Suite to accelerate development, improve code quality, automate SDLC activities, and identify opportunities for innovation.
Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.
- Implement data quality, governance, lineage, and security controls, ensuring compliance with regulatory requirements, PCI standards, metadata management, retention policies, and audit expectations.
- Develop and support cloud-native solutions on AWS, utilizing services such as S3, Glue, Lambda, ECS/EKS, Aurora/RDS, and Infrastructure-as-Code technologies including Terraform.
- Participate in architecture reviews and technical decision-making, contributing recommendations that improve platform performance, operational stability, cost efficiency, resiliency, and long-term scalability.
- Collaborate with data scientists, model developers, business stakeholders, and engineering teams, while mentoring junior engineers and promoting reuse-first, secure, and high-performing engineering practices across the organization.
Required Qualifications, Capabilities and Skills
- Formal training or certification in software engineering, computer science, data engineering, or a related discipline with 3+ years of applied industry experience.
- Strong hands-on experience with Databricks, Apache Spark, PySpark, Delta Lake, and modern Lakehouse architectures.
- Experience building and supporting large-scale batch and streaming data pipelines.
- Proficiency in Python and SQL with a strong understanding of distributed computing principles.
Hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment (e.g., for coding, test creation, troubleshooting, or documentation) with demonstrated ability to critically evaluate, validate, and refine AI-generated outputs for correctness, performance, and security.
Understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; ability to guide peers on safe and effective usage within team practices.
- Working knowledge of Databricks Unity Catalog, Delta Live Tables, and modern declarative pipeline frameworks.
- Experience with AWS cloud services including S3, Glue, EMR, Lambda, ECS/EKS, and related technologies.
- Experience implementing data-quality, data-governance, and lineage solutions.
- Familiarity with data security controls, encryption, and PCI data handling requirements.
Preferred Qualifications, Capabilities and Skills
- Professional certifications such as Databricks Data Engineer Associate/Professional and/or AWS Solutions Architect/Developer certifications are strongly preferred.
- Experience with Feature Engineering platforms including Databricks Feature Engineering, Feature Store, Feature Views, or similar enterprise feature management technologies.
- Hands-on experience building scalable real-time and event-driven data solutions, leveraging technologies such as Kafka, streaming frameworks, and distributed services architectures.
- Proven experience modernizing data platforms, including migration of legacy Spark/EMR workloads to Databricks-native architectures and adoption of Lakehouse best practices.
- Knowledge of cloud-native data platform technologies and governance, including Terraform, Snowflake, Databricks SQL, Unity Catalog, and Infrastructure-as-Code practices.