Vista previa de la oferta
Manager | GCP Data Engineer | Ahmedabad | Engineering | Data Modernization & Migration
manager · Technology / Software Development
Role Summary
We are looking for a GCP Data Engineering Manager / AD with 8 to 15+ years of experience to lead the delivery and optimization of cloud-based data platforms. The role combines strong expertise in GCP data services with hands-on performance engineering for large-scale Spark and Dataproc workloads.
Key Responsibilities
GCP Data Engineering and Platform Delivery
- Design and deliver data solutions using Cloud Run, Cloud Run Functions with Python, Cloud SQL PostgreSQL, Pub/Sub, GCS and BigQuery.
- Build and manage orchestration workflows using Cloud Composer / Managed Airflow.
- Drive Airflow and Composer optimization, including FinOps, capacity planning and cost governance.
- Support Databricks-based data processing and integration.
- Design and integrate REST APIs across cloud applications and data platforms.
- Provide technical leadership, architecture guidance and delivery oversight to engineering teams.
Performance Engineering and Dataproc Optimization
- Execute workload test waves and enhance Python-based execution templates.
- Troubleshoot Spark, YARN and Dataproc failures using Spark UI and platform diagnostics.
- Analyze approximately 60 workload profiles and capture cost-performance benchmarks.
- Define N1/N2 mapping strategies and translate custom machine requirements into standard GCP machine types.
- Recommend VM sizing, schema design and performance improvements for Big Data workloads.
Required Skills
- 8 to 15 years in Data Engineering, with strong experience on GCP.
- Python, Spark, Dataproc, GCS, BigQuery and Cloud Data Architecture.
- Cloud Run, Cloud Functions, Cloud SQL PostgreSQL and Pub/Sub.
- Cloud Composer / Airflow administration, optimization and FinOps.
- Spark UI debugging, YARN diagnostics and workload benchmarking.
- Strong stakeholder management, team leadership and solution design capabilities.
B.E/B.tech