PW – Data Engineer – Job5995
Summary
We are seeking a highly skilled and experienced Senior Core Data Engineer with a strong backend focus to join our dynamic team. This role is pivotal in leading the development, optimization, and modernization of our data infrastructure. The ideal candidate will play a critical role in building scalable, robust data pipelines and semantic layers that empower data-driven decision-making across the organization. By collaborating closely with engineering teams, this position will drive the migration of legacy systems, ensure compliance with data governance and security standards, and support the integration of AI/ML models for enhanced backend data access. This is an exciting opportunity to contribute to a forward-thinking company that leverages cutting-edge technologies to deliver impactful data solutions.
Responsibilities
- Lead the design, development, and optimization of scalable data pipelines primarily using Databricks and SQL to support business intelligence and analytics needs.
- Collaborate with cross-functional engineering teams to migrate legacy data systems to modern, cloud-based platforms, ensuring minimal disruption and maximum efficiency.
- Build and maintain semantic layers and metric views that enable consistent, scalable, and self-service reporting across multiple business units.
- Execute backlog items related to data infrastructure enhancements, bug fixes, and feature requests in an agile environment.
- Ensure strict adherence to data governance policies and data security standards across all data products and pipelines, safeguarding sensitive information and maintaining compliance.
- Support the integration and operationalization of AI/ML models by providing reliable backend data access and infrastructure support.
- Participate actively in agile ceremonies, including sprint planning, daily stand-ups, and retrospectives, fostering a collaborative and transparent team culture.
- Continuously evaluate and recommend new tools, technologies, and best practices to improve data engineering processes and infrastructure.
Requirements
Must-Have Skills
- SQL: Expert-level proficiency in SQL is essential for designing, querying, and optimizing complex data pipelines and semantic layers. The candidate must be adept at writing efficient, maintainable SQL code to handle large-scale data transformations and aggregations.
- Databricks: Extensive experience with Databricks is required to build and manage data pipelines, perform ETL/ELT processes, and leverage its collaborative environment for data engineering tasks. Familiarity with Databricks’ runtime, notebooks, and integration with cloud storage is critical.
- Python: Strong backend engineering skills with Python are necessary for scripting, automation, and developing data processing workflows. The candidate should be comfortable using Python libraries relevant to data engineering and integration tasks.
- Data Governance: Deep understanding of data governance principles, including data quality, lineage, cataloging, and compliance frameworks. The candidate must ensure that data infrastructure aligns with organizational policies and regulatory requirements.
- Data Security: Proven experience implementing data security best practices, including access controls, encryption, and secure data handling procedures to protect sensitive information across data products.
- Agile Methodologies: Familiarity with agile development practices and tools, enabling effective collaboration within cross-functional teams, iterative delivery, and continuous improvement. Experience working in agile environments such as Scrum or Kanban is expected.
Nice-to-Have Skills
- BigQuery: Experience with Google BigQuery for data warehousing and analytics is advantageous, especially for integrating and optimizing cloud-based data solutions.
- Terraform: Knowledge of Terraform for infrastructure as code (IaC) to automate provisioning and management of cloud resources, enhancing deployment consistency and scalability.
- Cloud Composer: Familiarity with Google Cloud Composer (managed Apache Airflow) for orchestrating complex data workflows and pipelines is a plus.
- AI/ML Integration: Experience supporting AI/ML model deployment and integration within data pipelines, ensuring seamless backend data access and operationalization of machine learning workflows.
- Databricks AI/BI: Understanding of Databricks’ AI and Business Intelligence capabilities to enhance data analytics and model deployment processes.