PP – Data Engineer B. – Job4232
Summary
We are seeking a highly skilled and experienced Senior Data Engineer to join our Credit Platform Data team. In this pivotal role, you will be responsible for designing, building, and maintaining robust data pipelines and ETL processes that ingest, transform, and load data from diverse sources into our data warehouse. Your work will directly impact the reliability, scalability, and efficiency of our data infrastructure, enabling data-driven decision-making across the organization. Collaborating closely with product managers, analysts, and other stakeholders, you will help translate complex data requirements into scalable and maintainable data solutions. This role offers an exciting opportunity to work with cutting-edge data engineering technologies and contribute to the growth and scalability of our credit platform.
Responsibilities
- Designing, building, and maintaining robust data pipelines and ETL processes to ingest, Transform, and load data from various sources into the data warehouse.
- Collaborating with product managers, analysts, and stakeholders to understand data requirements and develop solutions that accommodate large volumes of data.
- Ensuring the reliability, availability, and scalability of our data systems by monitoring performance and optimizing as needed.
- Implementing automated data quality checks and validation processes to ensure integrity and accuracy at scale.
- Troubleshooting data related issues promptly by identifying root causes and implementing effective solutions.
- Creating design documents and documentation for data pipelines, systems,and processes.
- Actively participating in design reviews as well as code reviews to maintain high quality standards.
- Staying current with emerging technologies in the field of data engineering to recommend improvements that support scalability and growth.
Requirements
- Bachelor’s degree in Computer Science, Engineering, or a related field.
- 3+ years of proven experience as a Data Engineer or similar role, with a strong background in database development, ETL processes, and software development.
- Proficiency in SQL and scripting languages such as Python, with experience working with relational databases.
- Proficiency in PySpark, Pandas or other data processing libraries.
- Familiarity with data warehousing concepts and tools, such as AWS Redshift, Google BigQuery, or Snowflake, and experience optimizing performance for large-scale data processing.
- Experience with data modeling, schema design, and optimization techniques for scalability.
- Strong analytical and problem-solving skills, with the ability to troubleshoot complex data issues and optimize data processing pipelines for scale.
- Experience with Automation test cases is a must too.
- Experience with Unix/Linux operating systems and shell scripting.
- Excellent communication and collaboration skills, with the ability to work effectively in a team environment.
- Self-motivated and proactive, with a passion for continuous learning and professional development.
- ETL strong experience, including a deep understanding of data extraction, transformation,and loading processes (must-have requirement).
- Proficiency in Python programming language for data processing tasks .
- Strong knowledge and experience in data modeling to design efficient and scalable database structures ..
- Proficiency in PySpark, Pandas, or other data processing libraries commonly used in Python
- Knowledge on development and engineering and software lifecycle is required