MS – Ssr. Data Engineer – Job0802
Summary
We are seeking a highly skilled Senior Data Engineer with deep expertise in Microsoft Azure technologies to join our dynamic data engineering team. This role is critical in designing, building, and maintaining scalable, efficient, and robust data pipelines that empower our organization to leverage data as a strategic asset. The ideal candidate will play a pivotal role in integrating advanced AI models for predictive analytics and automation, optimizing data storage solutions, and ensuring data quality and compliance. This position offers an exciting opportunity to work with cutting-edge Azure tools and collaborate closely with cross-functional teams to deliver impactful data-driven solutions.
Responsibilities
- Design, develop, and maintain scalable data pipelines using Azure Data Factory, Azure Synapse Analytics, and Azure Databricks to process and transform large volumes of structured and unstructured data.
- Architect and optimize data storage solutions leveraging Azure Data Lake and Azure Cosmos DB to ensure efficient, secure, and cost-effective data management.
- Develop and manage complex ETL (Extract, Transform, Load) processes that support business intelligence, analytics, and reporting needs.
- Integrate AI and machine learning models into data pipelines using Azure Machine Learning and Azure Cognitive Services to enable predictive analytics, automation, and intelligent data quality checks.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements, translate them into technical solutions, and ensure alignment with business goals and compliance standards.
- Implement AI-driven data quality frameworks to monitor, validate, and improve data accuracy, consistency, and reliability.
- Optimize performance and scalability of data workflows and storage, proactively identifying bottlenecks and implementing improvements.
- Ensure data governance, security, and compliance best practices are embedded in all data engineering activities.
- Mentor junior engineers and contribute to knowledge sharing and best practices within the team.
- Participate in DevOps practices for continuous integration and deployment of data pipelines using Azure DevOps or equivalent tools.
- Stay current with emerging Azure technologies, AI advancements, and industry trends to continuously enhance the data engineering capabilities of the organization.
Requirements
Must-Have Skills
- Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field; Master’s preferred.
- 3-5 years of experience in data engineering or a similar role, with a focus on
- Microsoft Azure ecosystem tools (Data Factory, Synapse, Databricks).
- Proficiency in Python, SQL, and Microsoft-specific data tools (e.g., Azure Data Lake, Azure Stream Analytics).
- Hands-on experience with AI integration in data pipelines, such as using Azure Cognitive Services or ML models for data processing.
- Strong knowledge of data modeling, ETL processes, and cloud-based data warehousing solutions.
- Demonstrated ability to design scalable data architectures and optimize for cost and performance.
- Excellent problem-solving skills and ability to work independently as well as collaboratively.
- Effective communication skills to convey technical concepts to non-technical stakeholders.
Nice-to-Have Skills
- Certification in Microsoft Azure Data Engineer Associate or equivalent.
- Experience with DevOps practices for pipeline deployment (e.g., Azure DevOps, CI/CD pipelines).
- Familiarity with real-time data streaming using Azure Event Hubs or IoT Hub.
- Knowledge of cloud security and governance within Azure environments.