MS – Ssr. Data Engineer B. – Job6642
Summary
We are seeking a senior-level Data Engineer to design, build, and optimize AI-enhanced data pipelines within the Microsoft Azure ecosystem. This role is central to delivering reliable, scalable data workflows that power analytics, machine learning, and business intelligence across the organization.
The position offers the opportunity to work with Azure Data Factory, Azure Synapse Analytics, Azure Databricks, and Azure Machine Learning to integrate AI-driven quality checks and predictive automation into ETL processes, improving data accessibility and trust for stakeholders.
Responsibilities
- Design, build, and maintain data pipelines using Azure Data Factory, Azure Synapse Analytics, and Azure Databricks.
- Develop and manage ETL processes to ingest and transform data from diverse sources into Azure data platforms.
- Integrate AI models and services (e.g., Azure Machine Learning, Azure Cognitive Services) into data workflows for predictive analytics and automation.
- Implement AI-driven data quality checks and anomaly detection within pipelines.
- Optimize data storage and retrieval strategies using Azure Data Lake, SQL Server, and Cosmos DB for performance and cost-efficiency.
- Monitor, troubleshoot, and tune pipeline performance; apply predictive techniques to anticipate and resolve bottlenecks.
- Collaborate with data scientists, analysts, and stakeholders to translate business requirements into pipeline designs that meet compliance standards (e.g., GDPR, HIPAA).
- Document pipeline architectures, configurations, and AI integrations to support knowledge transfer and best practices.
Requirements
- Bachelor’s degree in Computer Science, Information Technology, Engineering, or related field; Master’s preferred.
- 3-5 years of experience in data engineering or a similar role with hands-on work in the Microsoft Azure ecosystem.
- Proven experience building pipelines with Azure Data Factory, Azure Synapse Analytics, and Azure Databricks.
- Proficiency in Python and SQL for data processing and transformation.
- Hands-on experience integrating AI into data pipelines (Azure Machine Learning or Azure Cognitive Services).
- Strong knowledge of ETL processes, data modeling, and cloud-based data warehousing solutions.
- Experience with Azure Data Lake, SQL Server, and Cosmos DB for storage and retrieval optimization.
- Strong problem-solving skills and ability to work independently and collaboratively.
- Effective communication skills to present technical concepts to non-technical stakeholders.
Nice to Have
- Microsoft Azure Data Engineer Associate certification.
- Experience with DevOps practices and pipeline deployment using Azure DevOps.
- Familiarity with real-time data streaming using Azure Event Hubs or IoT Hub.
- Knowledge of cloud security and governance within Azure environments.