Senior Data Engineer (Python + Snowflake + Synapse Analytics)
What you will be doing
• Design, develop, and maintain robust data pipelines using Python to extract data from Snowflake and Azure Synapse Analytics and transform it for target systems.
• Implement efficient ETL processes, ensuring accurate, reliable, and scalable data movement across platforms.
• Collaborate with stakeholders to understand source systems, target systems, and business requirements, translating them into effective data pipeline solutions.
• Write clean, optimized, and scalable code capable of processing large data volumes while maintaining performance and reliability.
• Monitor, troubleshoot, and optimize data pipeline performance, resolving bottlenecks, failures, and data quality issues.
• Define and implement data transformation logic, including cleansing, filtering, aggregation, normalization, and standardization of data.
• Develop data mapping and schema conversion processes to ensure consistency and compatibility between systems.
• Establish and maintain connectivity with Snowflake and Azure Synapse Analytics through APIs, database connectors, and other integration methods.
• Integrate and synchronize data from multiple sources while maintaining consistency, accuracy, and data integrity.
• Partner with IT and platform teams to implement secure data transfer mechanisms aligned with governance and compliance requirements.
• Develop error handling and exception management processes to ensure resilience and reliability across data integrations.
• Document pipeline architecture, transformation rules, source specifications, and target system requirements.
• Collaborate with cross-functional teams including data analysts, data scientists, and business stakeholders to support data-driven initiatives.
• Participate in planning discussions and technical reviews to align data engineering activities with business goals.
• Stay current with emerging technologies, tools, and best practices in data engineering and recommend process improvements where appropriate.
• Implement efficient ETL processes, ensuring accurate, reliable, and scalable data movement across platforms.
• Collaborate with stakeholders to understand source systems, target systems, and business requirements, translating them into effective data pipeline solutions.
• Write clean, optimized, and scalable code capable of processing large data volumes while maintaining performance and reliability.
• Monitor, troubleshoot, and optimize data pipeline performance, resolving bottlenecks, failures, and data quality issues.
• Define and implement data transformation logic, including cleansing, filtering, aggregation, normalization, and standardization of data.
• Develop data mapping and schema conversion processes to ensure consistency and compatibility between systems.
• Establish and maintain connectivity with Snowflake and Azure Synapse Analytics through APIs, database connectors, and other integration methods.
• Integrate and synchronize data from multiple sources while maintaining consistency, accuracy, and data integrity.
• Partner with IT and platform teams to implement secure data transfer mechanisms aligned with governance and compliance requirements.
• Develop error handling and exception management processes to ensure resilience and reliability across data integrations.
• Document pipeline architecture, transformation rules, source specifications, and target system requirements.
• Collaborate with cross-functional teams including data analysts, data scientists, and business stakeholders to support data-driven initiatives.
• Participate in planning discussions and technical reviews to align data engineering activities with business goals.
• Stay current with emerging technologies, tools, and best practices in data engineering and recommend process improvements where appropriate.