Data Engineer

Data & Analytics

Reference: #214650

Start Date: ASAP Duration: > 12 months

Languages: English (Full Professional), Polish (Full Professional) Seniority: Mid Level

Project Information: * Industry: Insurance and IT services * Rate: Up to 160 zł/h netto + VAT * Location: Warsaw (first 2-3 months of office visits once a week, then occasionally) * Project Language: Polish, English

Summary

The Data Engineer will be responsible for designing, building, and maintaining Data Hubs that integrate multiple data sources for efficient analytics and operational purposes, with a focus on real-time data processing.

Main Responsibilities

• Data Hub Development: Design and implement scalable Data Hubs to support enterprise-wide data needs.
• Data Pipeline Engineering: Build and optimize ETL/ELT pipelines for efficient data ingestion, transformation, and storage.
• Logical Data Modeling: Structure Data Hubs to ensure efficient access patterns and support diverse use cases.
• Real-Time Analytics: Enable real-time data ingestion and updating models.
• Data Quality & Monitoring: Develop monitoring features to ensure high data reliability.
• Performance Optimization: Optimize data processing for large-scale datasets.
• Automation & CI/CD: Implement CI/CD pipelines for automating data workflows.
• Collaboration: Align data solutions with enterprise needs through teamwork.
• Monitoring & Maintenance: Continuously improve data infrastructure reliability.
• Agile Practices: Participate in Scrum/Agile methodologies.
• Documentation: Create and maintain clear documentation for data models and pipelines.

Other Details

This position falls under the Technology Stack category, focusing on tools like Databricks, Apache Spark, and Delta Lake. Additionally, it incorporates aspects of Cloud & Data Services, utilizing Azure Data Factory, ADLS, Azure SQL, and Azure DevOps.

• Strong Python skills (or other relevant language)
• Experience with Azure Data Factory, ADLS, and Azure SQL
• Hands-on experience in building ETL/ELT pipelines
• Experience with real-time data processing
• Understanding of data preparation for AI/ML applications
• Experience in building data validation and monitoring features
• Proficiency in SQL for data transformation
• Familiarity with CI/CD and infrastructure-as-code principles
• Understanding data security and compliance best practices
• Proficient in English (B2 level minimum)

Nice to Have

• Data Governance knowledge
• Experience with containerization technologies (Docker, Kubernetes/AKS)
• Agile collaboration experience
• Ability to produce high-quality technical documentation