Senior data engineer developing and operating Databricks / Spark pipelines and Delta Lake lakehouse layers on Azure for the Client. Accountable for pipeline reliability, data freshness and dataset quality against agreed SLAs.
Key tasks
- Develop and operate Databricks / Spark pipelines (PySpark, SQL, Delta Live Tables or Workflows).
- Design Delta Lake / lakehouse layers (bronze–silver–gold), partitioning and Unity Catalog governance.
- Build ETL/ELT jobs and orchestration with Azure Data Factory and/or Airflow; manage dependencies and retries.
- Implement data-quality checks and validation (expectations, reconciliation, anomaly alerts).
- Tune Spark job performance and cluster cost (autoscaling, Photon, job clusters, spot).
- Manage schema evolution and change control; document lineage and transformations.
- Pipeline reliability and data freshness against agreed SLAs.
- Accuracy and completeness of curated datasets.
- Schema change management and backward compatibility for downstream consumers.
- Documentation of data lineage and transformations (Unity Catalog, data catalogue).
MUST
- Bachelor's degree in Computer Science, Engineering, Information Systems or a related field, or equivalent practical experience.
- 7+ years in data engineering, of which 3+ on Databricks / Apache Spark and 2+ on Azure data services (ADF, ADLS, Delta Lake).
- Databricks (Workflows, Delta Live Tables, Unity Catalog), Apache Spark (PySpark, Spark SQL), Delta Lake.
- Azure: Data Factory, Data Lake Storage Gen2, Key Vault, Event Hubs, Synapse or SQL DB; Azure DevOps CI/CD for notebooks and jobs.
- Python and SQL at expert level; data modelling (dimensional, data vault) and ELT design.
- Orchestration (ADF, Airflow), data-quality frameworks (Great Expectations, DLT expectations), monitoring and alerting.
- Performance and cost tuning of Spark workloads; Git-based development and testing of pipelines.
- Databricks Certified Data Engineer Professional; Azure DP-203.
- Streaming (Structured Streaming, Kafka / Event Hubs).
- dbt, Power BI semantic models, MLflow.
- Experience in financial services, sovereign wealth / investment holding or other regulated enterprise environments.
- Experience working with distributed teams (onsite UAE with nearshore India / offshore Poland squads).
Luxoft DXC
2000
Luxoft, a DXC Technology Company is a global digital strategy and software engineering firm with over 17,000 international employees within its 58 offices in 29 countries. It is headquartered in Zug, Switzerland.
In January 2019, Luxoft was acquired by U.S. company DXC Technology. Luxoft partnered with LG Electronics to create a next-generation Autonomous Mobility concept vehicle that integrates consumers' personalized digital lifestyles into a driving experience. Luxoft enabled Switzerland's first Blockchain based e-vote platform with the City of Zug and Hochschule Luzern's Blockchain Lab.
Luxoft, a DXC Technology Company is a world-renowned company. It has been present on the Polish market for over 13 years. We have offices in Krakow, Warsaw, Wroclaw, and Gdansk. We employ over 2,000 professional experts carrying out projects for over 100 clients from the financial, automotive, medical, tourist industries, etc. We work for many international clients, including the USA, Great Britain, and Switzerland.
So far, Luxoft Poland has made a name for itself as a company that offers work on innovative projects, we offer various experiences in the field of IT, opportunities for rapid development, an extensive training program, and attractive benefits for employees.
At present, 62% of Luxoft Poland employees come from Poland, and 38% from around 50 countries, including Ukraine, Brazil, India, Turkey, Spain, Portugal, Italy, Romania, USA, etc.
At Luxoft, a DXC Technology Company almost 80 percent of employees are experts with the "Senior" experience level, with at least five years of experience. We care about our employees, so every day we try to provide them with the best possible conditions for work and development.
Technology is our passion! We focus on top engineering talent means that you will be working with the best industry professionals from around the world. Because of that, Luxoft is a global family with an epic atmosphere – we love what we do!