The project focuses on building a modern cloud-based data processing platform, enabling efficient management and real-time analysis of large datasets. Key aspects include implementing ETL/ELT pipelines, optimizing data processing, and ensuring compliance with industry regulations. The solution leverages Azure, Apache Spark, and Data Lakehouse architecture to support strategic business decisions.
We're looking for a Big Data Lead Engineer to:
- Engineer reliable data pipelines for sourcing, processing, distributing, and storing data in different ways, using cloud (Azure) data platform infrastructure effectively.
- Transform data into valuable insights that inform business decisions, making use of our internal data platforms and applying appropriate analytical techniques.
- Develop, train, and apply data engineering techniques to automate manual processes, and solve challenging business problems.
- Ensure the quality, security, reliability, and compliance of our solutions by applying our digital principles and implementing both functional and non-functional requirements.
- Build observability into our solutions, monitor production health, help to resolve incidents, and remediate the root cause of risks and issues.
- Understand, represent, and advocate for client needs.
- Codify best practices, methodology and share knowledge with other engineers in UBS
- Shape the Data and Distribution architecture and technology stack within our new cloud-based datalake-house.be a hands-on contributor, senior lead in the big data and data lake space, capable to collaborate and influence architectural and design principles across batch and real time flows
- Have a continuous improvement mindset, who is always on the look out for ways to automate and reduce time to market for deliveries
MUST
Your Expertise
- Experience in building Data Processing pipeline using various ETL/ELT design patterns and methodologies to Azure data solution, building solutions using ADLSv2, Azure Data factory, Databricks, Python and PySpark.
- Experience with at least one of the following technologies: Scala/Java or Python
- Deep understanding of the software development craft, with focus on cloud based (Azure), event driven solutions and architectures, with key focus on Apache Spark batch and streaming, Datalakehouses using medallion architecture. Knowledge of DataMesh principles is added plus.
- Ability to debug using tools Ganglia UI, expertise in Optimizing Spark Jobs
- The ability to work across structured, semi-structured, and unstructured data, extracting information and identifying linkages across disparate datasets.
- Expert in creating data structures optimized for storage and various query patterns for e.g. Parquet and Delta Lake
- Experience in traditional data warehousing concepts (Kimball Methodology, Star Schema, SCD) / ETL tools (Azure Data factory, Informatica)
- Experience in data modelling atleast one database technology such as:
o Traditional RDBMS (MS SQL Server, Oracle, PostgreSQL)
o NoSQL (MongoDB, Cassandra, Neo4J, CosmosDB, Gremlin)
o Understanding of Information Security principles to ensure compliant handling and management of data
NICE TO HAVE- Ability to clearly communicate complex solutions.
- Strong problem solving and analytical skills.
- Working experience in Agile methodologies (SCRUM)
Luxoft DXC
2000
Luxoft, a DXC Technology Company is a global digital strategy and software engineering firm with over 17,000 international employees within its 58 offices in 29 countries. It is headquartered in Zug, Switzerland.
In January 2019, Luxoft was acquired by U.S. company DXC Technology. Luxoft partnered with LG Electronics to create a next-generation Autonomous Mobility concept vehicle that integrates consumers' personalized digital lifestyles into a driving experience. Luxoft enabled Switzerland's first Blockchain based e-vote platform with the City of Zug and Hochschule Luzern's Blockchain Lab.
Luxoft, a DXC Technology Company is a world-renowned company. It has been present on the Polish market for over 13 years. We have offices in Krakow, Warsaw, Wroclaw, and Gdansk. We employ over 2,000 professional experts carrying out projects for over 100 clients from the financial, automotive, medical, tourist industries, etc. We work for many international clients, including the USA, Great Britain, and Switzerland.
So far, Luxoft Poland has made a name for itself as a company that offers work on innovative projects, we offer various experiences in the field of IT, opportunities for rapid development, an extensive training program, and attractive benefits for employees.
At present, 62% of Luxoft Poland employees come from Poland, and 38% from around 50 countries, including Ukraine, Brazil, India, Turkey, Spain, Portugal, Italy, Romania, USA, etc.
At Luxoft, a DXC Technology Company almost 80 percent of employees are experts with the "Senior" experience level, with at least five years of experience. We care about our employees, so every day we try to provide them with the best possible conditions for work and development.
Technology is our passion! We focus on top engineering talent means that you will be working with the best industry professionals from around the world. Because of that, Luxoft is a global family with an epic atmosphere – we love what we do!