The role of a DBT developer is pivotal in automating and optimizing data movement between critical data layers—Staging (STG), Integration (INT), Business (BUS), and Presentation (PRS)—in a data pipeline built on Azure. This role is required to bridge the gap between Azure Data Factory (ADF), which orchestrates the pipeline, and DBT, which executes transformations and ensures data readiness for consumption. ADF lacks the necessary permissions to manage data movement directly, making it essential for DBT to be triggered by ADF for accurate and secure data transformation and movement.
This DBT-driven automation enhances efficiency, minimizes manual intervention, and ensures that data is processed consistently and is ready for end-user consumption in a timely manner. It plays a crucial role in meeting business intelligence and reporting needs by ensuring that data flows seamlessly across layers, maintaining data integrity, security, and availability.
MUST
- DBT (Data Build Tool):
o Expertise in building, orchestrating, and managing data models using DBT, including deep knowledge of DBT commands, Jinja macros, and configurations.
o Ability to define and manage dependencies between data models across layers (STG, INT, BUS, PRS) to ensure smooth orchestration and pipeline execution.
o Proficiency in developing and managing DBT projects, performing testing, documenting models, and handling data transformation automation workflows.
- Snowflake Expertise:
o Proficient in working with Snowflake as the data warehouse solution, leveraging Snowflake's architecture to optimize DBT pipelines.
o Experience in writing and optimizing SQL queries for Snowflake, ensuring efficient use of virtual warehouses, data partitioning, and clustering.
o Knowledge of Snowflake roles, access control, and permissions management to ensure secure and seamless data operations across environments.
- SQL Proficiency:
o Strong SQL skills to write, optimize, and troubleshoot complex queries tailored for Snowflake's environment.
o Ability to design efficient transformations using DBT and SQL, handling large datasets with performance tuning in mind.
- Azure Data Platform:
o Proficiency with Azure services such as Azure Data Factory (ADF), Azure Synapse Analytics, and Data Lake, ensuring seamless integration between DBT, ADF, and the Snowflake data platform.
o Ability to use ADF to orchestrate and automate data pipelines and trigger DBT executions through APIs or custom activities.
- Version Control (e.g., Git):
o Hands-on experience with Git for version control of DBT models, managing branches, collaborating with teams, and resolving merge conflicts efficiently.
- Data Warehousing and ETL Concepts:
o Deep knowledge of data warehousing architectures and best practices, including experience in layer-based architecture (STG, INT, BUS, PRS) for organizing data flow.
o Expertise in ETL/ELT processes, data transformation, and ensuring consistent data quality across environments.
- CI/CD Pipelines:
o Experience with continuous integration and deployment (CI/CD) practices, particularly in automating DBT projects using ADF, Jenkins, or other CI tools.
o Familiarity with using GitHub Actions, Azure DevOps, or similar tools to automate the build, testing, and deployment of DBT models.
NICE TO HAVE- Orchestration Frameworks:
o Proficient in setting up DBT orchestration frameworks using ADF, Airflow, or Prefect to schedule and automate DBT runs, ensuring timely data processing across environments.
o Hands-on experience in managing orchestration dependencies between DBT, Snowflake, and ADF pipelines, ensuring reliable data flow.
- API Integration:
o Experience with integrating DBT Cloud or DBT Core with other systems using APIs, including triggering DBT executions via ADF custom activities or REST API calls.
o Ability to automate end-to-end data pipelines that span across Snowflake, DBT, and Azure services.
- Problem-Solving:
o Strong analytical and problem-solving skills, with the ability to troubleshoot and resolve issues in data pipelines, optimize transformations, and manage bottlenecks efficiently.
- Collaboration:
o Experience working closely with data engineers, analytics teams, and business stakeholders to ensure alignment on data needs, model design, and data quality.
o Ability to work cross-functionally, especially with Azure administrators, Snowflake architects, and other DBT developers, to ensure seamless data integration.
- Documentation:
o Ability to create and maintain comprehensive documentation for DBT models, workflows, and orchestration setups, ensuring all stakeholders are informed of data dependencies and transformations.
- Adaptability and Innovation:
o Willingness to continuously learn and adapt to emerging tools, techniques, and best practices in data orchestration, automation, and cloud-based data platforms.
Luxoft Poland
2000
Luxoft, a DXC Technology Company is a global digital strategy and software engineering firm with over 17,000 international employees within its 58 offices in 29 countries. It is headquartered in Zug, Switzerland.
In January 2019, Luxoft was acquired by U.S. company DXC Technology. Luxoft partnered with LG Electronics to create a next-generation Autonomous Mobility concept vehicle that integrates consumers' personalized digital lifestyles into a driving experience. Luxoft enabled Switzerland's first Blockchain based e-vote platform with the City of Zug and Hochschule Luzern's Blockchain Lab.
Luxoft, a DXC Technology Company is a world-renowned company. It has been present on the Polish market for over 13 years. We have offices in Krakow, Warsaw, Wroclaw, and Gdansk. We employ over 2,000 professional experts carrying out projects for over 100 clients from the financial, automotive, medical, tourist industries, etc. We work for many international clients, including the USA, Great Britain, and Switzerland.
So far, Luxoft Poland has made a name for itself as a company that offers work on innovative projects, we offer various experiences in the field of IT, opportunities for rapid development, an extensive training program, and attractive benefits for employees.
At present, 62% of Luxoft Poland employees come from Poland, and 38% from around 50 countries, including Ukraine, Brazil, India, Turkey, Spain, Portugal, Italy, Romania, USA, etc.
At Luxoft, a DXC Technology Company almost 80 percent of employees are experts with the "Senior" experience level, with at least five years of experience. We care about our employees, so every day we try to provide them with the best possible conditions for work and development.
Technology is our passion! We focus on top engineering talent means that you will be working with the best industry professionals from around the world. Because of that, Luxoft is a global family with an epic atmosphere – we love what we do!