Senior Data Engineer (with AI/ML exposure)
IDT (www.idt.net) is a communications and financial services company founded in 1990 and headquartered in New Jersey, US. Today it is an industry leader in prepaid communication and payment services and one of the world’s largest international voice carriers. We are listed on the NYSE, employ over 1700 people across 20+ countries, and have revenues in excess of $1.5 billion.
We are looking for a skilled Senior Data Engineer (with AI/ML exposure) to join our BI team and take an active role in designing, building, and maintaining the end-to-end data pipeline, architecture and design that powers our warehouse, LLM-driven applications, and AI-based BI. If you're looking for a company that will give you the maximum flexibility in choosing a location to work, this opportunity is for you!
- The interview process will be conducted in English.
Responsibilities:
- Design, develop, and maintain scalable data pipelines to support ingestion, transformation, and delivery into centralized feature stores, model-training workflows, and real-time inference services.
- Design, optimize, and maintain robust ETL/ELT pipelines and data structures within our cloud data warehouse (Snowflake/Redshift) to support core Business Intelligence.
- Build and optimize workflows for extracting, storing, and retrieving semantic representations of unstructured data to enable advanced search and retrieval patterns.
- Architect and implement lightweight analytics and dashboarding solutions that deliver natural language query experience and AI-backed insights.
- Define and execute processes for managing prompt engineering techniques, orchestration flows, and model fine-tuning routines to power conversational interfaces.
- Oversee vector data stores and develop efficient indexing methodologies to support retrieval-augmented generation (RAG) workflows.
- Partner with data stakeholders to gather requirements for language-model initiatives and translate into scalable solutions.
- Create and maintain comprehensive documentation for all data processes, workflows and model deployment routines.
- Should be willing to stay informed and learn emerging methodologies in data engineering, MLOps and LLM operations.