Senior Principal AI Engineer

A Moving Experience. What You Will Work On

• Design and   operate   distributed training systems for large neural networks   (autoregressive, diffusion ,   State Space Models   etc. )   across GPU clusters
• Optimise   multi ‑ node, multi ‑ GPU execution to   maximize   throughput and   utilization
• Diagnose   &   resolve bottlenecks across compute, memory, and netwo rk
• Improve training stability and fault tolerance at scale
• Partner with research and applied ML teams to productioni z e   large ‑ model   training pipelines

Core Responsibilities

• Distributed Training Infrastructure
• Build and   optimize   GPU cluster orchestration using:
• Slurm
• Kubernetes
• Ray
• RunAI
• Ensure efficient scheduling, isolation, and fairness across training workloads
• Communication & Networking
• Optimize   and debug distributed communication using:
• NCCL
• RDMA
• InfiniBand
• NVLink
• Minimize   networking bottlenecks that dominate   end ‑ to ‑ end   training time
• Training Frameworks
• Scale large-model training using:
• PyTorch   Distributed
• Megatron ‑ LM
• DeepSpeed
• Own   multi ‑ node   launch configurations, failure recovery, and performance tuning
• Memory & Performance Optimization
• Apply advanced memory optimization techniques:
• Activation checkpointing
• ZeRO   (Stage 1–3) and offload strategies
• Balance compute, memory, and communication to push model size and batch scale

What Success Looks Like

• GPU   utilization   consistently stays high (>80–90%)
• Training scales cleanly from single node to dozens or hundreds of GPUs
• Communication overhead is minimized and predictable
• Large training jobs run stably for days or weeks without failure
• New models can be trained faster, larger, and more reliably than before

Required Experience & Skills

…