Senior Data Scientist AI Evaluation

Your Role : We're looking for a Senior Data Scientist, AI Evaluation to design how Alpaca measures whether our models and agents are actually right. You'll be a senior individual contributor who turns ambiguous quality questions into ground truth, scoring methods, and eval loops that the company can trust—and uses those results to make the systems better. You'll build on an established data foundation, so the focus is raising quality and speeding up safe rollout. You'll own the quality bar, independent of the teams that build and optimize those systems.

This role is for someone who cares as much about whether an answer is correct as about whether a model can generate one. You'll partner with Product, Engineering, Analytics Engineering, and business stakeholders to define what "good" looks like, build the evaluations that test it, and close the loop so evals drive iteration. If you have a strong quantitative background, have shipped rigorous, measurable work (evaluation, experimentation, or model validation), and want ownership over a greenfield eval practice at a fast-growing brokerage-infrastructure company, this is the role.

• Design AI evaluations: Define ground truth, metrics, and scoring methods for models and agents.
• Build repeatable eval loops: Track quality over time and catch regressions before release.
• Partner on infrastructure: Work with engineering and analytics engineering to operationalize eval harnesses.
• Drive iteration: Translate eval results into actionable recommendations for system improvements.
• Establish quality standards: Set evaluation guidelines, documentation, and review practices.
• Mentor and align: Foster evaluation best practices and build a culture of measurable AI quality across the team.

What We're Looking For

…