Role Overview:
Data quality and diversity is the foundation for training the best agents in any domain. As a member of the Data Team at Reflection, you will play a pivotal role in shaping how we collect and analyze human, synthetic, and internet data. This is an interdisciplinary role that primarily requires engineering, research, and communication skills, along with a sharp attention to detail and willingness to “roll up your sleeves” and look at the data.
Key Responsibilities
1. Experiment and Benchmark Design
Develop techniques for collecting, augmenting, filtering, or synthesizing training and evaluation data using creativity and analytical thinking
Design experiments, in collaboration with machine learning researchers, to assess the impact of different datasets on model performance
When required, manage human annotators working on data collection efforts – this could include tracking payments and hours, training annotators, and providing technical support, feedback, and quality control
2. Qualitative and Quantitative Data Analysis
Analyze collected data, e.g. coding tasks, both qualitatively and quantitatively
Evaluate model behavior to identify its strengths and weaknesses
Clearly communicate findings with machine learning research and product teams
3. Data Engineering
Design, implement, and optimize scalable data pipelines to support reinforcement learning and supervised finetuning
Leverage LLMs to perform data filtering, cleaning, and augmentation
Qualifications:
Software engineering background with experience building data processing pipelines at scale, particularly with LLM integration
Proficiency in Python or other programming languages (Go, TypeScript, etc.)
Detail-oriented and analytical, with the ability to conduct careful qualitative and quantitative data analysis.
Excellent organizational and communication skills to collaborate closely with cross-functional teams and manage human data operations
Experience with machine learning, reinforcement learning, and LLMs is a plus, but not strictly required.
What We Offer:
The opportunity to work at the forefront of AI research and data collection for training cutting-edge models.
Collaboration with a team of world-class researchers and engineers from top AI labs and companies.
Competitive compensation and benefits, with opportunities for professional growth.
