Support advanced AI data and evaluation initiatives by designing high-quality evaluation criteria, reviewing preference data, and developing the technical infrastructure that supports data generation and assessment workflows. This project is ideal for an experienced software engineer who combines strong technical expertise with sound qualitative judgment and can translate complex findings into clear, actionable recommendations.
Location: United States (remote)
Contract: 3 months
Availability: Minimum 4 hours/day and 20 hours/week with 4 hours daily overlap with PST
This project is paid at an hourly rate based on experience level.
Key Responsibilities:
Design, refine, and iterate on rubrics and evaluation criteria for preference data
Review and label data with a strong focus on quality, accuracy, and subtle issues
Build and maintain pipelines and infrastructure supporting data generation, collection, and evaluation workflows
Synthesize findings into clear technical write-ups, project updates, and actionable recommendations
Collaborate with researchers and engineers to translate qualitative judgment into scalable processes
Navigate and contribute to large codebases while maintaining a high standard for technical quality
Work effectively in an ambiguous, fast-moving environment with evolving project priorities
Requirements:
At least 5 years of professional software engineering experience
Advanced proficiency in Python, particularly in building and maintaining pipelines or infrastructure, or advanced proficiency in TypeScript, Go, or JavaScript
Hands-on experience with preference data collection, rubric creation, or technical evaluation criteria
Strong written and verbal communication skills, with the ability to summarize complex technical work clearly and concisely
Demonstrated ability to assess the quality of technical examples and explain the reasoning behind those assessments
Experience contributing to or navigating large codebases
Preferred Skills: Experience contributing to large-scale developer tools or codebases such as Visual Studio Code, along with experience supporting AI data generation, evaluation, or quality-review workflows
Please direct any questions to [email protected]
Services rendered do not establish any employment rights with Arise® HALO or any client. Services will be provided on an independent contractor basis. Project availability is not guaranteed. Project matching is based on client demand.
