Overview
This independent contractor role supports evaluation of legal and professional service outputs before they are delivered to customers. Lawyers review, grade, and provide qualitative feedback on AI-generated work using written rubrics and realistic test materials.
The Evaluation Operations function partners with the Legal Product Architect team to scope and answer relevant research questions that inform key product decisions. Work is completed remotely in pre-release environments and is not client-facing.
Responsibilities
- Compare different AI-generated outputs for the same legal task (e.g., a memo, a contract redline, a research answer) and evaluate output quality against a written rubric.
- Test new product features end-to-end in a pre-release environment and provide qualitative feedback based on your legal experience.
- Review and assess grading rubrics for accuracy and completion.
- Build evaluation datasets that include realistic legal queries, contract markups, and/or synthetic legal documents.
- Work on short-notice projects when requested; timelines vary by project.
- Project timing may be 2 to 20 hours over multiple days.
- Some projects require feedback within 24–48 hours of the request.
- You may decline any project.
Requirements
- JD and an active license to practice in at least one U.S. jurisdiction.
- 3+ years of practice experience at an AmLaw 100 or comparable firm, or in an in-house legal department.
- Experience in either commercial litigation or transactional law.
- Experience using AI tools in professional or daily life.
- Clear, concise legal writing and the ability to express reasoning in plain language.
- Availability to pick up short-notice projects and occasionally work over weekends.
Preferred Qualifications
- Experience using Harvey or another legal AI tool in practice.
- Prior experience grading, annotating, and/or reviewing AI outputs.
- For litigators: Lexis or Westlaw proficiency and cite-checking experience.
- A combination of law-firm and in-house experience.
Compensation & Benefits
- Compensation: $125 USD per hour.
- Benefits: Not included.
Engagement Details
- Type: Independent contractor (1099), engaged directly by the organization. You invoice for hours worked.
- Commitment: No minimum. Work is offered per project; you accept or decline each one.
- Hours and location: Remote, on your own schedule, within each project’s deadline. US-based.
- Equipment and access: The organization ships you a laptop and provisions a company account for pre-release environments.
- Term: 6 months.
- Not included: Benefits, client representation, court work, or business development.
Location
Remote. US-based.