Overview
This is a contract, remote position for a Legal Quality Assurance Lead. The role involves overseeing quality, consistency, and trainer performance across legal AI training projects. The lead will review AI-generated legal content and trainer/QA work, evaluate output quality against project guidelines, provide precise written feedback, and ensure adherence to expected quality standards.
Responsibilities
- Monitor quality by spot-checking legal items, identifying issues, providing ongoing feedback, and escalating critical problems.
- Evaluate AI-generated legal explanations, research responses, contract analyses, policy interpretations, case summaries, compliance guidance, and issue-spotting workflows for accuracy, clarity, and appropriate caution.
- Communicate updates to trainers and QAs regarding new item guidelines, project changes, workflow updates, quality expectations, and legal-review standards.
- Respond promptly and clearly to trainer/QA questions related to legal reasoning, jurisdiction, citations, source quality, disclaimers, contract language, compliance interpretation, and rubric application.
- Manage activation of trainers and QAs by encouraging engagement, tracking follow-ups, and flagging availability issues.
- Create and maintain legal project documentation including style guides, trackers, FAQs, quality notes, examples, honeypots, calibration tasks, and onboarding materials.
- Schedule and conduct onboarding and training sessions to explain project expectations, workflows, rubrics, quality standards, and legal-specific review requirements.
- Ensure consistent application of legal-review guidelines among all trainers and QAs and communicate updates as projects evolve.
- Identify and flag unsafe, misleading, overconfident, or jurisdictionally inappropriate legal outputs, especially those resembling personalized legal advice.
- Propose workflow improvements and help develop scalable QA processes for legal AI training projects.
Requirements
- Bachelor’s degree, JD, LLB, LLM, or equivalent in Law, Legal Studies, Paralegal Studies, Compliance, Public Policy, or a closely related field.
- Strong command of English to follow project guidelines, communicate with teams, and provide clear legal-review feedback.
- Minimum of 3 years professional experience in legal practice, legal research, contract review, compliance, regulatory analysis, paralegal work, legal operations, legal writing, legal education, or related workflows.
- Solid understanding of core legal concepts including legal reasoning, issue spotting, contract interpretation, legal research, statutory interpretation, case analysis, compliance, risk assessment, and legal writing standards.
- Ability to evaluate legal content against detailed rubrics and identify issues such as unsupported conclusions, hallucinated citations, jurisdictional mismatches, overconfident advice, missing caveats, incorrect standards, incomplete reasoning, or unclear explanations.
- Familiarity with legal workflows or tools such as contract review, legal memoranda, case summaries, regulatory guidance, citation checking, legal databases, document review, compliance checklists, or policy analysis is preferred.
- Experience leading or supporting remote teams of trainers, annotators, reviewers, lawyers, paralegals, legal writers, researchers, or QAs is strongly preferred.
- Comfortable working in fast-paced remote environments using tools like Discord, Google Sheets, Google Docs, trackers, dashboards, and project management systems.
- Highly detail-oriented and organized, capable of maintaining style guides, FAQs, trackers, onboarding materials, honeypots, calibration tasks, and other quality documentation.
- Experience with AI training, data annotation, large language models, prompt/response evaluation, legal content QA, or rubric-based LLM evaluation is a strong plus.
Compensation & Benefits
- This is an hourly, remote contractor role.
- There is no immediate project available; however, qualified candidates will be among the first contacted for relevant future opportunities.
- Access to future projects through the expert network is provided.
Location