Domain Expert, AI Evaluation Task Writer: Legal (Contract, Remote)

Overview

This role involves designing realistic legal tasks and evaluation artifacts for AI model testing and improvement. Each task is based on a scenario provided by the team. You will create working documents that a qualified legal professional could produce, write the task request, solve the task, and define how answers should be graded.

Engagement type: Independent contractor (fully remote, asynchronous)

Responsibilities

• Build realistic evidence files from a provided scenario, including items such as:
  • Services agreements and amendments
  • Email threads
  • Pleadings and discovery requests
  • Privilege logs
  • Docket entries
  • Closing checklists and disclosure schedules
  • Signature packages
  • Cap tables
  • Contract playbooks and redlines
  • Request intake records
  • Data maps and retention schedules
• Write task requests that clearly state the deliverable and how uncertainty and justified assumptions should be handled.
• Solve each task end to end and document a reference solution with reasoning behind every step.
• Define explicit scoring criteria so two independent graders reach the same verdict, including:
  • Acceptable alternative answers
  • Limits of what the solver is authorized to decide
• Run tasks against current models, identify where and why they fail, and adjust difficulty.
• Review your own and peers’ tasks for legal accuracy, ambiguity, answers exposed in provided materials, and shortcuts that skip intended reasoning.
• Incorporate reviewer feedback within agreed turnaround times; flag guideline gaps or edge cases.

Areas Hiring For

• Civil litigation: paralegal and litigation support work across discovery and productions, privilege review and logs, depositions, pleadings and motions, subpoenas, dockets and deadlines under federal and local rules.
• Corporate and M&A transactions: transactional paralegal work on due diligence, data rooms, disclosure schedules, signature pages and closings, entity records and cap tables, and purchase and merger agreements.
• Commercial contracts: contract managers, specialists, and administrators who review and redline against playbooks; handle MSAs, amendments, renewals and notice periods; track obligations; and support procurement or vendor contracts.
• Privacy and data subject requests: privacy and data protection analysts who fulfill access and deletion requests end to end, maintain data maps, ROPAs and retention schedules, and apply GDPR and CCPA in practice.

Requirements

• 3+ years of hands-on experience in one or more of the four areas above (paralegal, legal assistant, contracts manager or specialist, privacy or data protection analyst, or attorney). Experience must include doing the work yourself, not only supervising it.
• Working knowledge of rules, procedures, and documentation standards governing your area (e.g., federal and local court rules and discovery practice, closing mechanics, contract playbooks and CLM workflows, GDPR and CCPA request handling).
• Careful reading and the habit of citing the clause, message, or docket entry behind every conclusion.
• Clear, precise writing in professional English.
• Ability to work independently and asynchronously against deadlines.

Nice to Have

• Paralegal certification (for example NALA CP or NFPA RP), a law degree, CIPP/US or CIPP/E, or a contract management credential.
• Experience with systems used in your area, such as Relativity, Everlaw or DISCO, a virtual data room, Ironclad, Icertis, Agiloft or Conga, or OneTrust, TrustArc or Transcend.
• Law firm, in-house, and legal operations experience across more than one setting.
• Prior experience writing or reviewing AI evaluation tasks, exam questions, case studies, or training material.
• Familiarity with how AI models fail (fabricated citations, misread clauses, miscounted deadlines, overconfident conclusions).

Compensation & Benefits

• Pay: $50–150/hr for task authoring, paid when the full task is submitted, plus a potential bonus for each accepted task.
• Strong performers may move up within that range or onto productivity-based pay.
• The trial task is paid.

Location and Schedule

• Location: Remote
• Commitment: 10+ hours per week
• Start: Rolling, as soon as you pass the trial task
• Engagement timing: fully remote, asynchronous

Standards (Must Follow)

• All tasks, documents, and solutions must be human-authored; LLM-generated content is not permitted.
• Tasks and solutions must be original; nothing may be copied from published or easily searchable sources.
• Never use real client, matter, employer, or personal data, or anything covered by privilege or a confidentiality obligation. All parties and scenarios must be fictional.
• Quality is judged on accuracy, rubric validity, difficulty, and reviewer agreement—not volume.

How Hiring Works

• Short AI interview and resume review
• One paid trial task, scored against a fixed quality checklist
• Onboarding and your first project
• Decision timing: aim to make a decision within two weeks of application

Equal Opportunity

This organization is an equal opportunity employer and provides consideration without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected characteristics.