Paralegal - AI Evaluation

Overview

This role focuses on evaluating whether frontier AI models can answer questions derived from real legal documents. You will read court opinions, filings (including exhibits), and commercial contracts, identify the kinds of questions a paralegal would need to answer from those documents, and assess model responses for correctness and support.

Responsibilities

• Read court opinions, filings with exhibits, and commercial contracts, then determine the questions a paralegal would face (e.g., tracing a contract’s defined terms and cross-references to what a party actually owes).
• Write challenging questions that require combining information from multiple parts of a document, including:
  • the correct answer
  • a checklist for grading responses
• Review answers written by AI models and other experts, flagging anything that is wrong, missing, or unsupported by the document.
• Provide clear feedback describing how responses should change.
• Judge whether each task reflects how these documents are used in real legal work, maintaining consistency across reviews.

Qualifications

• Paralegal certificate or bachelor’s degree.
• At least 3 years of experience as a paralegal in litigation, corporate, or contracts work.

Required Skills

• Experience reading contracts and court documents closely, including definitions, cross-references, exhibits, and schedules.
• Exceptional attention to detail and comfort applying the same standard across reviews.
• Strong English written communication skills.
• An ABA-approved paralegal certificate, a JD, or cite-checking experience is a plus (not required).

Compensation & Benefits

Not specified in the provided description.

Location

Not specified in the provided description.