Legal & Compliance Specialist – Rubric Design

Overview

This hourly, remote contractor role is for a Legal & Compliance Subject Matter Expert who will create challenging professional questions, evaluate AI-generated legal and regulatory content, and design high-quality rubrics that define what a complete, well-reasoned, and professionally appropriate response should include.

Interview language: English.

Responsibilities

• Create legal and compliance tasks: Develop realistic, difficult scenarios requiring expert legal analysis and professional judgment.
• Design detailed rubrics: Define relevant issues, rules, standards, analytical steps, exceptions, risks, and critical errors.
• Evaluate AI-generated analysis: Assess responses for accuracy, completeness, reasoning quality, relevance, and professional rigor.
• Review issue spotting: Determine whether the model identifies material legal or regulatory considerations.
• Assess reasoning quality: Flag unsupported conclusions, incorrect application of rules, missing exceptions, and faulty assumptions.
• Critique peer rubrics: Review rubrics for missing issues, ambiguity, redundancy, excessive subjectivity, or inaccurate legal expectations.
• Develop reference content: Create high-quality analyses, answer frameworks, critiques, and gold-standard examples.
• Support AI training: Produce questions, rubrics, evaluations, preference data, and reference-answer pairs for RL and SFT workflows.
• Maintain professional quality: Ensure content appropriately distinguishes factual analysis, legal interpretation, professional judgment, and uncertainty.

Requirements

• JD, LLB, LLM, law degree, compliance qualification, or equivalent professional background relevant to the stated specialty.
• Professional legal qualification or bar admission is strongly preferred for attorney-focused roles.
• Strong professional proficiency in English (minimum C1).
• 3+ years of professional experience in law, regulatory compliance, legal research, policy, risk, privacy, financial compliance, contracts, or a related field.
• Demonstrated expertise in a clearly defined legal, regulatory, or compliance domain.
• Ability to create challenging professional scenarios requiring issue spotting, interpretation, analysis, and judgment.
• Strong ability to design rubrics defining required issues, applicable standards, reasoning steps, relevant exceptions, risks, and critical omissions.
• Ability to distinguish between legitimate alternative legal interpretations and objectively deficient analysis.
• Comfortable reviewing and critiquing rubrics developed by other legal or compliance experts.
• Strong attention to detail when evaluating terminology, factual assumptions, jurisdiction, authority, exceptions, materiality, and conclusions.

Preferred Qualifications

• Experience with legal education, examination writing, peer review, legal editing, quality assurance, policy drafting, compliance testing, or supervisory review.
• Prior experience with AI evaluation, legal AI, data annotation, RLHF, model benchmarking, or rubric-based scoring.

Compensation & Benefits

• Contract / hourly arrangement (details not specified).

Location

• Remote