Legal & Compliance Specialist – Rubric Design

Overview

Hourly, remote contractor role serving as a Legal & Compliance Subject Matter Expert (SME). The work focuses on creating challenging professional questions, evaluating AI-generated legal and regulatory content, and designing high-quality rubrics that define what a complete, well-reasoned, and professionally appropriate response should include.

The assignments may involve legal analysis, regulatory interpretation, compliance, issue spotting, risk assessment, policy interpretation, contracts, privacy, financial regulation, employment matters, corporate compliance, or related fields, depending on specialization.

Responsibilities

• Create legal and compliance tasks by developing realistic, difficult scenarios requiring expert analysis and professional judgment.
• Design detailed rubrics defining:
  • Relevant issues
  • Applicable rules and standards
  • Analytical steps
  • Exceptions
  • Risks
  • Critical errors and omissions
  • Jurisdictional considerations
  • Expected professional reasoning quality
• Evaluate AI-generated analysis for accuracy, completeness, reasoning quality, relevance, and professional rigor.
• Review issue spotting to confirm material legal or regulatory considerations are identified.
• Assess reasoning quality by identifying unsupported conclusions, incorrect application of rules, missing exceptions, and faulty assumptions.
• Critique peer rubrics for missing issues, ambiguity, redundancy, excessive subjectivity, or inaccurate legal expectations.
• Develop reference content, including high-quality analyses, answer frameworks, critiques, and gold-standard examples.
• Support AI training by producing questions, rubrics, evaluations, preference data, and reference-answer pairs for RL and SFT workflows.
• Maintain professional quality by ensuring content appropriately distinguishes factual analysis, legal interpretation, professional judgment, and uncertainty.

Requirements

• JD, LLB, LLM, law degree, compliance qualification, or equivalent professional background relevant to the stated specialty.
• Professional legal qualification or bar admission is strongly preferred for attorney-focused roles.
• Strong professional proficiency in English (minimum C1).
• 3+ years of professional experience in one or more of the following: law, regulatory compliance, legal research, policy, risk, privacy, financial compliance, contracts, or a related field.
• Demonstrated expertise in a clearly defined legal, regulatory, or compliance domain.
• Ability to create challenging professional scenarios requiring issue spotting, interpretation, analysis, and judgment.
• Ability to design rubrics that clearly define required issues, applicable standards, reasoning steps, relevant exceptions, risks, and critical omissions.
• Ability to distinguish between legitimate alternative legal interpretations and objectively deficient analysis.
• Comfortable reviewing and critiquing rubrics developed by other legal or compliance experts.
• Strong attention to detail when evaluating terminology, factual assumptions, jurisdiction, authority, exceptions, materiality, and conclusions.

Preferred Qualifications

• Experience with legal education, examination writing, peer review, legal editing, quality assurance, policy drafting, compliance testing, or supervisory review.
• Prior experience with AI evaluation, legal AI, data annotation, RLHF, model benchmarking, or rubric-based scoring.

Compensation & Benefits

• Hourly contractor role. Compensation details are not specified.

Location

• Remote