LLM Turkey
Individual · ACD/01

EvalOpsSpecialist

Become an evaluator of AI language models.

The program teaches all 12 scenarios and 9 quality parameters of the Judex evaluation framework. Safety, data protection and eval set design are part of it too.

Enroll
12 weeks
Total length
8 + 4
Theory + Practice
Judex
Working platform
EvalOps Specialist
Graduation title
Evaluation framework

12 scenarios × 9 parameters

The program teaches the full Judex framework. In each scenario the 9 parameters carry different weights that add up to 100%. You learn to score model output with these weights and justify every score.

Jailbreaks, prompt injection, data protection and data leakage are not separate scenarios. They are attack and failure types evaluated within the Safety scenario.

12 Scenarios
  • ARITArithmetic & Numerical ReasoningP7 · 40%
  • SAFESafety & Harmful Content HandlingP5 · 35%
  • CODECoding & Technical Problem SolvingP7 · 25%
  • LOGICLogical ReasoningP7 · 45%
  • CONTEXTContext Understanding & UseP4 · 35%
  • HALUHallucination Detection & Factual ReliabilityP3 · 30%
  • TRTurkish Language ProficiencyP8 · 35%
  • LONGLong-Text Comprehension & SummarizationP4 · 25%
  • FMTFormat & Instruction ComplianceP2 · 40%
  • TOOLSTool-Use ReasoningP4 · 20%
  • CREACreative GenerationP8 · 30%
  • GENGeneral KnowledgeP3 · 30%
Heaviest weight →
9 Parameters
  • P1Helpfulness & Task Completion
  • P2Instruction & Format Following
  • P3Truthfulness & Factual Accuracy
  • P4Groundedness & Context Fidelity
  • P5Safety, Compliance & Risk Awareness
  • P6Bias, Fairness & Inclusivity
  • P7Reasoning & Problem Solving Quality
  • P8Clarity, Tone & Communication
  • P9Robustness, Consistency & Recoverability
Part 1

Theoretical Foundations

8 weeks

Two hours a week: the first hour is theory, the second is example-heavy practice. Weeks 1–7 complete all 12 scenarios and 9 parameters; week 8 is dedicated to building eval sets.

  1. Week01

    LLM Foundations

    Theory + Practice
    Transformer · Token · Prompt · Model Output
    • What a large language model is; intro to the Transformer
    • Tokens, prompts and how a model response is formed
    • Why models get things wrong; root causes of hallucination
    • The rater role: one prompt, two outputs compared
    • First look at the 12 scenarios and 9 parameters
    FocusCore understanding
    PracticeReviewing two model answers to the same prompt; incomplete, verbose and fabricated answers
    OutcomeUnderstands that model output is judged on several quality dimensions, not a single good/bad verdict.
  2. Week02

    Reasoning

    Theory + Practice
    Arithmetic & Numerical Reasoning · Logical Reasoning
    • Calculation, multi-step problems and numerical consistency
    • Inference, assumptions, contradictions and conclusions
    • Correct answer vs. correct reasoning
    • Scoring task completion, accuracy and reasoning separately
    FocusP1 · P3 · P7
    PracticeRight answer with wrong reasoning, wrong answer with convincing explanation; comparing two outputs by weight
    OutcomeSeparates answer accuracy from reasoning quality and applies weighted scoring in both reasoning scenarios.
  3. Week03

    Safety & Reliability

    Theory + Practice
    Safety & Harmful Content Handling · Hallucination Detection
    • Safe answers, correct refusal and over-refusal
    • Jailbreaks, prompt injection, multi-turn and obfuscated attacks
    • KVKK (Turkish data protection), personal data masking and data leakage
    • Fabricated sources, dates and procedures; overconfidence
    • Risk Severity: justifying the impact level of a failure
    FocusP3 · P4 · P5 · P6 · P9
    PracticeEvaluating document-embedded prompt injection, multi-turn attacks and contexts containing personal data
    OutcomeDistinguishes safety violations from hallucinations and evaluates attack types and data-protection risks within the safety scenario.
  4. Week04

    Context & Long Text

    Theory + Practice
    Context Understanding & Use · Long-Text Comprehension & Summarization
    • Fidelity to the given source and context
    • Overriding the provided context with general knowledge
    • Losing critical details or adding fabrications in long text
    • Summary quality and task completion
    • Consistency across paraphrase, ordering and typo variants
    FocusP1 · P2 · P4 · P8 · P9
    PracticeEvaluating answers against a company policy; scoring helpfulness, instruction following and context fidelity separately
    OutcomeScores context fidelity separately from helpfulness and instruction following.
  5. Week05

    Instructions, Format & Language

    Theory + Practice
    Format & Instruction Compliance · Turkish Language Proficiency
    • Format constraints such as JSON, tables, bullet lists and word limits
    • Instruction following vs. task completion
    • Turkish fluency, naturalness and cultural context
    • Tone, clarity and inclusivity
    FocusP1 · P2 · P6 · P8 · P9
    PracticePlain text where JSON was asked, 5 bullets instead of 3, unnatural Turkish and off-tone answers
    OutcomeEvaluates format compliance, helpfulness and language quality independently.
  6. Week06

    Technical Evaluation

    Theory + Practice
    Coding & Technical Problem Solving · Tool-Use Reasoning
    • Working code vs. a correct solution
    • Edge cases, error handling and code security
    • Choosing the right tool and parameters; interpreting tool output
    • Recovering from a failed tool call
    FocusP1 · P2 · P4 · P5 · P7 · P9
    PracticeWorking but insecure code, wrong tool choice and misread tool output
    OutcomeJudges technical output not just on whether it runs, but on task, safety, context and reasoning.
  7. Week07

    Creative Generation & General Knowledge

    Theory + Practice
    Creative Generation · General Knowledge · Framework wrap-up
    • Task fit, originality and communication in creative output
    • Accuracy, uncertainty and source limits in general knowledge
    • Why the same parameter carries different weight per scenario
    • Choosing the right scenario for ambiguous cases
    • Full review of the 12 scenarios and 9 parameters
    FocusP1–P9
    PracticeComparing two model outputs across all parameters and a mini rater calibration
    OutcomeUses the 12 scenarios and 9 parameters for scenario selection, weighting and justified scoring.
  8. Week08

    Building Eval Sets & Calibration

    Theory + Practice
    Test cases · Rubrics · Gold / proxy-gold · IRR / RQI
    • Anatomy of a good test case: scenario, expected behavior, critical failure, rubric
    • Normal, edge-case, adversarial and regression tests
    • Dataset hygiene: duplicate tests and answer leakage
    • Adding safety tests: jailbreak, prompt injection, data leakage
    • Rater calibration, IRR / RQI and rubric revision
    FocusP1–P9
    PracticeWriting normal, edge and adversarial tests for one scenario; building the final mini eval set
    OutcomeDesigns a measurable, consistently scorable mini eval set and calibrates raters with IRR / RQI.
Part 2

Practice

4 weeks

Real evaluation + IRR (Inter-Rater Reliability).

  1. Week09

    Core Evaluation

    Full Practice
    • A/B model comparison
    • 9-parameter scoring
    • Justified selection
    • Short report writing
    OutcomeScores two models across 9 parameters with justification.
  2. Week10

    Critical-Domain Evaluation

    Full Practice
    • Risk analysis
    • Safety evaluation
    • Truthfulness testing
    • Flagging high-risk responses
    OutcomeStructurally flags risks in critical-domain responses.
  3. Week11

    Human-Factor Evaluation

    Full Practice
    • Fairness analysis
    • Safety check
    • Originality measurement
    • Human-preference comparison
    OutcomeReports originality, fairness, and safety analyses together.
  4. Week12

    Final Judex Assessment

    Individual Assessment
    • Full evaluation report
    • Comparative model analysis
    • Evaluation specialist report
    • Oral defense
    OutcomeProduces and defends a complete comparative evaluation report.
By the end of the program
  • ✓Scores model output with the 12 scenarios and 9 parameters
  • ✓Writes justified evaluations based on scenario weights
  • ✓Evaluates jailbreaks, prompt injection and multi-turn attacks
  • ✓Detects data-protection, masking and data-leakage risks
  • ✓Designs eval sets with rubrics and gold / proxy-gold references
  • ✓Calibrates raters with IRR / RQI
Graduation title
EvalOps Specialist
Enroll