LLM Turkey

New professional program · 2027

Agentic AI Engineering & EvalOps

Building, Securing and Evaluating Agentic AI Systems with Claude Code

Build real agentic AI systems with Claude Code. Learn MCP, RAG and agent orchestration. Measure the accuracy, safety and performance of what you build with LLM & Agent Evaluation.

14 weeks
Duration
70 hours
Live instruction
2 days a week
2h theory · 3h practice
10–15
Class size
Online
Live and hands-on

About the program

For engineers who design, build and measure agentic systems.

The program covers AI systems for real business problems end to end: designing the architecture, building with Claude Code, wiring up tool and data integrations, applying security controls and measuring system quality with LLM and Agent Evaluation.

This is not a Claude Code tutorial. Claude Code is the program's main development tool; the engineering principles taught are not tied to any particular model or provider.

Level
Intermediate – Advanced
Format
Online · live · hands-on
Theory
1 day a week · 2 hours · 28 hours total
Practice
1 day a week · 3 hours · 42 hours total
Main tool
Claude Code
Class size
10–15 participants

Who is it for?

For technical professionals with software development experience.

  • —Software developers
  • —AI engineers
  • —Software engineers
  • —Tech leads
  • —Software architects
  • —DevOps engineers
  • —Experienced technical professionals who want to build agentic AI systems

Prerequisites

  • 01Basic software development experience
  • 02Working knowledge of Git
  • 03Comfort with the terminal
  • 04Familiarity with APIs and basic backend concepts

This is not a beginner program and is not suitable for participants without coding experience.

Four engineering competencies

Design, build, evaluate, operate. Then repeat.

The program treats the four competencies as one continuous engineering lifecycle rather than separate courses. Every week closes at least one part of the loop with a concrete deliverable.

  1. 01 · DESIGN

    System design

    • Agentic AI architectures
    • The agentic loop
    • Tool calling
    • Workflow design
    • Context engineering
    • RAG architecture
    • Architecture decisions and risk analysis
  2. 02 · BUILD

    Software and agent development

    • Claude Code
    • Agent harness
    • Skills and hooks
    • Git-based development
    • MCP server development
    • API integrations
    • Stateful workflows
    • Multi-agent orchestration
    • Test automation
  3. 03 · EVALUATE

    LLM & Agent Evaluation

    • Ground truth
    • Evaluation datasets
    • Rubric design
    • LLM-as-a-Judge
    • Human evaluation
    • Task success
    • Tool-call accuracy
    • Trajectory evaluation
    • Prompt injection and security testing
    • Failure analysis
  4. 04 · OPERATE

    Operations and continuous improvement

    • Tracing
    • Observability
    • CI/CD
    • Quality gates
    • Regression testing
    • Token and API costs
    • Latency measurement
    • Continuous evaluation
Evaluation at every stage

Learning outcomes

By the end of the program, participants can:

  • 01Design agentic AI architectures.
  • 02Run professional software development with Claude Code.
  • 03Build MCP and API integrations.
  • 04Build RAG-based enterprise knowledge systems.
  • 05Design stateful and multi-agent workflows.
  • 06Apply agent security and access controls.
  • 07Create LLM and agent evaluation datasets.
  • 08Use ground truth, rubrics and LLM-as-a-Judge.
  • 09Evaluate agent behaviour by task success and tool-call accuracy.
  • 10Integrate evaluation results into CI/CD and quality processes.
  • 11Measure the performance, cost and reliability of AI systems.

14-week curriculum

Every week: theory, practice, a concrete deliverable.

Each week has a 2-hour theory session and a 3-hour practice session. Every week ends with a piece added to the final project.

  1. Week 01Agentic AI Engineering Foundations

    Theory

    Agentic systems, how agents differ from LLMs, harnesses, the agentic loop and system architecture.

    Practice

    Setting up Claude Code, analysing a repository and laying out the final project's architecture.

    Outcome

    Tells the components of an agentic system apart; defines the project skeleton and its first evaluation criteria.

  2. Week 02Tool Calling & Agent Execution

    Theory

    Tool calling, agent state, tool execution and task management.

    Practice

    A first working agent and its tool integrations.

    Outcome

    Builds a working agent that calls tools and manages state, with tool calls logged.

  3. Week 03MCP & System Integration

    Theory

    MCP architecture: client–server, tools, resources; integration security and access control.

    Practice

    A custom MCP server and an enterprise API integration.

    Outcome

    Writes an MCP server with defined permission boundaries and connects it to the agent.

  4. Week 04Context Engineering & RAG

    Theory

    Retrieval, context management, embeddings, grounding and source verification.

    Practice

    Source-grounded retrieval over enterprise documents.

    Outcome

    Builds a RAG layer that cites sources and starts measuring answer groundedness.

  5. Week 05Stateful Workflows

    Theory

    Workflow orchestration, state, memory, retries, fallbacks and error handling.

    Practice

    Running multi-step enterprise tasks.

    Outcome

    Builds a traceable multi-step workflow that recovers from failures.

  6. Week 06Multi-Agent Engineering

    Theory

    Delegation, orchestration, handoffs and multi-agent architecture decisions.

    Practice

    Comparing single-agent and multi-agent approaches on the same task.

    Outcome

    Compares single- and multi-agent designs on cost, complexity and success, with justification.

  7. Week 07Reliability Engineering

    Theory

    Structured outputs, error handling, reliability and test strategy.

    Practice

    Unit and integration tests, failure scenarios and fallback behaviour.

    Outcome

    Covers the system's critical paths with tests and documents failure scenarios.

  8. Week 08Agent Security

    Theory

    Prompt injection, tool permissions, unauthorized access, sandboxing and data security.

    Practice

    Controlled security scenarios and permission-boundary tests.

    Outcome

    Produces a threat model for the agent and adds prompt-injection and privilege tests.

  9. Week 09LLM Evaluation

    Theory

    Ground truth, rubrics, metrics, human evaluation and LLM-as-a-Judge; the nine quality dimensions.

    Practice

    Building an evaluation dataset and scoring model outputs.

    Outcome

    Prepares an evaluation dataset with rubrics and ground truth, and scores outputs with justification.

  10. Week 10Agent Evaluation

    Theory

    Task success, tool-call accuracy, trajectory evaluation and failure analysis.

    Practice

    Multi-step evaluation of agent behaviour.

    Outcome

    Evaluates an agent on the path it took and the tools it used, not just its final answer.

  11. Week 11EvalOps & Regression Evaluation

    Theory

    Rater calibration, dataset versioning, evaluation automation and regression.

    Practice

    Building a repeatable evaluation pipeline.

    Outcome

    Sets up a versioned evaluation pipeline that re-runs on every change.

  12. Week 12Production Engineering

    Theory

    Observability, tracing, CI/CD, quality gates; token and latency measurement.

    Practice

    Wiring automated quality and evaluation checks into the development process.

    Outcome

    Uses evaluation results as a CI/CD quality gate and tracks cost and latency.

  13. Week 13Integrated Engineering Lab

    Theory

    System quality criteria, integrated test strategy and production readiness.

    Practice

    System testing of the final project; performance and security improvements.

    Outcome

    Runs the final project through integrated tests and improves it based on the findings.

  14. Week 14Final Engineering Assessment

    Theory

    Technical evaluation methodology, system reporting and results analysis.

    Practice

    Final project presentations, independent tests and technical assessment.

    Outcome

    Defends the system, its evaluation report and engineering decisions in a technical presentation.

How it works

Two sessions a week, one project.

Theory day · 2 hours
Concepts, architecture decisions and evaluation methods, through real examples.
Practice day · 3 hours
Live development with Claude Code. Each week's topic is added to the final project.
One project, 14 weeks
You grow the same system every week; the final builds on 13 weeks of work rather than starting from scratch.
Small class
With 10–15 participants there is time for code review and questions.

EvalOps integration

EvalOps, built into the engineering process.

LLM Turkey's EvalOps evaluation approach is now part of this program. All nine quality dimensions are taught; hands-on work goes deep on a selected set of scenarios.

9 quality dimensions

  1. P1Instruction Following
  2. P2Truthfulness
  3. P3Safety
  4. P4Bias
  5. P5Reasoning
  6. P6Clarity
  7. P7Robustness
  8. P8Explainability
  9. P9Efficiency

Agent evaluation

  • Task completion
  • Tool-call correctness
  • Trajectory evaluation
  • Policy compliance
  • Cost / performance

Operational metrics

  • Latency
  • Token and API cost
  • Tracing

Final project

Enterprise Operations Agent

Participants build an agentic AI system that handles enterprise operations requests. It uses API and MCP tools, retrieves enterprise documents through RAG, runs multi-step tasks, asks for human approval at critical steps and ships with its own evaluation infrastructure.

  1. 01Operations request
  2. 02Orchestrator agent
  3. 03MCP / API tools
  4. 04Enterprise docs · RAG
  5. 05Human approval
  6. 06Security controls
  7. 07Result + trace
  8. 08Evaluation · tracing · quality gate

What the system does

  • —Handles enterprise operations requests
  • —API and MCP integrations
  • —Source-grounded retrieval with RAG
  • —Multi-step task execution
  • —Human approval mechanisms
  • —Security controls
  • —Agent evaluation
  • —Tracing and performance monitoring

Deliverables

  • 01A working agentic AI application
  • 02Architecture diagram and technical documentation
  • 03API and MCP integrations
  • 04Tested workflow mechanisms
  • 05Security controls and test results
  • 06Evaluation dataset
  • 07LLM and agent evaluation report
  • 08Observability and performance measurements
  • 09Final technical presentation

The goal is a working system, tested under realistic conditions with its limits documented — not a claim of full autonomy or zero errors.

Assessment

A program that teaches evaluation is transparent about its own.

Weekly deliverables
A reviewable addition to the project at the end of every practice session.
Final project
A working system, technical documentation, an evaluation dataset and an evaluation report.
Independent tests
In the final week, systems are also tested against scenarios the participant did not write.
Technical presentation
Presenting the architecture and evaluation decisions with their rationale.

Instructors and program management

Two instructors, two distinct roles.

Program lead and theory instructor

Yasemin Arslan, MSc.

Responsible for curriculum design, theory sessions, engineering methodology and the evaluation approach. Founder of LLM Turkey.

LinkedIn ↗

Practice instructor

Doğukan Veziroğlu

AI ENGINEER

Responsible for the 42 hours of live practice sessions and technical labs.

Pre-registration

Pre-register for the first cohort.

We contact pre-registered people once cohort dates and enrollment terms are confirmed. Pre-registration is free and non-binding.

No payment is taken; pre-registration is non-binding.

Cohort
To be announced
Start
To be announced
Schedule
To be announced
Class size
10–15 participants

Frequently asked questions

Can someone without software experience join?

No. The program is intermediate to advanced. You are expected to have basic software development experience, working knowledge of Git and the terminal, and familiarity with APIs and backend concepts. If you are looking for evaluation training that does not require coding, write to us.

Is this a Claude Code course?

No. Claude Code is the program's main development tool; the subject is designing, building, securing and evaluating agentic AI systems. The engineering principles are not tied to a particular model or provider.

What is the weekly time commitment?

Two live sessions a week: 2 hours of theory and 3 hours of practice, 70 hours in total. Session days and times are announced on this page when the cohort opens.

How is this different from the EvalOps Specialist program?

EvalOps Specialist focused on evaluating model outputs and did not require coding. This program also covers building agentic systems; EvalOps' nine quality dimensions and evaluation methods are integrated into its Evaluate and Operate stages.

How can I pay?

By credit card through iyzico secure payments, or by bank transfer.

What are the cancellation and refund terms?

They are set out in the Refund and Cancellation Terms and the Distance Sales Agreement, which you accept during registration.

Learn to build and measure agentic AI systems.

14 weeks, 70 live hours, and a working system you can defend at the end.