Skip to content

llama-eval : add AIME 2026 dataset support - #23058

Merged
ggerganov merged 1 commit into
masterfrom
gg/llama-eval-aime2026
May 15, 2026
Merged

ggerganov merged 1 commit into
masterfrom
gg/llama-eval-aime2026

Conversation

@ggerganov

@ggerganov ggerganov commented May 14, 2026 •

Copy link
Copy Markdown
Member

Overview

cont #21152

Add Aime2026Dataset class loading from MathArena/aime_2026 on HuggingFace. 30 problems (two sets of 15), single config/split. Answers are already int64 so no normalization needed.

Usage: --dataset aime2026

Changes:

  • New Aime2026Dataset class (loads MathArena/aime_2026, config default, split train)
  • Added aime2026 to GRADER_PATTERNS, SAMPLE_ANSWERS, TEMPLATE_REGISTRY
  • Added aime2026 branch in EvalState.load_dataset()
  • Added aime2026 to --dataset CLI choices
image

Requirements

Add Aime2026Dataset class loading from MathArena/aime_2026 on
HuggingFace. 30 problems (two sets of 15), single config/split.

Usage: --dataset aime2026

Assisted-by: llama.cpp:local pi
@github-actions github-actions Bot added examples python python script changes labels May 14, 2026
@ggerganov
ggerganov marked this pull request as ready for review May 14, 2026 20:01
@ggerganov
ggerganov merged commit d5dc2e0 into master May 15, 2026
6 checks passed
@ggerganov
ggerganov deleted the gg/llama-eval-aime2026 branch May 15, 2026 10:58
baramofme pushed a commit to baramofme/llama-cpp-turboquant that referenced this pull request May 23, 2026
Add Aime2026Dataset class loading from MathArena/aime_2026 on
HuggingFace. 30 problems (two sets of 15), single config/split.

Usage: --dataset aime2026

Assisted-by: llama.cpp:local pi
carlosfundora pushed a commit to carlosfundora/llama.cpp-1-bit-turbo that referenced this pull request May 24, 2026
Add Aime2026Dataset class loading from MathArena/aime_2026 on
HuggingFace. 30 problems (two sets of 15), single config/split.

Usage: --dataset aime2026

Assisted-by: llama.cpp:local pi
(cherry picked from commit d5dc2e0)
zommiommy pushed a commit to zommiommy/llama.cpp that referenced this pull request Aug 18, 2026
Add Aime2026Dataset class loading from MathArena/aime_2026 on
HuggingFace. 30 problems (two sets of 15), single config/split.

Usage: --dataset aime2026

Assisted-by: llama.cpp:local pi
zbrad pushed a commit to zbrad/llama.cpp that referenced this pull request Sep 10, 2026
Add Aime2026Dataset class loading from MathArena/aime_2026 on
HuggingFace. 30 problems (two sets of 15), single config/split.

Usage: --dataset aime2026

Assisted-by: llama.cpp:local pi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

examples python python script changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant