Skip to content

fix(chunking): reject a sliding-window step that skips or never advances - #2538

Open
laoyyu wants to merge 1 commit into
unclecode:developfrom
laoyyu:fix/sliding-window-step-validation
Open

laoyyu wants to merge 1 commit into
unclecode:developfrom
laoyyu:fix/sliding-window-step-validation

Conversation

@laoyyu

@laoyyu laoyyu commented Nov 12, 2026

Copy link
Copy Markdown

What

SlidingWindowChunking accepted any step, which produced two failure modes:

  • step > window_size — the windows never touch, so the words in the gap belong to no chunk and are silently lost:

    text = "w0 w1 ... w11"
    SlidingWindowChunking(window_size=4, step=6).chunk(text)
    # ['w0 w1 w2 w3', 'w6 w7 w8 w9', 'w8 w9 w10 w11']  -> w4, w5 are gone
  • step <= 0 — range(..., 0) is empty, so the loop variable i is never bound and the trailing-window branch raises UnboundLocalError: cannot access local variable 'i' before returning anything.

Fix

Validate window_size and step in __init__ (window_size >= 1, 1 <= step <= window_size) and initialise the loop cursor so the trailing branch no longer depends on the loop having run.

OverlappingWindowChunking already rejects overlap >= window_size for the same reason, so this keeps the two window strategies consistent.

How it was checked

  • New tests/general/test_sliding_window_chunking.py (8 tests): every word must land in some chunk for several (window_size, step) pairs, both invalid steps raise ValueError, and text shorter than the window is returned unchanged.
  • pytest tests/general/test_sliding_window_chunking.py → 8 passed.
  • Both bugs were reproduced on the unmodified develop file before the change.

SlidingWindowChunking accepted any step. A step bigger than window_size left
the words between two windows in no chunk at all, and a non-positive step made
the loop body never run, so the trailing-window branch raised
UnboundLocalError on `i`.

Validate both arguments up front and initialise the loop cursor so the trailing
branch cannot depend on the loop running.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant