Skip to content

(feat): Order-aware compile-time escape detection#38

Merged
mgyoo86 merged 21 commits intomasterfrom
feat/compile_warning
Mar 27, 2026
Merged

(feat): Order-aware compile-time escape detection#38
mgyoo86 merged 21 commits intomasterfrom
feat/compile_warning

Conversation

@mgyoo86
Copy link
Copy Markdown
Member

@mgyoo86 mgyoo86 commented Mar 26, 2026

Summary

Rewrites the compile-time escape detector's taint tracking from two separate order-insensitive passes to a single forward pass. Fixes false negatives, false positives, and adds a new warning for ambiguous reassignment patterns.

What the detector now catches

Escape through pre-acquire assignment (was missed)

@with_pool pool function test()
    v = 1
    v = acquire!(pool, Float64, 3, 3)
    return (v,)  # PoolEscapeError ← previously passed silently
end

The prior two-pass approach saw v = 1 and removed v from the tainted set regardless of statement order. Now a single forward pass tracks taint per statement position.

Escape through alias with pre-assignment (was missed)

@with_pool pool function test()
    v = acquire!(pool, Float64, 3, 3)
    v2 = 1
    v2 = v
    return [v2]  # PoolEscapeError ← previously passed silently
end

Ambiguous reassignment v = f(v) (new warning)

@with_pool pool function test()
    v = acquire!(pool, Float64, 10)
    v = f(v)       # f might return same array (identity, reshape)
    return v       # PoolReassignEscapeWarning (not error)
end

Known-safe functions are excluded (no warning):

  • collect, copy, deepcopy, similar
  • Broadcast: .+, .*, f.(v)

What the detector no longer falsely flags

Inner-scope shadowing (was false positive)

@with_pool pool begin
    v = "hello"
    function helper()
        v = acquire!(pool, Float64, 5)
        sum(v)
    end
    helper()
    return v  # safe (outer v = "hello") ← previously threw PoolEscapeError
end

function, ->, macro blocks create new variable scopes. The detector now skips these during recursion. (let is NOT skipped — return inside let exits the enclosing function, so pool variables there can escape.)

Reassigned containers (was false positive warning)

@with_pool pool begin
    vac = (wv = acquire!(pool, Float64, 3),)
    vac = (name = "ok",)
    return vac.name  # safe ← previously warned PoolContainerEscapeWarning
end

Swap patterns (still correctly safe)

@with_pool pool begin
    v = acquire!(pool, Float64, 10)
    x = zeros(10)
    v, x = x, v    # atomic destructuring: v gets zeros, x gets pool array
    return v        # safe (v holds zeros) ← correctly no error
end

Other improvements

  • Container and reassignment warnings now fire for CUDA/Metal block forms (was CPU-only)
  • Removed dead _remove_flat_reassigned! (replaced by _extract_ordered_acquired)
  • Pluralized "Escaping return(s):" in warning output
  • Added positive test for PoolContainerEscapeWarning output path

mgyoo86 added 14 commits March 26, 2026 11:06
Add conservative compile-time detection for pool arrays escaping via
container field access (e.g., `vac = (wv=zeros!(pool,...)); return vac.wv`).

Emits @warn (not error) since false positives are possible — macro-level
AST analysis cannot distinguish pool-backed vs non-pool fields in a
container. Users can rely on RUNTIME_CHECK=1 for precise detection.

New helpers:
- _tuple_contains_acquire_call: detects acquire calls inside tuple literals
- _extract_container_vars: finds vars assigned from tuples-of-acquires
- _find_dot_access_exposure: detects container.field in return expressions
- _warn_compile_time_container_escape: orchestrator, called from all 4 macro paths
Include N-D dimensionality, wrapper dims tuple, and backing vector
length in @warn messages for easier debugging. Update test regex
patterns to match new message format.
When a pool slot is reused with a different dimensionality (e.g., 1D
after 2D), the old wrapper retains a stale MemoryRef with dims=(0,0).
_check_wrapper_mutation! was comparing this stale pointer against the
resized vector, producing a false-positive "reallocation detected"
warning on every warmup cycle.

Fix: skip wrappers with prod(dims)==0 — these were already invalidated
by a previous rewind and are not in active use.
…or style

Reformat container-escape warning to show:
- Variable summary with origin annotation
- Declaration site with line number
- Escaping return expression with line number
- Fix suggestion (collect() or restructure)
- False positive note with RUNTIME_CHECK pointer
Replace @warn with direct printstyled to stderr, matching PoolEscapeError's
color scheme: yellow=variable, cyan=declaration, magenta=return, bold=fix.
@warn renders everything in flat yellow which makes structure hard to read.
- Add [1], [2] numbered indices for declarations and returns
- Add file:line location in brackets (reuse _format_location_str)
- Remove "Fix:" section (too prescriptive for a warning)
- Soften "False positive?" to "Note:" with gentler wording
Show actionable command for enabling RUNTIME_CHECK instead of just
mentioning the setting name. Users can copy-paste directly.
Bug fix: the escape detector used two separate order-insensitive passes
(_extract_acquired_vars + _remove_flat_reassigned!) which caused false
negatives when a non-pool assignment appeared before an acquire! call:
  v = 1; v = acquire!(pool, ...); return v  ← was silently passing

New _extract_ordered_acquired() combines both into a single forward pass
that tracks taint per statement order. Handles atomic tuple destructuring
(e.g., swap patterns v,x = x,v).

New PoolReassignEscapeWarning for ambiguous patterns like v = f(v) where
f might return the same pool-backed array. Excludes known-safe functions
(collect, copy, deepcopy) and broadcast calls (.+, f.(v)).

Also fix 3 pre-existing test_state.jl runtime escape failures caused by
helper functions returning pool-backed arrays as implicit block returns.
…ives from inner scopes

_extract_acquired_vars now skips scope-introducing expressions (let, function,
->, macro) during recursion. Variables inside these are in a different scope
and must not taint outer variables.

Previously, `v = 1; tmp = let v = acquire!(pool,...); sum(v) end; return v`
would false-positive because the inner `v` polluted the outer taint set.

Added 6 test cases covering let blocks, lambdas, nested functions, and
do blocks — all verify no false positives from inner-scope shadowing.
… warning

- Remove dead _remove_flat_reassigned! (replaced by _extract_ordered_acquired)
- Add PoolReassignEscapeWarning output test (stderr capture verification)
- Add unit tests for _is_safe_copy_call, _rhs_call_contains_sym,
  _find_reassign_maybe_tainted (including tuple destructuring)
- Add tuple destructuring to _find_reassign_maybe_tainted (tuple-literal
  RHS only; opaque destructuring treated as transform, not identity)
- Add similar to _SAFE_COPY_FUNCTIONS (always returns new array)
…warnings

_extract_container_vars now uses a forward pass for top-level block statements,
removing container taint when a variable is reassigned to a non-acquire value.

Previously, `vac = (wv=acquire!(pool,...),); vac = (name="ok",); vac.name`
would false-positive warn because _extract_container_vars! only accumulated
symbols and never removed them on reassignment (same pattern as the original
_remove_flat_reassigned! bug).

Also added scope-filtering (let/function/->/macro) to _extract_container_vars!
for consistency with _extract_acquired_vars.
After invalidation (zeroing dims), wrapper slots are now set to nothing.
This lets _check_wrapper_mutation! use the existing `wrapper === nothing`
check to skip stale wrappers, instead of the _wrapper_prod_size == 0
heuristic that also skipped legitimately zero-sized arrays.

Previously, resize!(v, 10000); resize!(v, 0) would cause a pointer change
that went undetected because the size-0 check skipped the wrapper entirely.
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 26, 2026

Codecov Report

❌ Patch coverage is 93.81720% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.23%. Comparing base (eccb87c) to head (afff7f3).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
src/macros.jl 93.73% 23 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #38      +/-   ##
==========================================
- Coverage   96.65%   96.23%   -0.43%     
==========================================
  Files          14       14              
  Lines        2753     3107     +354     
==========================================
+ Hits         2661     2990     +329     
- Misses         92      117      +25     
Files with missing lines Coverage Δ
src/debug.jl 95.85% <100.00%> (+0.05%) ⬆️
src/macros.jl 96.70% <93.73%> (-1.29%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates AdaptiveArrayPools’ compile-time escape detection to be order-aware (single forward taint pass), adds new compile-time warnings for ambiguous reassignment and container-field escapes, and tightens runtime structural mutation diagnostics (including improved warning messages and a rewind-time mutation edge case).

Changes:

  • Replace order-insensitive taint tracking with an order-aware forward pass (_extract_ordered_acquired) used by compile-time escape and mutation checks.
  • Add compile-time warnings for (1) potential container dot-access escapes and (2) ambiguous v = f(v) reassignment escapes, with supporting unit tests.
  • Update runtime wrapper-mutation warning messages and adjust tests accordingly; tweak rewind invalidation behavior for wrapper bookkeeping.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
src/macros.jl Implements order-aware taint extraction and adds new compile-time warning passes + helpers.
src/state.jl Changes typed-pool wrapper invalidation behavior during rewind.
src/debug.jl Refines structural mutation warning messages for TypedPool Array wrappers.
test/test_compile_escape.jl Adds new test coverage for ordering, scope shadowing, reassignment warning, and container warning behavior.
test/test_runtime_mutation.jl Updates log-message expectations to match new runtime warning strings.
test/test_state.jl Prevents pool-backed implicit returns in a few scenarios (test stabilization under stricter analysis).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mgyoo86 and others added 7 commits March 26, 2026 15:54
- Remove :let from scope skip list: `return` inside `let` exits the
  enclosing function, so pool variables acquired there can escape.
  (function/->/ macro still skipped — their `return` is local)
- Remove Array/Vector/Matrix from _SAFE_COPY_FUNCTIONS: these can
  return the same object via convert (e.g., Vector(v::Vector) === v)
- Deduplicate _capture_stderr test helper (define once, reuse)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…ing output path

Exercises _find_dot_access_exposure, _collect_dot_access_syms,
_find_container_declaration, _find_line_for_expr, and _contains_expr
which were previously unreachable (only negative tests existed).
_generate_pool_code_with_backend (CUDA/Metal block form) was missing
_warn_compile_time_container_escape and _warn_compile_time_reassign_escape.
Both force_enable and !force_enable paths now have full parity with CPU.
@mgyoo86 mgyoo86 merged commit 0b5a35c into master Mar 27, 2026
13 of 14 checks passed
@mgyoo86 mgyoo86 deleted the feat/compile_warning branch March 27, 2026 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants