(feat): Order-aware compile-time escape detection#38
Conversation
Add conservative compile-time detection for pool arrays escaping via container field access (e.g., `vac = (wv=zeros!(pool,...)); return vac.wv`). Emits @warn (not error) since false positives are possible — macro-level AST analysis cannot distinguish pool-backed vs non-pool fields in a container. Users can rely on RUNTIME_CHECK=1 for precise detection. New helpers: - _tuple_contains_acquire_call: detects acquire calls inside tuple literals - _extract_container_vars: finds vars assigned from tuples-of-acquires - _find_dot_access_exposure: detects container.field in return expressions - _warn_compile_time_container_escape: orchestrator, called from all 4 macro paths
Include N-D dimensionality, wrapper dims tuple, and backing vector length in @warn messages for easier debugging. Update test regex patterns to match new message format.
When a pool slot is reused with a different dimensionality (e.g., 1D after 2D), the old wrapper retains a stale MemoryRef with dims=(0,0). _check_wrapper_mutation! was comparing this stale pointer against the resized vector, producing a false-positive "reallocation detected" warning on every warmup cycle. Fix: skip wrappers with prod(dims)==0 — these were already invalidated by a previous rewind and are not in active use.
…or style Reformat container-escape warning to show: - Variable summary with origin annotation - Declaration site with line number - Escaping return expression with line number - Fix suggestion (collect() or restructure) - False positive note with RUNTIME_CHECK pointer
- Add [1], [2] numbered indices for declarations and returns - Add file:line location in brackets (reuse _format_location_str) - Remove "Fix:" section (too prescriptive for a warning) - Soften "False positive?" to "Note:" with gentler wording
Show actionable command for enabling RUNTIME_CHECK instead of just mentioning the setting name. Users can copy-paste directly.
Bug fix: the escape detector used two separate order-insensitive passes (_extract_acquired_vars + _remove_flat_reassigned!) which caused false negatives when a non-pool assignment appeared before an acquire! call: v = 1; v = acquire!(pool, ...); return v ← was silently passing New _extract_ordered_acquired() combines both into a single forward pass that tracks taint per statement order. Handles atomic tuple destructuring (e.g., swap patterns v,x = x,v). New PoolReassignEscapeWarning for ambiguous patterns like v = f(v) where f might return the same pool-backed array. Excludes known-safe functions (collect, copy, deepcopy) and broadcast calls (.+, f.(v)). Also fix 3 pre-existing test_state.jl runtime escape failures caused by helper functions returning pool-backed arrays as implicit block returns.
…ives from inner scopes _extract_acquired_vars now skips scope-introducing expressions (let, function, ->, macro) during recursion. Variables inside these are in a different scope and must not taint outer variables. Previously, `v = 1; tmp = let v = acquire!(pool,...); sum(v) end; return v` would false-positive because the inner `v` polluted the outer taint set. Added 6 test cases covering let blocks, lambdas, nested functions, and do blocks — all verify no false positives from inner-scope shadowing.
… warning - Remove dead _remove_flat_reassigned! (replaced by _extract_ordered_acquired) - Add PoolReassignEscapeWarning output test (stderr capture verification) - Add unit tests for _is_safe_copy_call, _rhs_call_contains_sym, _find_reassign_maybe_tainted (including tuple destructuring) - Add tuple destructuring to _find_reassign_maybe_tainted (tuple-literal RHS only; opaque destructuring treated as transform, not identity) - Add similar to _SAFE_COPY_FUNCTIONS (always returns new array)
…warnings _extract_container_vars now uses a forward pass for top-level block statements, removing container taint when a variable is reassigned to a non-acquire value. Previously, `vac = (wv=acquire!(pool,...),); vac = (name="ok",); vac.name` would false-positive warn because _extract_container_vars! only accumulated symbols and never removed them on reassignment (same pattern as the original _remove_flat_reassigned! bug). Also added scope-filtering (let/function/->/macro) to _extract_container_vars! for consistency with _extract_acquired_vars.
After invalidation (zeroing dims), wrapper slots are now set to nothing. This lets _check_wrapper_mutation! use the existing `wrapper === nothing` check to skip stale wrappers, instead of the _wrapper_prod_size == 0 heuristic that also skipped legitimately zero-sized arrays. Previously, resize!(v, 10000); resize!(v, 0) would cause a pointer change that went undetected because the size-0 check skipped the wrapper entirely.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #38 +/- ##
==========================================
- Coverage 96.65% 96.23% -0.43%
==========================================
Files 14 14
Lines 2753 3107 +354
==========================================
+ Hits 2661 2990 +329
- Misses 92 117 +25
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR updates AdaptiveArrayPools’ compile-time escape detection to be order-aware (single forward taint pass), adds new compile-time warnings for ambiguous reassignment and container-field escapes, and tightens runtime structural mutation diagnostics (including improved warning messages and a rewind-time mutation edge case).
Changes:
- Replace order-insensitive taint tracking with an order-aware forward pass (
_extract_ordered_acquired) used by compile-time escape and mutation checks. - Add compile-time warnings for (1) potential container dot-access escapes and (2) ambiguous
v = f(v)reassignment escapes, with supporting unit tests. - Update runtime wrapper-mutation warning messages and adjust tests accordingly; tweak rewind invalidation behavior for wrapper bookkeeping.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
src/macros.jl |
Implements order-aware taint extraction and adds new compile-time warning passes + helpers. |
src/state.jl |
Changes typed-pool wrapper invalidation behavior during rewind. |
src/debug.jl |
Refines structural mutation warning messages for TypedPool Array wrappers. |
test/test_compile_escape.jl |
Adds new test coverage for ordering, scope shadowing, reassignment warning, and container warning behavior. |
test/test_runtime_mutation.jl |
Updates log-message expectations to match new runtime warning strings. |
test/test_state.jl |
Prevents pool-backed implicit returns in a few scenarios (test stabilization under stricter analysis). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…d arrays" This reverts commit 345e5b5.
- Remove :let from scope skip list: `return` inside `let` exits the enclosing function, so pool variables acquired there can escape. (function/->/ macro still skipped — their `return` is local) - Remove Array/Vector/Matrix from _SAFE_COPY_FUNCTIONS: these can return the same object via convert (e.g., Vector(v::Vector) === v) - Deduplicate _capture_stderr test helper (define once, reuse)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…/Matrix due to unpredictable behavior
…ing output path Exercises _find_dot_access_exposure, _collect_dot_access_syms, _find_container_declaration, _find_line_for_expr, and _contains_expr which were previously unreachable (only negative tests existed).
_generate_pool_code_with_backend (CUDA/Metal block form) was missing _warn_compile_time_container_escape and _warn_compile_time_reassign_escape. Both force_enable and !force_enable paths now have full parity with CPU.
Summary
Rewrites the compile-time escape detector's taint tracking from two separate order-insensitive passes to a single forward pass. Fixes false negatives, false positives, and adds a new warning for ambiguous reassignment patterns.
What the detector now catches
Escape through pre-acquire assignment (was missed)
The prior two-pass approach saw
v = 1and removedvfrom the tainted set regardless of statement order. Now a single forward pass tracks taint per statement position.Escape through alias with pre-assignment (was missed)
Ambiguous reassignment
v = f(v)(new warning)Known-safe functions are excluded (no warning):
collect,copy,deepcopy,similar.+,.*,f.(v)What the detector no longer falsely flags
Inner-scope shadowing (was false positive)
function,->,macroblocks create new variable scopes. The detector now skips these during recursion. (letis NOT skipped —returninsideletexits the enclosing function, so pool variables there can escape.)Reassigned containers (was false positive warning)
Swap patterns (still correctly safe)
Other improvements
_remove_flat_reassigned!(replaced by_extract_ordered_acquired)PoolContainerEscapeWarningoutput path