Go: Add overlay annotations from script by owen-mc · Pull Request #21370 · github/codeql

owen-mc · 2026-02-25T09:58:51Z

#20623 enabled basic overlay support for Go. Performance testing showed that this led to a significant speedup for larger projects, but for smaller projects it was causing a small slowdown. This PR adds some overlay annotations to the CodeQL Go library by running the script config/add-overlay-annotations.py. This increases the number of calculations that do not have to be redone. Performance testing shows that this is a significant improvement. Not all projects have a speedup, but even when limiting to small repos (which I'm defining as <100s analysis time) we see a speedup on average.

Copilot

Pull request overview

This PR expands Go’s overlay support by adding overlay annotations and explicit module; declarations across the Go CodeQL library, aiming to reduce recomputation and improve performance (especially on smaller projects) under overlay analysis.

Changes:

Add overlay[local?] + module; declarations to many Go library .qll files so they participate in overlay-aware compilation/caching.
Add overlay[caller?] annotations to selected inline/local flow & taint helper predicates to enable inlining across the overlay frontier.
Apply the same overlay/module pattern to Go inline-expectations test implementation library.

Reviewed changes

Copilot reviewed 188 out of 188 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
go/ql/lib/utils/test/internal/InlineExpectationsTestImpl.qll	Add overlay/module header for overlay-aware test library compilation.
go/ql/lib/semmle/go/security/ZipSlipCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/ZipSlip.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/Xss.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/XPathInjectionCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/XPathInjection.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/WeakSensitiveDataHashingCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/UrlConcatenation.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/UnsafeUnzipSymlinkCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/UnsafeUnzipSymlink.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/UncontrolledAllocationSizeCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/UncontrolledAllocationSize.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/TaintedPathCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/TaintedPath.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/StringBreakCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/StringBreak.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/StoredXssCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/StoredXss.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/StoredCommand.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/SqlInjectionCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/SqlInjection.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/SensitiveActions.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/Sanitizers.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/SafeUrlFlowCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/SafeUrlFlow.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/RequestForgeryCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/RequestForgery.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/ReflectedXssCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/ReflectedXss.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/OpenUrlRedirectCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/OpenUrlRedirect.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/MissingJwtSignatureCheckCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/MissingJwtSignatureCheck.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/LogInjectionCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/LogInjection.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/InsecureRandomnessCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/InsecureRandomness.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/InsecureFeatureFlag.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/IncorrectIntegerConversionLib.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/HardcodedCredentials.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/FlowSources.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/ExternalAPIs.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/CookieWithoutSecure.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/CookieWithoutHttpOnly.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/CommandInjectionCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/CommandInjection.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/CleartextLoggingCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/CleartextLogging.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/AllocationSizeOverflowCustomizations.qll	Add overlay/module header.
go/ql/lib/semmle/go/security/AllocationSizeOverflow.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/Unsafe.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/TextTemplate.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/TextTabwriter.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/Syscall.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/Strings.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/Strconv.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/Regexp.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/Reflect.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/PathFilepath.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/Path.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/Os.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/NetTextproto.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/NetHttpHttputil.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/NetHttp.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/Net.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/MimeQuotedprintable.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/MimeMultipart.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/Log.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/IoFs.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/Io.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/HtmlTemplate.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/Html.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/Fmt.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/Errors.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/EncodingXml.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/EncodingPem.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/EncodingJson.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/EncodingGob.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/EncodingCsv.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/EncodingAsn1.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/DatabaseSql.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/CryptoTls.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/CompressZlib.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/CompressLzw.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/CompressGzip.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/CompressFlate.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/Bufio.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/ArchiveZip.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/stdlib/ArchiveTar.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Zap.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Yaml.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/XPath.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/XNetHtml.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/WebSocket.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Twirp.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Testing.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/SystemCommandExecutors.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Stdlib.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Squirrel.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Spew.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/SQL.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/RsCors.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Revel.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Protobuf.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/NoSQL.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Mux.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Macaron.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Logrus.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/K8sIoClientGo.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/K8sIoApimachineryPkgRuntime.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/K8sIoApiCoreV1.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Jwt.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Gqlgen.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Gorqlite.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/GoMicro.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/GoKit.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/GoJose.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Glog.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/GinCors.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Gin.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Fasthttp.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Encoding.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Email.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/ElazarlGoproxy.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Echo.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/CryptoLibraries.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Couchbase.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Bun.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/BeegoOrm.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Beego.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/AwsLambda.qll	Add overlay/module header.
go/ql/lib/semmle/go/frameworks/Afero.qll	Add overlay/module header.
go/ql/lib/semmle/go/dependencies/SemVer.qll	Add overlay/module header.
go/ql/lib/semmle/go/dependencies/Dependencies.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/internal/TaintTrackingUtil.qll	Add overlay/module header; mark selected inline predicates `overlay[caller?]`.
go/ql/lib/semmle/go/dataflow/internal/TaintTrackingImplSpecific.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/internal/FlowSummaryImpl.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/internal/ExternalFlowExtensions.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/internal/DataFlowUtil.qll	Add overlay/module header; mark local flow predicate `overlay[caller?]`.
go/ql/lib/semmle/go/dataflow/internal/DataFlowPrivate.qll	Add overlay/module header; mark inline helper `overlay[caller?]`.
go/ql/lib/semmle/go/dataflow/internal/DataFlowNodes.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/internal/DataFlowImplSpecific.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/internal/DataFlowImplConsistency.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/internal/DataFlowImplCommon.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/internal/DataFlowImpl.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/internal/DataFlowDispatch.qll	Add overlay/module header; mark parameter/arg helpers `overlay[caller?]`.
go/ql/lib/semmle/go/dataflow/internal/ContainerFlow.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/barrierguardutil/UrlCheck.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/barrierguardutil/RegexpCheck.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/barrierguardutil/RedirectCheckBarrierGuard.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/TaintTracking.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/SsaImpl.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/SSA.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/Properties.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/GlobalValueNumbering.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/FunctionInputsAndOutputs.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/FlowSummary.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/ExternalFlow.qll	Add overlay/module header.
go/ql/lib/semmle/go/dataflow/DataFlow.qll	Add overlay/module header.
go/ql/lib/semmle/go/controlflow/IR.qll	Add overlay/module header.
go/ql/lib/semmle/go/controlflow/ControlFlowGraphImpl.qll	Add overlay/module header.
go/ql/lib/semmle/go/controlflow/ControlFlowGraph.qll	Add overlay/module header; mark dominance helper `overlay[caller?]`.
go/ql/lib/semmle/go/controlflow/BasicBlocks.qll	Add overlay/module header.
go/ql/lib/semmle/go/concepts/HTTP.qll	Add overlay/module header.
go/ql/lib/semmle/go/concepts/GeneratedFile.qll	Add overlay/module header.
go/ql/lib/semmle/go/VariableWithFields.qll	Add overlay/module header.
go/ql/lib/semmle/go/Util.qll	Add overlay/module header.
go/ql/lib/semmle/go/Types.qll	Add overlay/module header.
go/ql/lib/semmle/go/StringOps.qll	Add overlay/module header.
go/ql/lib/semmle/go/Stmt.qll	Add overlay/module header.
go/ql/lib/semmle/go/Scopes.qll	Add overlay/module header.
go/ql/lib/semmle/go/PrintAst.qll	Add overlay/module header.
go/ql/lib/semmle/go/Packages.qll	Add overlay/module header.
go/ql/lib/semmle/go/Locations.qll	Add overlay/module header.
go/ql/lib/semmle/go/HTML.qll	Add overlay/module header.
go/ql/lib/semmle/go/GoMod.qll	Add overlay/module header.
go/ql/lib/semmle/go/Files.qll	Add overlay/module header.
go/ql/lib/semmle/go/Expr.qll	Add overlay/module header.
go/ql/lib/semmle/go/Errors.qll	Add overlay/module header.
go/ql/lib/semmle/go/DiagnosticsReporting.qll	Add overlay/module header.
go/ql/lib/semmle/go/Decls.qll	Add overlay/module header.
go/ql/lib/semmle/go/Concepts.qll	Add overlay/module header.
go/ql/lib/semmle/go/Comments.qll	Add overlay/module header.
go/ql/lib/semmle/go/Architectures.qll	Add overlay/module header.
go/ql/lib/semmle/go/AST.qll	Add overlay/module header.
go/ql/lib/ideContextual.qll	Add overlay/module header.
go/ql/lib/go.qll	Add overlay/module header.
go/ql/lib/Customizations.qll	Add overlay/module header.

owen-mc · 2026-02-25T10:24:33Z

I've had to rebase on main to get the tests to run. The overlay DCA runs were done on older commits. I don't think this affects their validity, as I don't think anything relating to this has changed recently.

owen-mc · 2026-02-25T22:34:04Z

Ah, running DCA (not in overlay mode) highlighted that maybe these annotations has caused a performance regression. Let me investigate that before this gets reviewed. I'll move it into draft.

We should be using all subtypes of `FieldBase`. This allows us to find more type expressions, and is also simpler to evaluate.

The Go libraries follow their own naming convention for "query libraries". These need to be exempted from automatic `overlay[local?]` annotations since otherwise it appears that too many predicates are evaluated, possibly because of inadequate use of sentinels.

This commit is auto-generated with: python3 config/add-overlay-annotations.py go

Copilot AI review requested due to automatic review settings February 25, 2026 09:58

owen-mc requested a review from a team as a code owner February 25, 2026 09:58

github-actions bot added the Go label Feb 25, 2026

Copilot started reviewing on behalf of owen-mc February 25, 2026 09:59 View session

owen-mc added the no-change-note-required This PR does not need a change note label Feb 25, 2026

Copilot AI reviewed Feb 25, 2026

View reviewed changes

owen-mc force-pushed the owen-mc/go/overlay-annotations branch from fe0db02 to 369e122 Compare February 25, 2026 10:23

owen-mc force-pushed the owen-mc/go/overlay-annotations branch from 369e122 to fcfc24e Compare February 25, 2026 13:47

owen-mc marked this pull request as draft February 25, 2026 22:34

owen-mc force-pushed the owen-mc/go/overlay-annotations branch from fcfc24e to 4dff589 Compare March 10, 2026 10:38

owen-mc and others added 9 commits March 11, 2026 12:34

Add pragmas to improve performance

955c20d

Add and use exprRefersToNil predicate

8c67dee

Expand isTypeExprTopDown

72bd958

We should be using all subtypes of `FieldBase`. This allows us to find more type expressions, and is also simpler to evaluate.

Improve QLDoc for isTypeExprTopDown

13675c4

Add missing QLDoc

2be73c0

Use "database" instead of "snapshot" in QLDocs

ab54b54

Go: annotate the standard library with for overlay

d6d4cb8

This commit is auto-generated with: python3 config/add-overlay-annotations.py go

Mark various files as overlay[local]

9faada0

owen-mc force-pushed the owen-mc/go/overlay-annotations branch from 4dff589 to 9faada0 Compare March 11, 2026 14:32

Add overlay annotations in 4 PrintAst query files

4e0a6ac

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Go: Add overlay annotations from script#21370

Go: Add overlay annotations from script#21370
owen-mc wants to merge 10 commits intomainfrom
owen-mc/go/overlay-annotations

owen-mc commented Feb 25, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

owen-mc commented Feb 25, 2026

Uh oh!

owen-mc commented Feb 25, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

owen-mc commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

owen-mc commented Feb 25, 2026

Uh oh!

owen-mc commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

owen-mc commented Feb 25, 2026 •

edited

Loading

owen-mc commented Feb 25, 2026 •

edited

Loading