Adding english documentation on how to configure lambdas on cloudprem. by fulmicoton-dd · Pull Request #35254 · DataDog/documentation

fulmicoton-dd · 2026-03-12T18:29:14Z

What does this PR do? What is the motivation?

We recently introduced a new feature that allows to offload Logs Cloudprem work to AWS Lambda.
This feature is experimental and AWS specific.

Merge instructions

Merge readiness:

Ready for merge

[6/5/2025] Merge queue has been disabled on the documentation repo. If you have write access to the repo, the PR has been reviewed by a Documentation team member, and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #documentation channel in Slack.

AI assistance

I used to Claude code for the initial draft. It based it off our repository documentation.
https://github.com/quickwit-oss/quickwit/blob/main/docs/configuration/lambda-config.md

Copilot

Pull request overview

Adds documentation for configuring Lambda-based search offloading in CloudPrem (AWS-only) and links it from the CloudPrem “Configure” landing page.

Changes:

Introduces a new lambda.md page describing Lambda search offloading, IAM prerequisites, and Helm values configuration.
Updates the CloudPrem configure index to include a “Lambda Search Offloading” navigation entry.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
`content/en/cloudprem/configure/lambda.md`	New doc page covering feature overview, IAM role policies, and Helm configuration for Lambda search offloading.
`content/en/cloudprem/configure/_index.md`	Adds a “Lambda Search Offloading” link in the Configure section’s “what’s next” list.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-12T18:34:32Z

+- **The Lambda execution role**: the role assumed by the Lambda function itself at runtime. This role needs read access to your index data in S3. Its arn has to be set in the
+`searcher.lambda.auto_deploy.execution_role_arn` key as we will see in [Configuration](#configuration).


In this paragraph the documented config key path (searcher.lambda.auto_deploy.execution_role_arn) is inconsistent with the later Helm values example (cloudprem.searcher.lambda...). This is likely to confuse users; consider using the full Helm values path consistently (and also capitalize ARN as "ARN").

Suggested change

- **The Lambda execution role**: the role assumed by the Lambda function itself at runtime. This role needs read access to your index data in S3. Its arn has to be set in the

`searcher.lambda.auto_deploy.execution_role_arn` key as we will see in [Configuration](#configuration).

- **The Lambda execution role**: the role assumed by the Lambda function itself at runtime. This role needs read access to your index data in S3. Its ARN has to be set in the

`cloudprem.searcher.lambda.auto_deploy.execution_role_arn` key as we will see in [Configuration](#configuration).

Copilot · 2026-03-12T18:34:33Z

+      "Action": [
+        "lambda:InvokeFunction"
+      ],
+      "Resource": "arn:aws:lambda:<REGION>:<ACCOUNT_ID>:function:quickwit-lambda-search:*"


The lambda:InvokeFunction policy resource ARN ends with :*, which does not match the unqualified function ARN (no suffix). If CloudPrem ever invokes the function without specifying a version/alias (or during create/validation steps), this policy will deny the call. Consider including both the base function ARN and the :* variant (or using a wildcard that covers both).

Suggested change

"Resource": "arn:aws:lambda:<REGION>:<ACCOUNT_ID>:function:quickwit-lambda-search:*"

"Resource": [

"arn:aws:lambda:<REGION>:<ACCOUNT_ID>:function:quickwit-lambda-search",

"arn:aws:lambda:<REGION>:<ACCOUNT_ID>:function:quickwit-lambda-search:*"

]

Copilot · 2026-03-12T18:34:33Z

+
+<div class="alert alert-warning">If a lambda configuration is present, but invalid, Cloudprem's searcher will fail to start.</div>
+
+After the <link to prerequisites> iam permissions have been properly set up, to enable Lambda offloading, add a `lambda` section under `cloudprem.searcher` in your Helm chart values file:


<link to prerequisites> appears to be a placeholder and will render literally in the published docs. Replace it with an actual link (for example, to the "Prerequisite: IAM permissions" section anchor) and capitalize "IAM".

Suggested change

After the <link to prerequisites> iam permissions have been properly set up, to enable Lambda offloading, add a `lambda` section under `cloudprem.searcher` in your Helm chart values file:

After the [Prerequisite: IAM permissions](#prerequisite-iam-permissions) have been properly set up, to enable Lambda offloading, add a `lambda` section under `cloudprem.searcher` in your Helm chart values file:

Copilot · 2026-03-12T18:34:33Z

+
+## Configuration
+
+<div class="alert alert-warning">If a lambda configuration is present, but invalid, Cloudprem's searcher will fail to start.</div>


Spelling/capitalization in this warning is inconsistent with the rest of the CloudPrem docs ("lambda"/"Cloudprem" should be "Lambda"/"CloudPrem"). Consistent product/feature capitalization helps searchability and avoids looking like a typo in an alert banner.

Suggested change

<div class="alert alert-warning">If a lambda configuration is present, but invalid, Cloudprem's searcher will fail to start.</div>

<div class="alert alert-warning">If a Lambda configuration is present, but invalid, CloudPrem's searcher will fail to start.</div>

Copilot · 2026-03-12T18:34:34Z

+      auto_deploy:
+        execution_role_arn: arn:aws:iam::123456789012:role/cloudprem-lambda-role
+        memory_size: 5 GiB


The memory size value is written as 5 GiB (with a space), but elsewhere in CloudPrem docs units are written without a space (for example, 2GiB, 500MiB in content/en/cloudprem/operate/sizing.md:115-137). Aligning the unit format here would reduce ambiguity and keep documentation consistent.

github-actions · 2026-03-12T18:36:51Z

Preview links (active after the `build_preview` check completes)

New or renamed files

https://docs-staging.datadoghq.com/paul.masurel/cloudprem-lambda/cloudprem/configure/lambda

Modified Files

https://docs-staging.datadoghq.com/paul.masurel/cloudprem-lambda/cloudprem/configure/

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 46711aa132

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-03-19T09:15:19Z

+      "Action": [
+        "lambda:CreateFunction",
+        "lambda:GetFunction",
+        "lambda:UpdateFunctionCode",
+        "lambda:PublishVersion",


Add lambda:InvokeFunction to the sample node policy

This policy is presented as the minimum permission set to "invoke and deploy" the offload function, but the action list only covers create/get/update/publish/list/delete. If readers apply it as written, the startup dry-run invocation described earlier on this page and the actual overflow offloads both fail with AccessDenied because lambda:InvokeFunction is never granted.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-03-19T09:15:19Z

+## Prerequisite: IAM permissions
+
+Lambda search offloading requires specific permissions for two separate IAM roles:
+- **The CloudPrem node role**: the role attached to the Kubernetes nodes (or pod identity) running CloudPrem.  The role is defined in the `serviceAccount` section of your `values.yaml`. This role will need permissions to invoke and deploy the Lambda function.


Rename the IAM principal to the service-account role used on EKS

On the documented AWS install path, CloudPrem gets AWS credentials from the IAM role annotated onto the Kubernetes service account (content/en/cloudprem/install/aws_eks.md:193-203), not from the EC2 worker-node role. Referring to this as the "CloudPrem node role" here points readers at the wrong principal, so they can grant the Lambda permissions to the node instance profile and still see startup validation/offloading fail because CloudPrem continues to call AWS as the service-account role.

Useful? React with 👍 / 👎.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dd324fd30e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-03-19T10:33:12Z

+
+### CloudPrem node permissions
+
+The IAM role running CloudPrem needs the following permissions to invoke and deploy the Lambda function:


Scope deployment IAM rights to auto_deploy only

Later on this page, auto_deploy is documented as optional, but this section presents lambda:CreateFunction, lambda:DeleteFunction, and iam:PassRole as required for every CloudPrem runtime role. In setups that pre-create the Lambda and only set function_name, readers will over-privilege the service account unnecessarily, and in locked-down AWS accounts this makes Lambda offloading look unsupported even though invoke-only access should be sufficient.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-03-19T10:33:12Z

+        "lambda:ListVersionsByFunction",
+        "lambda:DeleteFunction"
+      ],
+      "Resource": "arn:aws:lambda:<REGION>:<ACCOUNT_ID>:function:cloudprem-*"


Allow versioned Lambda ARNs in the function resource

The startup-validation section above says CloudPrem checks the deployed function version, so the eventual InvokeFunction permission must cover qualified Lambda ARNs. AWS denies invokes when the policy only names the unqualified function ARN, which means this arn:aws:lambda:<REGION>:<ACCOUNT_ID>:function:cloudprem-* pattern is too narrow for the dry-run and real offloads once InvokeFunction is granted; it needs either :* for versions or a broader wildcard.

Useful? React with 👍 / 👎.

estherk15

Thanks for drafting this, left a few suggestions for formatting. Let me know if you have any questions!

Co-authored-by: Esther Kim <esther.kim@datadoghq.com>

fulmicoton-dd requested a review from Copilot March 12, 2026 18:29

Copilot started reviewing on behalf of fulmicoton-dd March 12, 2026 18:31 View session

fulmicoton-dd force-pushed the paul.masurel/cloudprem-lambda branch from f71ed05 to adb1f15 Compare March 12, 2026 18:32

Copilot AI reviewed Mar 12, 2026

View reviewed changes

fulmicoton-dd force-pushed the paul.masurel/cloudprem-lambda branch 5 times, most recently from 37f1aa2 to e957dd3 Compare March 12, 2026 20:48

fulmicoton-dd marked this pull request as ready for review March 19, 2026 09:08

fulmicoton-dd requested a review from a team as a code owner March 19, 2026 09:08

chatgpt-codex-connector bot reviewed Mar 19, 2026

View reviewed changes

fulmicoton-dd added 2 commits March 19, 2026 11:21

Adding english documentation on how to configure lambdas on cloudprem.

963ba25

fixing link

dd324fd

Platane force-pushed the paul.masurel/cloudprem-lambda branch from 46711aa to dd324fd Compare March 19, 2026 10:21

chatgpt-codex-connector bot reviewed Mar 19, 2026

View reviewed changes

estherk15 self-assigned this Mar 19, 2026

estherk15 added the editorial review Waiting on a more in-depth review label Mar 19, 2026

estherk15 reviewed Mar 23, 2026

View reviewed changes

fulmicoton-dd and others added 2 commits March 23, 2026 14:41

Apply suggestions from code review

5808234

Co-authored-by: Esther Kim <esther.kim@datadoghq.com>

Apply suggestions from code review

c5e5093

Co-authored-by: Esther Kim <esther.kim@datadoghq.com>

estherk15 approved these changes Mar 23, 2026

View reviewed changes

Comment thread content/en/cloudprem/configure/lambda.md Outdated

estherk15 added 2 commits March 23, 2026 15:17

Update content/en/cloudprem/configure/lambda.md

e7acb1a

Merge branch 'master' into paul.masurel/cloudprem-lambda

0f0072f

estherk15 merged commit 958689e into master Mar 23, 2026
16 checks passed

estherk15 deleted the paul.masurel/cloudprem-lambda branch March 23, 2026 20:00

		- The Lambda execution role: the role assumed by the Lambda function itself at runtime. This role needs read access to your index data in S3. Its arn has to be set in the
		`searcher.lambda.auto_deploy.execution_role_arn` key as we will see in [Configuration](#configuration).


		<div class="alert alert-warning">If a lambda configuration is present, but invalid, Cloudprem's searcher will fail to start.</div>

		After the <link to prerequisites> iam permissions have been properly set up, to enable Lambda offloading, add a `lambda` section under `cloudprem.searcher` in your Helm chart values file:

	After the <link to prerequisites> iam permissions have been properly set up, to enable Lambda offloading, add a `lambda` section under `cloudprem.searcher` in your Helm chart values file:
	After the [Prerequisite: IAM permissions](#prerequisite-iam-permissions) have been properly set up, to enable Lambda offloading, add a `lambda` section under `cloudprem.searcher` in your Helm chart values file:


		## Configuration

		<div class="alert alert-warning">If a lambda configuration is present, but invalid, Cloudprem's searcher will fail to start.</div>


		### CloudPrem node permissions

		The IAM role running CloudPrem needs the following permissions to invoke and deploy the Lambda function:

Conversation

fulmicoton-dd commented Mar 12, 2026 • edited by Platane Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do? What is the motivation?

Merge instructions

AI assistance

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 12, 2026

Preview links (active after the build_preview check completes)

New or renamed files

Modified Files

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

estherk15 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fulmicoton-dd commented Mar 12, 2026 •

edited by Platane

Loading

Preview links (active after the `build_preview` check completes)