HDDS-14908. Container scanner should check for updates before persisting merkle tree

[errose28]

What changes were proposed in this pull request?

Currently there is no coordination between reconciliation and container scanner by design since they are both long running background processes. As part of this design, there was a known corner case that could cause the checksum to deviate before eventually converging if scanner and reconciliation ran at the same time:

The container scanner builds the merkle tree in memory as it is running, and then flushes it to disk when it finishes scanning the container. If the container is updated (via reconciliation) while the scanner is running, the scanner may not see these updates and as a result persist a stale tree at the end of its scan. A subsequent container scan would then see the fully updated container contents and write the correct tree, restoring the system to a stable state.

This change prevents this case by having the container scanner check the data checksum when it starts, and then again before it persists the tree. If the checksum has changed, the scanner knows its work is now invalid and it must rescan the container to build a new tree.

A maximum number of retries per container is added so that the scanner does not get stuck in a loop on a single container and fail to process others. When the max retries is hit, the scan will be skipped without any affect on the container and its last scan timestamp will not be updated. This will cause the container to be retried first after the whole container set is finished scanning and the next round of scanning begins.

Changes generated by Cursor with review and modifications by me.

What is the link to the Apache JIRA

HDDS-14908

How was this patch tested?

Unit tests added for the following cases:

• Retry with eventual success.
• Retry without success requiring the scan to be skipped.
• Container iterator correctly orders containers when some timestamps are not updated during processing, causing skipped containers to be tried first in the next iteration.

This change does not apply to the metadata scanner, which does not read or write container data checksums.