chore: Remove config option for native_iceberg_compat

[andygrove]

Which issue does this PR close?

Partially address #4020.

Rationale for this change

Removing support for native_iceberg_compat reduces the maintenance burden of the project.

This PR makes it impossible for users to select native_iceberg_compat and stops running tests for that scan impl.

Subsequent PRs will remove the implementation code.

What changes are included in this PR?

• COMET_NATIVE_SCAN_IMPL config now only allows auto or native_datafusion
• CometScanRule no longer uses the value of config option COMET_NATIVE_SCAN_IMPL and just uses native_datafusion scan
• Tests have been updated to stop testing with native_iceberg_compat
• Tests that were specific to native_iceberg_compat have been removed
• Golden files regenerated - we no longer generate different versions per scan impl
• Updated documentation

How are these changes tested?

[comphead]

Thanks @andygrove 👀

[mbutrovich]

Thanks @andygrove, let's get that CI matrix and tech debt down.

[comphead]

it is LGTM, I was thinking if we need it for some debug/benchmark usecases but nothing comes to my head. So let it go

[parthchandra]

Is there still a functionality gap between native_iceberg_compat and native_datafusion? I see some tests with IgnoreCometNativeDataFusion in the diff files.

[andygrove]

Is there still a functionality gap between native_iceberg_compat and native_datafusion? I see some tests with IgnoreCometNativeDataFusion in the diff files.

% grep "IgnoreCometNativeDataFusion(\"" dev/diffs/3.5.8.diff
+    IgnoreCometNativeDataFusion("https://github.com/apache/datafusion-comet/issues/3442")) {
+    IgnoreCometNativeDataFusion("https://github.com/apache/datafusion-comet/issues/3720")) {
+    IgnoreCometNativeDataFusion("https://github.com/apache/datafusion-comet/issues/3720")) {
+    IgnoreCometNativeDataFusion("https://github.com/apache/datafusion-comet/issues/3720")) {
+    IgnoreCometNativeDataFusion("https://github.com/apache/datafusion-comet/issues/3720")) {
+    IgnoreCometNativeDataFusion("https://github.com/apache/datafusion-comet/issues/3720")) {
+      IgnoreCometNativeDataFusion("https://github.com/apache/datafusion-comet/issues/3720")) {
+    IgnoreCometNativeDataFusion("https://github.com/apache/datafusion-comet/issues/3720")) {

So #3720 and #3442. We should be able resolve #3442 now that #4011 is merged.