[PyRSA]

Purpose

Fixes #10711

S3FileIO may continue reusing a cached S3AFileSystem after its underlying Apache HTTP connection pool has been shut down, causing subsequent S3 operations to fail with IllegalStateException: Connection pool shut down.

Changes

• Detect Connection pool shut down failures through an AWS SDK execution interceptor and mark the affected S3A filesystem as unusable.
• Evict the unusable filesystem from the shared cache using identity-based removal to avoid removing a replacement instance.
• Recreate the filesystem on subsequent access, allowing other S3FileIO instances with the same cache key to recover as well.
• Handle filesystem initialization and cache publication failures safely, preventing unusable instances from being published or retained.
• Add regression tests covering cache eviction and replacement, concurrent access, initialization failures, and unrelated exceptions.

Scope

• Applies to the Hadoop S3A client using the Apache HTTP transport.
• Does not automatically retry failed S3 operations.
• Does not cover the Netty async client.

[Akash3121]

Existing scheme test rejects the new subclass:
Introducing this subclass changes the runtime class name, but paimon-filesystems/paimon-s3/src/test/java/org/apache/paimon/s3/S3ASchemeTest.java:145 assertEquals("org.apache.hadoop.fs.s3a.S3AFileSystem", fs.getClass().getName()); still requires the exact name org.apache.hadoop.fs.s3a.S3AFileSystem. Both parameterized scheme cases consequently fail in the required JDK 8 and JDK 11 CI jobs. Please update that test to verify S3A ancestry/behavior without requiring the old concrete class name; note that this plugin test may use a separate classloader, so a simple cross-classloader instanceof may not be reliable.

[PyRSA]

Thanks for pointing this out. I've updated S3ASchemeTest to verify the S3A superclass hierarchy by class name instead of asserting the exact runtime class name. This avoids relying on instanceof, which may be unreliable across isolated classloaders.

[JingsongLi]

Verified actual recovery on Hadoop 3.4.2 / AWS SDK 2.29.52 against a loopback endpoint. I closed the Apache transport, observed the first HEAD fail, then verified both S3FileIO instances converged on a replacement and the peer HEAD succeeded. All 12 cache tests passed. The inline comments cover two additional publication and retention reproductions.