在 S3 中迭代对象时的“连接池超时异常”

java amazon-s3 amazon-web-services apache-httpcomponents

2022-09-04 00:47:53

我已经使用aws java API工作了一段时间，没有那么多问题。目前我正在使用库1.5.2版本。

当我使用以下代码迭代文件夹中的对象时：

AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials(MyClass.class.getResourceAsStream("AwsCredentials.properties")));

String s3Key = "folder1/folder2";


String bucketName = Constantes.S3_BUCKET;
String key = s3Key +"/input_chopped/";

ObjectListing  current = s3.listObjects(new ListObjectsRequest()
        .withBucketName(bucketName)
        .withPrefix(key));

boolean siguiente  = true;

while (siguiente) {    

    siguiente &= current.isTruncated();
    contador += current.getObjectSummaries().size();

    for (S3ObjectSummary objectSummary : current.getObjectSummaries()) {        
        S3Object object = s3.getObject(new GetObjectRequest(bucketName, objectSummary.getKey()));
        System.out.println(object.getKey());
    }

    current=s3.listNextBatchOfObjects(current);

}

要点：链接：https://gist.github.com/fgblanch/6038699 我得到以下异常：

INFO  (AmazonHttpClient.java:358) - Unable to execute HTTP request: Timeout waiting for connection from pool
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
    at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:232)
    at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:199)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:456)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:315)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:199)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2994)
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:918)
    at com.madiva.segmentacion.tests.ListaS3.main(ListaS3.java:177)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caught an AmazonClientException, which means the client encountered a serious internal problem while trying to communicate with S3, such as not being able to access the network.
Error Message: Unable to execute HTTP request: Timeout waiting for connection from pool

任何想法如何避免这个错误。它只发生在有许多对象的文件夹中，在这种情况下，里面有463个文件。谢谢

答案 1

我发现S3Object为每个对象打开一个连接。即使对象被垃圾回收，也不会释放该对象，因此需要执行 object.close（）以释放与池的连接。

因此，更正后的代码将是：

 for (S3ObjectSummary objectSummary : current.getObjectSummaries()) {        
        S3Object object = s3.getObject(new GetObjectRequest(bucketName, objectSummary.getKey()));             
        System.out.println(object.getKey());
        object.close();
    }

答案 2

检查 HttpResponseHandler 是否正在关闭来自池的连接。AmazonHttpClient 具有参数“leaveHttpConnectionOpen”，它指示连接是否应关闭。