S3 bucket content is not getting listed till last

2019-09-11 17:32发布

I started listing the contents of a S3 bucket using foreach loop.

$contents = $s3->getBucket("bucket-name");
foreach($contents as $value) 
    {
      print $value['name']; echo "<br>"; 
    }

But the problem is that it is not listing above 1000 index. The last few files are not listing. I tried getting the length of the array and I found that last few files are not even getting stored in that array.

What should I do to list them all?

2条回答
The star\"
2楼-- · 2019-09-11 17:47

As stated in the S3 developer documentation:

As buckets can contain a virtually unlimited number of keys, the complete results of a list query can be extremely large. To manage large result sets, Amazon S3 API support pagination to split them into multiple responses. Each list keys response returns a page of up to 1,000 keys with an indicator indicating if the response is truncated. You send a series of list keys requests until you have received all the keys. AWS SDK wrapper libraries provide the same pagination.

That means you need multi page requests to iterate over multiple "pages". This will depend on the implementation of your "s3"-Class. Which library are you using for s3?

If the implementation you are using offers a "getBucket"-Method, there will be an optional parameter $marker. Place here the key of the last object to retrieve the next 1000 objects after that object.

Documentation:

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html

查看更多
爷的心禁止访问
3楼-- · 2019-09-11 17:51

The default number of listed files is set to 1000, so thats why you cannot get more than 1000 files:

Here you have a description of a function:

public static function getBucket($bucket, $prefix = null, $marker = null, $maxKeys = null, $delimiter = null) {

Use $marker to set where to start getting files and $maxKeys as a limit.

WWW: http://www.drupalcontrib.org/api/drupal/contributions!media_mover!contrib!mm_s3!S3.php/function/S3%3A%3AgetBucket/6

If you use the same library.

查看更多
登录 后发表回答