How to Use “list_bucket()” Functions in S3 Using Boto3?

AWS has become the most popular and powerful cloud computing platform. With its exceptional services for performing complex data-related tasks, AWS has caught the eye of the tech enthusiast by offering cost-optimal and a wide variety of services. S3 bucket is one of the names among the most widely used services of AWS. The S3 bucket is widely used for its flexibility of remote access and wide support of integration for multiple AWS services. 

Learn more: Getting Started With Buckets: Overview of Amazon Simple Storage Service

What is the Client Interface in Boto3?

Boto3 SDK which is provided by Python is used to access and manage the AWS services. There are two ways for using Boto3 with AWS i.e., via Client or Resource. The client abstraction of the Boto3 SDK is a low-level interface. By using the Client interface, users can access the AWS resources via various built-in methods. The methods that are provided by the Client interface include some of the common operations such as creating the bucket, downloading files, deleting objects, or uploading files, etc. In this tutorial, we will be implementing the list() method of the Boto3 SDK using the Client interface.

Prerequisite

Before getting started with the implementation of the list() method, there are some prerequisites for this tutorial: 

• AWS CLI is installed on the local machines.
• Visual Studio Code for executing the code
• Python runtime environment for using Boto3 SDK.

What is the “list()” Function in Boto3?

The list() method returns a list of the specified resources. When using this method with the S3 bucket, this list() function will display a list of all the buckets or objects within a bucket. The list() method uses the pagination approach where 1000 buckets or objects are listed per single response.

Common Parameters 

The list() function accepts various parameters. Some of these parameters are required while the others are optional. The common parameters in the list() function are:

• isTruncated: It is a boolean-type field that determines whether there are more keys to list or not. If there is more data, it returns the true value and vice versa. 
• NextContinuationToken: It specifies the starting point from where the next listing should begin. This value is given in the output when the “isTruncated” is true.
• ContinuationToken: This field is generated by the S3 bucket and is used for pagination when list functions are used. The continuation tokens are used when there is a huge number of keys and objects to be listed and the output does not contain every key. The value of the NextContinuationToken is provided to the ContinuationToken field within the input.
• ExpectedBucketOwner: This field inputs the account ID of the expected bucket owner. If the account ID provided lies in a different account, the program execution will be ended with the 403 status code. This status code defines the forbidden access or access denied error.
• RequestPayer: This field confirms that the requester agrees to deduct the charges for the request. The user is not required to specify this parameter. However, if this field is enabled on the source or destination bucket, the requester will be charged.

Initial Configurations

Before implementing the code, we will install the Boto3 SDK from Python. For this purpose, create a folder and open this folder in the VS code:

Inside the Boto3 folder, click on the “New file” option and provide the name of the file with the “.py” extension:

To install “Boto3”, open the terminal inside this folder. For this purpose, click on the “Terminal” option from the toolbar. Tap the “New terminal” option from the drop-down list:

Provide the following command to the terminal for installing the Boto3:

pip install boto3

After the SDK has been successfully installed, the next step is to log in to the AWS account by providing the following command in the terminal:

aws configure

Provide the Secret Access key, Access key, etc of the AWS account to the terminal:

How to Use the “list_bucket_analytics_configurations()” Method Using Boto3?

To list the analytics configurations of a bucket, the list_bucket_analytics_configurations() function of the Client Interface is used. The output of this function only lists 100 configurations per response.To enlist more configurations, the list_bucket_analytics_configuration() function uses certain parameters that will be discussed later in this article.

Syntax

The syntax of this function is as follows:

response = client.list_bucket_analytics_configurations(
    Bucket='string',
    ContinuationToken='string',
    ExpectedBucketOwner='string'
)

In the above syntax:

• Bucket: This is a required parameter that specifies the name of the bucket whose analytics configurations are to be listed.

Example: List the S3 Analytics Configuration Using the list_bucket_analtyics_configuration()

To list the analytics configuration in S3 using the Boto3 client, paste the following code in the IDE:

import boto3
s3=boto3.client('s3')

response = s3.list_bucket_analytics_configurations(
    Bucket='newbucket191008'
)

print(response)

Below is a brief description of the above-mentioned code:

• s3.list_bucket_analytics_configurations(): This function is used to list the analytical configuration of a bucket in S3.
• Bucket: For this demo, we have provided the name of the bucket to list its analytical configurations e.g., newbucket191008. Note that this value is to be replaced by the user with the name of the bucket configured in your AWS S3 dashboard. 
• output=response[‘ResponseMetadata’]: As the response returned by this code is of the “dictionary” data type, therefore, we will print its key-value pairs to increase the readability of the code. The response dictionary will return all the values stored for the “ResponseMetadadata” key to the “output” variable.
• for loop: the for loop in the code will filter the output based on the key-value pair of the output dictionary. This filtration of contents is achieved by using the item() function which is a built-in function.

Output

The code file can be executed by using the following command:

python <FileName>

The output generated by this function is of dictionary data type. Different fields will be represented by different key-value pairs. The output contains information about the HostId, StatusCode, Content length, etc. The highlighted field “IsTruncated” is of boolean data type. This indicates that all the results are listed in this single response:

This is all from this section.

How to Use the “list_buckets()” Method Using Boto3?

This function is used to list all the buckets configured within the AWS account. The output of this function is of dictionary data type and contains key-value pairs. The list_buckets() method will return a list of the configured buckets within an AWS account after authenticating the request.

Syntax

Below is the syntax for the list_buckets() function:

S3.Client.list_buckets()

Example: List buckets by Using the list_buckets() method

For listing the buckets, use the following code inside the Python file to list all the buckets:

import boto3
s3=boto3.client('s3')
listbucket=s3.list_buckets()
i=1
for buckets in listbucket["Buckets"]:
    print("Bucket number", i, ":", buckets)
    print("\n")
    i=i+1

Below given is the description of the above-mentioned parameters:

• s3.list_buckets(): This statement will be used to list all the buckets from the dashboard of the S3 service. 
• listbuckets: The list returned by the list_buckets() function will be then stored in the “list bucket” variable.
• i=1: The value “i” is a variable whose initial value is 1. It is used to verify the number of buckets printed.
• for buckets in listbucket[]: The list_buckets() function will return a list of names of buckets. Hence, the for loop is used to iterate over each value of the list. The value “[Buckets]” is the keyword that refers to the one bucket present at index 0, 1, 2, etc. 
• print(“\n”): This statement is used to add extra spaces among the buckets to improve the readability.
• i=i+1: This statement will increment the value of i by one whenever the data of one bucket is iterated and printed by the for loop.

Output

To execute the Python file, use the following command in the terminal:

python <FileName>

In the Terminal window, the names of all three buckets along with the “CreationDate” have been displayed:

That is all from this section.

How to Use the “list_objects_v2()” Method Using Boto3?

In this tutorial’s section, we will list all the objects uploaded inside a bucket. For this purpose, the list_objects_v2() function of boto3 SDK is used. Currently, there is only 1 object uploaded inside the “snsbucket2023”:

To implement this functionality, provide the following code to the Python file:

import boto3
s3=boto3.client('s3')

list_objects=s3.list_objects_v2(Bucket="snsbucket2023")
i=1
for listobj in list_objects["Contents"]:
    print("Object number", i , listobj)
    print("\n")
    i=i+1

Below given is a description of the mentioned fields:

• s3.list_objects_v2(Bucket= “name”): The “list_objects_v2()” function is used to get the names of all the objects stored in a bucket. The term “Bucket” refers to the name of the specific bucket whose objects are to be determined.
• list_objects: This variable will be used to store the list of the objects returned by the list_objects_v2() function.
• i=1: The variable “i” is used to determine the number of objects returned and to improve the readability of the output.
• for list obj in list_objects[“Contents”]: The for loop is used to iterate over the list returned by the function. The “list obj” will contain the data of each object uploaded inside the bucket.

Output

This Python file is executed using the following command:

python <FileName>

The output of the code is given below:

That is all from this section.

How to Use the “list_bucket_intelligent_tiering_configurations()” Method Using Boto3?

S3 Intelligent-tiering configurations are one of the storage classes of the S3 bucket that is used for cost optimization. This storage class is best suited for unpredictable data patterns, object size, or retention period. This storage class determines which data is accessed frequently or occasionally and hence, moves them to suitable tiers for reducing the AWS spending. 

The list_bucket_intelligent_tiering_configuration accepts two arguments. The output generated by this function is of dictionary data type which provides the information about the intelligent tiering configurations. 

Read more: How to Optimize Data Storage Costs in Amazon S3 With Intelligent-Tiering?

Syntax

The syntax of this function is given below:

response = client.list_bucket_intelligent_tiering_configurations(
    Bucket='string',
    ContinuationToken='string'
)

The following are the parameters of the above-given syntax:

• Bucket: It specifies the name of the bucket for which the intelligent tiering configurations are to be listed.

By using the above-mentioned request syntax, let’s explore a practical application of this method.

Example: Implementing the list_bucket_intelligent_tiering_configurations() Method

This example demonstrates the implementation of the list_bucket_intelligent_tiering_configurations() function of the Client interface: 

import boto3
s3=boto3.client('s3')

response = s3.list_bucket_intelligent_tiering_configurations(
    Bucket='myawsbucket20203'
)

print("\n")
print(response)
print("\n")

In the above lines of code:

• Bucket: This parameter inputs the name of the specified bucket to list its intelligent tiering configurations. Replace the value of this field with the name of the bucket configured in your AWS account. 

Output

To execute the code file, provide the name of the file to the <FileName> within the command and hit “Enter” from the keyboard:

python <FileName>

The output of the above code is as follows:

That is all from this guide.

How to use the “list_bucket_inventory_configurations()” Method in Boto3?

This function is used to list the inventory configurations. As the inventory configurations can be either delivered daily or weekly, this function helps the user to list not more than 100 configurations per response. Similar to other list functions, the output of this function also generates the “isTruncated” element.

To implement this functionality programmatically, the bucket owner must be equipped with the required permissions i.e., s3:GetInventoryConfigurations action. By default, the bucket owner has the permission to perform this action. 

Syntax

The list_bucket_inventory_configurations() method accepts three arguments. These arguments can either be required or optional. The output of this function is of dictionary type and contains different key-value pairs. 

The syntax for implementing this function is given below:

response = client.list_bucket_inventory_configurations(
    Bucket='string',
    ContinuationToken='string',
    ExpectedBucketOwner='string'
)

The following is a brief description of its parameters:

• Bucket: it is a required parameter and inputs the name of the bucket for which the inventory configurations are to be listed.

Example: Implementing the list_bucket_inventory_configurations() Method

Before implementing this feature, the inventory configuration feature must be enabled on your S3 bucket. To implement this feature on your S3 bucket using the Console, refer to this article: “How to create inventory configuration in S3 bucket” .

For this demo, we have already implemented this feature for the specified S3 bucket. Below is the code for using list_bucket_inventory_configurations() method:

import boto3
s3=boto3.client('s3')

response = s3.list_bucket_inventory_configurations(
    Bucket='inventoryconfigurationdemobucket'
)

print(response)

Output

Provide the following command to the terminal after replacing the <FileName> with your Python file’s name:

python <FileName>

The output of the above code is as follows:

That is all from this section of the tutorial.

How to Use the “list_bucket_metrics_configurations()” Method in Boto3?

This function is used to list the metrics configuration of the specified S3 bucket. The list_bucket_metrics_configurations() are only for the request metrics of the bucket. The configurations listed in the response do not provide any information about the daily storage metrics. 

This function supports list pagination and only displays information of 100 configurations in a single response. The output of the function contains the “isTruncated”, continuation token, and next continuation token element.

Syntax

The list_bucket_inventory_configurations() method accepts three arguments. The syntax for implementing this function is given below:

response = client.list_bucket_metrics_configurations(
    Bucket='string',
    ContinuationToken='string',
    ExpectedBucketOwner='string'
)

Below is a brief description of its parameters:

• Bucket: it is a required parameter and inputs the name of the bucket for which the inventory configurations are to be listed.

Example: Implementing the list_bucket_metrics_configurations() Method

Before implementing this feature, the metrics configuration must be implemented on the S3 buckets. The response generated by this output will contain a specified field “MetricsConfigurationList” that provides the information on the ID, prefix, tags, filter, ARN, etc.

For this demo, we have already implemented this feature for the specified S3 bucket. Below is the code for using the list_bucket_metrics_configurations() method: 

import boto3
s3=boto3.client('s3')

response = s3.list_bucket_metrics_configurations(
    Bucket='s3bucketconfiguraationdemo2023'
)

print(response)

Output

Replace the <FileName> with the name of the Python code file and execute the following command:

python <FileName>

The output of the above code is as follows. Within the output, we can see the filter applied to the configurations:

That is all from this part of the tutorial.

How to use the “list_object_versions()” Method in Boto3?

This function is used to provide information about the versions of the object. The version feature of the S3 service allows the user to maintain a history of different versions of the object. It forms a  tree-like hierarchy and allows the users to retrieve the older versions of the existing file. 

Syntax

The list_object_versions() method accepts three arguments out of which some are required and some are optional. The syntax for implementing this function is given below:

response = client.list_object_versions(
    Bucket='string',
    Delimiter='string',
    EncodingType='url',
    KeyMarker='string',
    MaxKeys=123,
    Prefix='string',
    VersionIdMarker='string',
    ExpectedBucketOwner='string',
    RequestPayer='requester',
    OptionalObjectAttributes=[
        'RestoreStatus',
    ]
)

The following is a brief description of its parameters:

• Bucket: it is a required parameter and refers to the name of the bucket that contains the object. The information on the version configuration will be listed for this specified object.
• Delimiter: A delimiter is used to group keys. A delimiter is often used with the prefix to organize files and folders. The group of the keys will be considered as one in the output of the list_object_versions() method. 
• EncodingType: This field is used to encode the object keys in the output of the function. Furthermore, it represents the encoding method to use. The object key can be any Unicode character; however, these characters are not processed and parsed by XML 1.0 such as ASCII characters whose values lie between 0-10.
• KeyMarker: it specifies the keys that serve as the starting point for the listing of the objects. 
• MaxKeys: This field represents the number of keys to be listed within a single response. 
• Prefix: This field is used to list only those keys or objects that begin with the specific prefix in a bucket.
• VersionIdMarker: This field specifies the object version from where the listing of the objects should begin.
• OptionalObjectAttributes: It takes input of names of those fields that a user wants to specify in the output of the code.

Example: Implementing the list_object_versions() Method

Before implementing this feature, ensure that the specified bucket is enabled while maintaining the versioning feature. To implement this functionality using the AWS S3 console, refer to this article: “How to Configure AWS S3 Bucket Versioning”.

At the moment, we have this feature enabled on the S3 bucket. Following list the version configuration by using the list_object_versions() in S3:

import boto3
s3=boto3.client('s3')

response = s3.list_object_versions(
    Bucket='myawsbucketversiondemo2023'
)

print("\n")
print(response)
print("\n")

Output

Provide the command to the terminal after replacing the <FileName> with your file name:

python <FileName>

The output is given as follows:

That’s all from this part of the blog.

How to use list_objects() Method in Boto3?

The list_objects() method returns the names of the objects present within the bucket. There are multiple parameters available for customizing the response such as limiting the response within an output, size, etc. As it falls under the category of the list() function, it also supports the pagination format and returns a 200 status code.

Syntax

The syntax for implementing this function is given below:

response = client.list_objects(    Bucket='string',    Delimiter='string',    EncodingType='url',    Marker='string',    MaxKeys=123,    Prefix='string',    RequestPayer='requester',    ExpectedBucketOwner='string',    OptionalObjectAttributes=[        'RestoreStatus',    ])

The following is a brief description of its parameters:

• Bucket: it is a required parameter and refers to the name of the bucket that contains the object. 
• Delimiter: A delimiter is used to group keys. A delimiter is often used with the prefix to organize files and folders. The group of the keys will be considered as one in the output of the list_objects() method. 
• EncodingType: This field is used to encode the object keys in the output of the function. Furthermore, it represents the encoding method to use. The object key can be any Unicode character; however, these characters are not processed and parsed by XML 1.0 such as ASCII characters whose values lie between 0-10.
• Marker: it specifies the keys from where the listing of the objects should begin.
• MaxKeys: This field represents the number of keys to be listed within a single response. 
• Prefix: This field only lists those objects that begin with the specified prefix. 
• OptionalObjectAttributes: It takes input of names of those fields that a user wants to specify in the output of the code.

Example: Implementing the list_objects() Method

Before implementing this code, ensure that there is some data (objects) uploaded to the S3 bucket. Replace the name of the Bucket with your S3 bucket’s name:

import boto3
s3=boto3.client('s3')

response = s3.list_objects(
    Bucket='myawsbucketversiondemo2023'
)
print("\n")
print(response)
print("\n")

Read more: “How to Upload Objects in Amazon Simple Storage Service”. 

Output

Provide the command to the terminal after replacing the <FileName> with your file name:

python <FileName>

The output of the code is given below:

That is all from this section of the tutorial.

How to use the “list_multipart_uploads()” Method in Boto3?

This function is used to list the in-progress multi-part for a specified object within the bucket. The list() functions support the pagination format and thus only list 1000 responses per request. Similar to other list functions, users can also limit the number of responses within this request.  The output of this method is sorted by the keys following the ascending order.

Syntax

The syntax for implementing this function is given below:

response = client.list_multipart_uploads(    Bucket='string',    Delimiter='string',    EncodingType='url',    KeyMarker='string',    MaxUploads=123,    Prefix='string',    UploadIdMarker='string',    ExpectedBucketOwner='string',    RequestPayer='requester')

The following is a brief description of its parameters:

• Bucket: it is a required parameter and refers to the name of the bucket within which the multi-part operation was first initiated.
• Delimiter: A delimiter is used to group keys. A delimiter is often used with the prefix to organize files and folders. The group of the keys will be considered as one in the output of the list_bucket_inventory_configurations() method. 
• EncodingType: This field is used to encode the object keys in the output of the function. Furthermore, it represents the encoding method to use. The object key can be any Unicode character; however, these characters are not processed and parsed by XML 1.0 such as ASCII characters whose values lie between 0-10.
• KeyMarker: it specifies the multi-part upload after which the listing should begin.
• MaxUpload: This field limits the maximum number of uploads for a multi-part operation.
• UploadIdMarker: This field specifies the object listing after a specific upload id.
• Prefix: It is used to list only those multi-part objects that match a specific prefix.

Example: Implementing the list_mulitipart_uploads() Method

Below given code provides a practical application of this function:

import boto3
s3=boto3.client('s3')

response = s3.list_multipart_uploads(
    Bucket='myawsbucketversiondemo2023',
)
print("\n")
print(response)
print("\n")

In the above code:

• Bucket: Replace the Bucket name with your bucket name for which the multiparts are to be listed.

Output

To execute this code file, provide the name of the file to the <FileName>. Use the below-mentioned command to begin executions:

python <FileName>

The output of the code is given below:

That is all from this section.

How to use the “list_parts()” Method Using Boto3?

The list_parts() method is used to list the parts for a particular multipart operation. This function also follows the pagination format and only lists 1000 responses per request. The user can specify the number of responses per request by using the max-part parameter. 

Syntax

The syntax for implementing this function is given below:

response = client.list_parts(
    Bucket='string',
    Key='string',
    MaxParts=123,
    PartNumberMarker=123,
    UploadId='string',
    RequestPayer='requester',
    ExpectedBucketOwner='string',
    SSECustomerAlgorithm='string',
    SSECustomerKey='string',

)

The following is a brief description of its parameters:

• Bucket: it refers to the name of the buckets to which the multi-part operation is performed.
• Key: This field specifies the name of the object for which the list_part() operation is to be performed.
• MaxPart: This field allows the user to limit the number of responses within the output. 
• Prefix: This field only lists those keys that begin with the specified prefix.
• PartNumberMarker: This field specifies the listing part from where the listing of the objects should begin.
• SSECustomerAlgorithm: It refers to the Server-side encryption format.
• SEECustomerKey: This field inputs the Server-side customer-managed key for encryption.

Example: Implementing the list_parts() Method

Before implementing this feature, ensure that a multipart request has been made for an object within the bucket. This will return a UploadID which will determine the different parts for a specific multi-part operation.

At the moment, we have already performed the multipart operation and have provided the UploadId within the code. Replace this “UploadId” with your multi-part upload id:

import boto3
s3=boto3.client('s3')

response = s3.list_parts(
    Bucket='myawsbucketversiondemo2023',
    Key="downloaded1.txt",
    UploadId="pkgObkMNN1U739VN_A1PBQyaTAcOY3Nz3wYJenqBDfPddK09vNtUz9PIshxlRr7nOeKCKcRo7Is0Hh9z1jjsmu2jz.LHsPE88m.JOigWVOQznnNnHn3_XntV6TBVX4tuphm26CDFPSaPctkJo7CZag--"
)
print("\n")
print(response)
print("\n")

Output

Replace the “<FileName>” with the code file name. Use the following command for execution after modification:

python <FileName>

The output of the file is given as follows:

This is all from this section.

Conclusion

To use this list() method in Boto3 with S3, import the client interface and specify the arguments e.g., Bucket name, key name, etc, within the function. Execute the Python code file by using the mentioned command. It is worth noting here that before executing the code file, access the AWS console by signing into the account via the terminal. This article discusses different functions of the list() method along with a step-by-step demonstration of them with Boto3.