Upload files to AWS S3 in JMeter using Groovy

I use my personal AWS S3 to store all my personal and confidential documents. There are three primary reasons for choosing AWS S3: affordable, speed and reliable. If you are working on the AWS cloud, the usage of S3 is inevitable. S3 plays a critical role in storing objects in hot and cold storage. Sometimes you need to upload a payload or file objects to S3 programmatically via your performance test script. This blog article will help you to upload files to AWS S3 in JMeter using Groovy.

What is S3?

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.

S3 comes with various storage classes: S3 Standard, S3 Intelligent Tiering, S3 Glacier Instant Retrieval and more.

Prerequisites

The following are the prerequisites for upload files to AWS S3:

• AWS Account
• AWS IAM User with S3 access policy,
• basic knowledge of AWS services,
• basic knowledge of JMeter
• basic knowledge of AWS SDK (Java)
• file(s) to upload

AWS IAM User

Login into your AWS account and open IAM service. Click Users under Access Management.

Click Add users button to set the user details and credential type as shown below.

Enter jmeter-s3-access in User name and check Access key - Programmatic access.

This user will have access to the AWS service programmatically, not from the user console. Click Next: Permissions

In the Set permissions section, click Attach existing policies directly and filter the policies of S3 by typing s3.

For the demonstration purpose, let us go with AmazonS3FullAccess. Check AmazonS3FullAccess and then click Next: Tags. But for the production server, follow the zero trust framework.

Adding tags is optional, but it is recommended to have relevant key-pair values.

Click Review and then click on Create user.

Copy the Access key ID and Secret access key to a secured location. Alternatively, you can download the .csv file.

JMeter Test Plan

By default, JMeter doesn’t have the feature to upload the artifacts to AWS S3. To extend the functionality, we must leverage the JSR223 Sampler.

Here is the complete playlist of JMeter series which will help you to become a hero within 1 week.

JSR223 Sampler

It is not possible to write upload to S3 code block natively. We must leverage the latest version of AWS SDK for Java. To add the AWS SDK as a dependency to the JSR223 Sampler, the easiest way is to leverage Grape. Personally, I have not tried the JMeter Maven plugin. I found Grape is simple to get start.

What is Grape?

The Groovy Adaptable Packaging Engine or Groovy Advanced Packaging Engine, Grape, is a JAR dependency manager which is built-in with Groovy.

Using the @Grab annotations in JSR223, you can add maven repository dependencies to the classpath.

@Grab(group='software.amazon.awssdk', module='s3', version='2.17.172', scope='test')

The above annotation will download the AWS SDK S3 dependencies. To download multiple dependencies, use @Grapes annotations.

@Grapes(
    @Grab(group='software.amazon.awssdk', module='s3', version='2.17.172', scope='test'),
    @Grab(group='software.amazon.awssdk', module='sts', version='2.17.172', scope='test')

)

To change the source, use @GrabResolver annotation.

@GrabResolver(name='restlet', root='http://maven.restlet.org/')

Hello Time using Grape in JSR223

Let us understand how Grape works in JSR223 Sampler in JMeter by writing a simple snippet with @Grab annotations. To explain it with a simple Hello world, pardon Hello time example, let us use the below code snippet. Copy and paste the below code into your JSR223 Sampler in JMeter.

@Grapes(
    @Grab(group='joda-time', module='joda-time', version='2.10.14')
)
import org.joda.time.LocalDateTime

LocalDateTime currentDateTime = new LocalDateTime()
log.info "Local Time is " + currentDateTime

Let us slice each line. @Grapes is the optional inception annotation which has @Grab annotation. One or more @Grab annotations can be placed anywhere in the JSR223. In this example, @Grab annotation will manage the joda-time dependency and prints the current date and time in the Log Viewer. By default, it will download the dependencies from mvnrepository.com.

Click on the Run button in JMeter to see the log message in the Log Viewer. Once you click on Run button, the above script will download the dependencies from the source and keep the JARs in /Users/<user>/.groovy/grapes in Mac, C:\Users\<user>\.groovy\grapes in Windows OS.

The dependencies download will happen only for the first execution. Eventual execution in JMeter will be fast.

AWS S3 SDK

There are multiple methods available to upload the artifacts to S3, e.g. via AWS CLI, AWS SDK, HTTP requests and more. The official AWS SDK opens the door to building applications for AWS via its API.

By leveraging the AWS SDK for Java, it is easy to interact with the AWS services. The latest version of the AWS SDK for Java is v2. v2 is a major rewrite of v1. The v2 version is packed with a nonblocking I/O architecture using Netty where it achieves high concurrency with fewer threads. Also, it supports HTTP/2 and automatic pagination.

Upload files to S3 in Groovy

We are going to leverage Grape in Groovy in the JSR223 Sampler by adding AWS SDK for S3. Head to https://mvnrepository.com/artifact/software.amazon.awssdk/s3 and select the latest version. At this time of writing, the latest version is 2.17.172.

Click Grape and copy the annotation as shown below.

Then, copy and paste the below snippet into the JSR223 Sampler.

@Grapes(
    @Grab(group='software.amazon.awssdk', module='s3', version='2.17.172', scope='test')
)
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider
import software.amazon.awssdk.regions.Region
import software.amazon.awssdk.services.s3.S3Client
import software.amazon.awssdk.services.s3.model.*

import java.io.File
import java.nio.file.Paths

// Configurations
String accessKey = vars.get("AWS_ACCESS_KEY")
String secretKey = vars.get("AWS_SECRET_KEY")
String bucketName = vars.get("AWS_BUCKET_NAME")
String strFilename = "C:\\temp\\result.json"

try {
	// Set region
	Region region = Region.US_EAST_1
	
	// Create credentials
	AwsBasicCredentials awsCreds = AwsBasicCredentials.create(
	      accessKey,
	      secretKey)
	
	// Build S3 Client
	S3Client s3 = S3Client.builder()
	      .region(region)
	      .credentialsProvider(StaticCredentialsProvider.create(awsCreds))
	      .build()
	
	// Create file object
	File s3Obj = new File(strFilename)
	
	// Create PUT request
	PutObjectRequest request = PutObjectRequest.builder()
	      .bucket(bucketName)
	      .key(s3Obj.getName())
	      .build()
	
	// Upload file
	s3.putObject(request, Paths.get(strFilename))
}

// Catching exception and displaying it in the Sample result
catch (S3Exception e){
	SampleResult.setSuccessful(false)
	SampleResult.setResponseMessage(e.getMessage())
	SampleResult.setResponseCode(null)
}

Let us slice the above code.

1. After adding the import statements, the first block represents the configurations such as AWS Access, Secret Key, and Bucket name.
2. The next configuration is the AWS region.
3. AwsBasicCredentials block creates the credentials to access the bucket
4. S3Client block creates a S3 client using the AwsBasicCredentials credentials.
5. s3Obj is the file to be uploaded.
6. PutObjectRequest will build the request.
7. s3.putObject will upload the file by leveraging the request.

If any exceptions occurs, JMeter will display the exceptions in the sampler with the exception details for troubleshooting.

The variables AWS_ACCESS_KEY, AWS_BUCKET_NAME, and AWS_SECRET_KEY is available in the Test Plan.

Here is the repository to download the sample JMeter test plan for your reference.

Never ever store the AWS credentials in the test plan. Pass the credentials via command line, environment variables, or programmatically generate them.

Congratulations! Now you know how to upload artifacts to S3 programmatically in JMeter using JSR223 Sampler.

Conclusion

Uploading artifacts to S3 is just the beginning. By leveraging the AWS SDK for Java, it is possible to interact with AWS services from JMeter using Groovy. E.g, after JMeter test execution, you can upload the results in a zip file using the above snippet. If you have any other usecases for JMeter, please let me know in the comments.