Book a Demo
Book a Demo

How to set up your AWS S3 bucket for input data files and output result files for MMCloud Air

YT Liang

Introduction

This how-to guide walks you through how to set up your AWS S3 bucket and then use the S3 bucket to provide input data files to MMCloud Air to run workload and MMCloud will save output result files to your S3 bucket.

Pre-requisites

  1. An AWS account
  2. An AWS user, either
    a. root user
    b. AWS IAM user that has the privilege to create AWS S3 bucket

Overview of the Setup

  1. Create an AWS S3 bucket for your input data files and output result files
  2. Create an AWS IAM Policy which permits access your AWS S3 bucket
  3. Create an AWS IAM User and attach the AWS IAM Policy to the AWS IAM User
  4. Create access key and secret key for the AWS IAM User

By the end of the setup, you will be able to use the following information

  • S3 Bucket URI
  • S3 Region
  • S3 Access Key
  • S3 Secret Key

for input data files and output result files for MMCloud Air.

Please see next section for more detailed steps.

Steps

Step 1 - Create an AWS S3 bucket for your input data files and output result files

Step 1.1 - Visit your AWS Console. Search for s3. Click on Services - S3.

Step 1.2 - Pick the region for your AWS S3 bucket from the right-upper corner. Below is an example of region as us-west-1 US West (N. California) , and once selected, you can noticed the URL from your browser is https://us-west-1.console.aws.amazon.com/s3/home?region=us-west-1, which indicates region as us-west-1.

Step 1.3 - Create an AWS S3 bucket. Use default settings pre-populated by AWS. Below is an example of bucket name as s3-for-air, created at region us-west-1 US West (N. California).

Step 1.4 - Once created, you will be able to see a success message from your AWS Console and you can go into your AWS S3 bucket to create folder and upload input data files. Up until now, you have the following information ready -

  • S3 Bucket URI
    • s3://[your-bucket-name]
    • In the example above, it would be s3://s3-for-air
  • S3 Region
    • region identifier
    • In the example above, it would be us-west-1

Step 2 - Create an AWS IAM Policy which permits access your AWS S3 bucket

Step 2.1 - Visit your AWS Console. Search for iam. Click on Services - IAM.

Open image-20240425-173214.png
image-20240425-173214.png
Step 2.2 - Click on Policies on the left hand side, and click on Create policy.

Step 2.3 - Click on JSON on the right hand side to bring up the Policy editor in JSON format. Replace with the following policy which can read the input data files from your S3 bucket for MMCloud Air and write output result files to your S3 bucket from MMCloud Air. Which in this example, is s3-for-air.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": [
				"s3:*"
			],
			"Resource": [
				"arn:aws:s3:::s3-for-air",
				"arn:aws:s3:::s3-for-air/*"
			]
		}
	]
}

Step 2.4 - Give a policy name and click Create policy. In this example the policy name is policy-for-air.

Step 2.5 - Once created, you will be able to see a success message from your AWS Console and you can view your AWS IAM Policy.

Step 3 - Create an AWS IAM User and attach the AWS IAM Policy to the AWS IAM User

Step 3.1 - Click on Users on the left hand side, and click on Create user.

Step 3.2 - Provide an AWS IAM User name. In this example, we use user-for-air.

Step 3.3 - Select Attach policies directly to attach the AWS IAM Policy to the AWS IAM user and search for the policy name just created. In this example, it would be AWS IAM Policy policy-for-air attached to AWS IAM user user-for-air.

Step 3.4 - Once created, you will be able to see a success message from your AWS Console and you can view your AWS IAM User and the AWS IAM Policy attached to this AWS IAM User.

Step 4 - Create access key and secret key for the AWS IAM User

Step 4.1 - Go to the AWS IAM User just created. In this example, it would be user-for-air. Click on Create access key.

Step 4.2 - Select Application running outside AWS and create access key.

Step 4.3 - Record your access key (a string with 20 characters) and secret key (a string with 40 characters).

Step 4.4 - Up until now, you have additional following information ready -

  • S3 Access Key
    • a string with 20 characters
  • S3 Secret Key
    • a string with 40 characters

What’s Next

You will be able to use the information to set up your storage for MMCloud Air. Below is a screenshot of MMCloud Air adding an S3 bucket using the example we included in this how-to document.

Please visit https://air.mmcloud.io to start your first MMCloud Air workflow!

For support, please send email to support-air@memverge.com or join slack group [to be updated].

MMCloud Air