Book a Demo
Book a Demo

AWS Batch Memory Machine Engine 1.0

Achyutha Harish

Overview

This document guides you through deploying the updated CloudFormation stack with an enhanced reporting feature. This stack deploys IAM roles, EC2 instances, EFS, and an AWS Batch Compute Environment. While you can still use your default AWS VPC, subnet, and security group, we recommend creating dedicated resources for better isolation and security. For foundational setup instructions, refer to our Deployment Guide, which covers the prerequisites and steps for deploying this stack.

Quick Recap from the Initial Guide:

  1. Prepare AWS Resources: Set up the essential AWS resources, including a VPC, subnet, security group, EC2 key pair, and GitHub token for repository access. The guide recommends creating dedicated resources for added security.

  2. Deploy the CloudFormation Stack: Log into AWS, input required parameters (e.g., VPC ID, subnet ID, key pair name), and deploy the stack. Using a unique prefix helps to organize resources effectively.

  3. Monitor and Access Resources: Track the deployment in the CloudFormation console. Once complete, you can access the AWS Batch environment, configure job definitions, and verify the setup.

  4. Run Nextflow Jobs (Optional): After deployment, Nextflow users can submit jobs to AWS Batch, supporting streamlined batch job management.

The new reporting feature is seamlessly integrated into this deployment process, so no additional setup steps are required.

Real-Time Reporting for Job Metrics

About the Reporting Feature

This release introduces a real-time reporting dashboard that offers insights into your batch job metrics. Accessible via a web interface, the dashboard provides a quick overview of:

• Total Jobs Executed: The total count of all processed jobs.

• Spot Reclaim Protection Events: Tracks successful recovery during spot instance interruptions, minimizing downtime and potential data loss.

• Total Runtime (in seconds): Shows the cumulative execution time of all jobs.

• Total Time Saved: Indicates time saved through efficient instance recovery, giving a clear view of performance gains.

Accessing the Reporting Dashboard

After deploying the stack as shown in the initial guide, follow these steps to access the reporting feature:

  1. Find the AnalyticsEC2Stack: Once deployed, the stack will automatically create a nested stack named AnalyticsEC2Stack.

  1. Locate the EC2 Instance:

    • In the AnalyticsEC2Stack under the CloudFormation console, navigate to the Resources tab.

    • Find the EC2 instance ID and click on it to open the EC2 console.

  2. Access the mmspotviewer Instance:

    • Locate the mmspotviewer instance.

    • In the instance details, find the Public IP address.

  1. Access the Reporting Dashboard:

    • Open a browser and enter the IP address using HTTP (not HTTPS), followed by :5000 to access the main dashboard.

With these enhancements, your AWS Batch deployment now includes a streamlined tool for monitoring and optimizing job execution, ensuring efficient workflow performance and resource management.

AWSAWS Batch Memory Machine Engine