## Known Issues-1: AWS Rebalance Recommendation Signal

1. **Frequent Rebalance Recommendation Signals**:

**Issue**: AWS may send a [Rebalance Recommendation signal](https://docs.memverge.com/MMCloud/latest/User%20Guide/Exploring%20MMCloud%20Features/SpotSurfer/?h=rebala#aws-rebalance-recommendation-signal) if it determines that a Spot Instance has an increased likelihood of being reclaimed. This signal can be intercepted by the OpCenter, which uses a rules-based approach to decide whether to maintain the current Spot Instance or proactively capture a snapshot and transition to a new instance.
 
**Impact**: Frequent rebalance signals may lead to numerous interruptions, which can cause jobs to fail with checkpoint errors due to multiple spot reclaims.
 
**Example**: In the job example screenshot below, the OpCenter encountered numerous rebalance signals, resulting in 17 interruptions over a short period. These interruptions reset the buffer cache, causing processing delays and timeouts.

![](https://baywatch-api.memverge.com/media/upload/202408/240809162426315505.png)


2. **Default Rebalance Threshold Settings**:

**Issue**: The rebalance signal threshold on a new OpCenter is set to 64GB by default. This setting might trigger unnecessary rebalance actions that don't lead to actual spot reclaims.

**Impact**: This can cause jobs that require substantial buffer cache to fail or slow down significantly due to frequent cache resets and timeouts.
 
**Resolution**: Increasing the rebalance threshold to 125GB can help by ensuring that jobs are only checkpointed on actual spot reclaim signals rather than on every rebalance signal.
 
> **Note**: Changes to the rebalance threshold will only apply to newly launched jobs and not to those already in progress.

# Setup MMC OpCenter on AWS


## Setup & Download EC2 Key-Pair for OpCenter Access

* Login to `AWS` & navigate to the `EC2` console and choose the appropriate region

![](https://baywatch-api.memverge.com/media/upload/202406/240626233908180703.png)

* Navigate to `Network & Security` from the left navigation bar and click on `Key Pairs`.

![](https://baywatch-api.memverge.com/media/upload/202406/240626234021897622.png)

* Create a key-pair if you don't have one and save this key. **NOTE: You will need this key only to SSH into the opcenter instance.** 

![](https://baywatch-api.memverge.com/media/upload/202408/240807081513238257.png)

---

## AWS Marketplace - Memory Machine Cloud Subscription

* Next, navigate to `AWS Marketplace` and search for `memory machine cloud` and click on `Continue to Subscribe`

![](https://baywatch-api.memverge.com/media/upload/202408/240807071354851639.png)

![](https://baywatch-api.memverge.com/media/upload/202408/240807071450108428.png)

* Next click on `Continue to Configuration`

![](https://baywatch-api.memverge.com/media/upload/202408/240807071628691257.png)

* Make sure the region is correct and leave the rest to defaults and click on `Continue to Launch`

![](https://baywatch-api.memverge.com/media/upload/202408/240807071801687942.png)

* Next choose `Launch CloudFormation` from the drop down box and click on the `Launch` button

![](https://baywatch-api.memverge.com/media/upload/202408/240807071930819017.png)

* You will be re-directed to a new window to create a stack. Use defaults on this page for tempaltes and click `Next`

![](https://baywatch-api.memverge.com/media/upload/202408/240807072107784401.png)

* Next, specify a stack name

![](https://baywatch-api.memverge.com/media/upload/202408/240807072307600844.png)

* Next, select an instance type for MMCE OpCenter under `OpCenterType`

![](https://baywatch-api.memverge.com/media/upload/202408/240807072442790376.png)

| Type | CPU | Memory(GB) | Price/hour | Guidance |
| - | - | - | - | - |
| POC | 2 | 2 | $0.0188 | All historic jobs < 13K| 
| Small | 2 | 4 | $0.0765 | All historic jobs < 26K |
| Medium | 4 | 8 | $0.1530 | All historic jobs < 80K |
| Large | 4 | 16 | $0.2016 | All historic jobs < 185K |

* If you prefer a `CustomizedInstanceType` specify other instance type for MMCE OpCenter. Note this field is **OPTIONAL**.

![](https://baywatch-api.memverge.com/media/upload/202408/240807072958088594.png)

* Next specify disk type for MMCE OpCenter. Available types are `gp2` & `gp3`. Choose default `gp3` for better performance.

![](https://baywatch-api.memverge.com/media/upload/202408/240807073248757580.png)


* Next, select an EC2 Key Pair for all instances

![](https://baywatch-api.memverge.com/media/upload/202408/240807073435594906.png)

* Next, select a web-accessible VPC to run MMCE service

![](https://baywatch-api.memverge.com/media/upload/202408/240807073559749009.png)

* Next, select a subnet in the VPC to run OpCenter instance

![](https://baywatch-api.memverge.com/media/upload/202408/240807073712437070.png)

* Next, select the Availability Zone in which the subnet resides

![](https://baywatch-api.memverge.com/media/upload/202408/240807073830333880.png)

* Next, select `True` if you want to assign public IP to MMCE service

![](https://baywatch-api.memverge.com/media/upload/202408/240807073938118039.png)

* Next, define the IP address range to access MMCE service under `ExternalAccessCidr` & `SshCidr`.

> `0.0.0.0/0` to allow all access

![](https://baywatch-api.memverge.com/media/upload/202408/240807074118592777.png)

* Next, select `True` if you want to enable worker nodes to access other nodes internally. (Default is `False`)

![](https://baywatch-api.memverge.com/media/upload/202408/240807074355912386.png)

* Next, For `Configure stack options` options keep all defaults on this page and click `Next`

![](https://baywatch-api.memverge.com/media/upload/202408/240807074626143844.png)

* Next on the `Review and create` page, review all the parameters and finally select the check-box to allow CloudFormation to create resources and click on `Submit`

![](https://baywatch-api.memverge.com/media/upload/202408/240807074750728081.png)

* Next, you will see the stack creation in progress

![](https://baywatch-api.memverge.com/media/upload/202408/240807075129129926.png)

* Successful creation of stack will look as shown below with status `CREATE_COMPLETE`

![](https://baywatch-api.memverge.com/media/upload/202408/240807075426259908.png)

* Click on the `Resources` tab and click on the `mvOpCenter` id to re-direct to the EC2 instance assigned to the MMC OpCenter

![](https://baywatch-api.memverge.com/media/upload/202408/240807075508326039.png)

* Next, click on the `Instance ID` for more details of the instance

![](https://baywatch-api.memverge.com/media/upload/202408/240807075659076667.png)

* Next, click on the open address link of `Public IPv4 address` to be re-directed to the OpCenter

![](https://baywatch-api.memverge.com/media/upload/202408/240807075735012857.png)

> NOTE: When logging in for the first time to the opcenter, you might see a warning about invalid security certificate. Click on `Advanced` -> `Accept the Risk and Continue`

![](https://baywatch-api.memverge.com/media/upload/202408/240807075838231038.png)

* The MMCloud OpCenter landing page will look as below.

> The opcenter can now be accessed via its public IP address: `https://54.295.216.250/#/login`
> Bookmark this link for easy access

![](https://baywatch-api.memverge.com/media/upload/202408/240807080247340972.png)

> NOTE:
> default username: `admin`
> default password: `memverge`

---

# Change Admin Password

* We recommend changing the default username (`admin`) and password (`memverge`) for the opcenter after logging in for the first time.

* Click on `Users & Groups` from the left navigation menu and for the admin user click on `Actions` and change the password as necessary.

![](https://baywatch-api.memverge.com/media/upload/202408/240807080723259525.png)

![](https://baywatch-api.memverge.com/media/upload/202408/240807080855526736.png)

---

# Activate MMCloud License

* Next, activate your mmcloud license by clicking on the star icon in the top-right corner and enter your mmcloud account credentials

> Not registered? Visit www.mmcloud.io

![](https://baywatch-api.memverge.com/media/upload/202408/240807081016728068.png)

---

# Download FLOAT CLI

* The float CLI binary helps connect a user from any local machine to the MMCloud opcenter. To download the CLI, click on the terminal icon towards the top right corner of the opcenter and choose the version of CLI tool based on the operating system

![](https://baywatch-api.memverge.com/media/upload/202402/240207015753981576.png)

* From macOS 10.15 (Catalina) onwards, zsh is the default shell. In zsh, float is a reserved word. To use the float CLI with macOS, either change the shell to bash or alias the word float to the float binary, as follows:

```bash=
alias float = /path/to/float_binary/float
```
> where `/path/to/float_binary/` is the path to where you placed the float binary.

* To change the MAC-OS default shell from `zsh` to `bash` see the following [guide](https://phoenixnap.com/kb/change-zsh-to-bash-mac)

---

# Juiceflow - AWS

## Introduction

Juiceflow combines JuiceFS and Nextflow on MMCloud, offering a powerful, scalable solution for managing and executing workflows in the cloud.

<details>
<summary>Expand for more details on JuiceFS</summary>

[JuiceFS](https://juicefs.com/docs/community/introduction/) is an open-source, high-performance distributed file system designed specifically for cloud environments. It offers unique features, such as:

* Separation of Data and Metadata: JuiceFS stores files in chunks within object storage like Amazon S3, while metadata can be stored in various databases, including Redis.
* Performance: Achieves millisecond-level latency and nearly unlimited throughput, depending on the object storage scale.
* Easy Integration with MMCloud: MMCloud provides pre-configured nextflow head node templates with JuiceFS setup, simplifying deployment.
* Comparison with S3FS: For a detailed comparison between JuiceFS and S3FS, see [JuiceFS vs. S3FS](https://juicefs.com/docs/community/comparison/juicefs_vs_s3fs/). JuiceFS typically offers better performance and scalability.


</details>

---

## Pre-requisites

- A VPC Security Group with inbound-rule for port `6868`.
 - **Navigation:** `AWS EC2 console -> Network & Security -> Security Groups`

![](https://baywatch-api.memverge.com/media/upload/202406/240626212749399541.png)

> **Inbound rules should include:**
 - Custom TCP over TCP on port `6868`, used by the Redis server in this juiceflow setup.

- AWS S3 buckets for Nextflow work and output directories.
- AWS S3 access, secret keys & MMC OpCenter Credentials stored as float secrets

---

## Overview of the Setup

This solution leverages two scripts:
- `transient_JFS_AWS.sh`: Formats the work directory S3 bucket to JuiceFS format.
- `hostTerminate_AWS.sh`: Allows for graceful exit of workflow from nextflow head-node
- `job_submit_AWS.sh`: Contains Nextflow input parameters and MMC config.

---

## Steps

### Download Scripts

* Host-init script (you don't have to edit this script, but you'll need it later)

```bash!
wget https://mmce-data.s3.amazonaws.com/juiceflow/v1/aws/transient_JFS_AWS.sh
```

* Host-terminate script (you don't have to edit this script, but you'll need it later)

```bash!
wget https://mmce-data.s3.amazonaws.com/juiceflow/v1/aws/hostTerminate_AWS.sh
```

* Job-submit script (Download the template or create one locally based on the template below with your Nextflow inputs and run configurations)

```bash!
wget https://mmce-data.s3.amazonaws.com/juiceflow/v1/aws/job_submit_AWS.sh
```

<details>
<summary>Expand to view a sample `job_submit_AWS.sh` script</summary>

```bash!
#!/bin/bash

# ---- User Configuration Section ----
# These configurations must be set by the user before running the script.

# ---- Optional Configuration Section ----
# These configurations are optional and can be customized as needed.

# JFS (JuiceFS) Private IP: Retrieved from the WORKER_ADDR environment variable.
jfs_private_ip=$(echo $WORKER_ADDR)

# Work Directory: Defines the root directory for working files. Optional suffix can be added.
workDir_suffix=''
workDir='/mnt/jfs/'$workDir_suffix
mkdir -p $workDir # Ensures the working directory exists.
cd $workDir # Changes to the working directory.
export NXF_HOME=$workDir # Sets the NXF_HOME environment variable to the working directory.

# ------------------------------------------
# ---- vvv DO NOT EDIT THIS SECTION vvv ----
# ------------------------------------------

function install_float {
 # Install float
 local address=$(echo "$FLOAT_ADDR" | cut -d':' -f1)
 wget https://$address/float --no-check-certificate --quiet
 chmod +x float
}

function get_secret {
 input_string=$1
 local address=$(echo "$FLOAT_ADDR" | cut -d':' -f1)
 secret_value=$(./float secret get $input_string -a $address)
 if [[ $? -eq 0 ]]; then
 # Have this secret, will use the secret value
 echo $secret_value
 return
 else
 # Don't have this secret, will still use the input string
 echo $1
 fi
}

# Set Opcenter credentials
install_float 
access_key=$(get_secret AWS_BUCKET_ACCESS_KEY)
secret_key=$(get_secret AWS_BUCKET_SECRET_KEY)
export AWS_ACCESS_KEY_ID=$access_key
export AWS_SECRET_ACCESS_KEY=$secret_key

opcenter_ip_address=$(get_secret OPCENTER_IP_ADDRESS)
opcenter_username=$(get_secret OPCENTER_USERNAME)
opcenter_password=$(get_secret OPCENTER_PASSWORD)

# ------------------------------------------
# ---- ^^^ DO NOT EDIT THIS SECTION ^^^ ----
# ------------------------------------------

# ---- Nextflow Configuration File Creation ----
# This section creates a Nextflow configuration file with various settings for the pipeline execution.

# Use cat to create or overwrite the mmc.config file with the desired Nextflow configurations.
# NOTE: S3 keys and OpCenter information will be concatted to the end of the config file. No need to add them now

# Additionally, please add your STAGE MOUNT BUCKETS here
cat > mmc.config << EOF
// enable nf-float plugin.
plugins {
 id 'nf-float'
}

// Process settings: Executor, error strategy, and resource allocation specifics.
process {
 executor = 'float'
 errorStrategy = 'retry'
 extra = '--dataVolume [opts=" --cache-dir /mnt/jfs_cache "]jfs://${jfs_private_ip}:6868/1:/mnt/jfs --dataVolume [size=120]:/mnt/jfs_cache --vmPolicy [spotOnly=true,retryLimit=10,retryInterval=300s]'
}

// Directories for Nextflow execution.
workDir = '${workDir}'
launchDir = '${workDir}'

// OpCenter connection settings.
float {
 address = '${opcenter_ip_address}'
 username = '${opcenter_username}'
 password = '${opcenter_password}'
}

// AWS S3 Client configuration.
aws {
 client {
 maxConnections = 20
 connectionTimeout = 300000
 }
 accessKey = '${access_key}'
 secretKey = '${secret_key}'
}

EOF

# ---- Data Preparation ----
# Use this section to copy essential files from S3 to the working directory.

# For example, copy the sample sheet and params.yml from S3 to the current working directory.
# aws s3 cp s3://nextflow-input/samplesheet.csv .
# aws s3 cp s3://nextflow-input/scripts/params.yml .

# ---- Nextflow Command Setup ----
# Important: The -c option appends the mmc config file and soft overrides the nextflow configuration.

# Assembles the Nextflow command with all necessary options and parameters.
nextflow_command='nextflow run <nextflow-pipeline> \
-r <revision-number> \
-c mmc.config \
-params-file params.yml \
--input samplesheet.csv \
--outdir 's3://nextflow-output/rnaseq/' \
-resume '

# ---------------------------------------------
# ---- vvv DO NOT EDIT BELOW THIS LINE vvv ----
# ---------------------------------------------
# The following section contains functions and commands that should not be modified by the user.

# Create side script to tag head node - exits when properly tagged
cat > tag_nextflow_head.sh << EOF
#!/bin/bash

runname="\$(cat .nextflow.log 2>/dev/null | grep nextflow-io-run-name | head -n 1 | grep -oP '(?<=nextflow-io-run-name:)[^ ]+')"
workflowname="\$(cat .nextflow.log 2>/dev/null | grep nextflow-io-project-name | head -n 1 | grep -oP '(?<=nextflow-io-project-name:)[^ ]+')"

while true; do

 # Runname and workflowname will be populated at the same time
 # If the variables are populated and not tagged it, tag the head node
 if [ ! -z \$runname ]; then
 ./float modify -j "$(echo $FLOAT_JOB_ID)" --addCustomTag run-name:\$runname 2>/dev/null
 ./float modify -j "$(echo $FLOAT_JOB_ID)" --addCustomTag workflow-name:\$workflowname 2>/dev/null
 break
 fi

 runname="\$(cat .nextflow.log 2>/dev/null | grep nextflow-io-run-name | head -n 1 | grep -oP '(?<=nextflow-io-run-name:)[^ ]+')"
 workflowname="\$(cat .nextflow.log 2>/dev/null | grep nextflow-io-project-name | head -n 1 | grep -oP '(?<=nextflow-io-project-name:)[^ ]+')"

 sleep 1s

done
EOF

# Start tagging side-script
chmod +x ./tag_nextflow_head.sh
./tag_nextflow_head.sh &

# Start Nextflow run
$nextflow_command

if [[ $? -ne 0 ]]; then
 echo $(date): "Nextflow command failed."
 exit 1
else 
 echo $(date): "Nextflow command succeeded."
 exit 0
fi

```

</details>

### Job-Submit Script Adjustments

* Modify the `process.extra` within the `mmc.config` section to customize the `vmPolicy` for individual nextflow processes (Default policy is `spotOnly` in the job submit script. Adjust as needed e.g., `onDemand`, `spotFirst`). You may find more options when calling `float submit -h`:

```bash
--vmPolicy [spotOnly=true,retryLimit=10,retryInterval=300s]
```

For handling nextflow code repository, samplesheets and params-file, you have two options: download them or create them directly in the script. Here’s how to do both:

### Downloading Samplesheet and Params File

* Provide download commands for users to obtain samplesheet and params file for Nextflow, ensuring you replace `<download-link>` with the actual URLs:

```bash
# Download samplesheet
aws s3 cp s3://nextflow-input/samplesheet.csv .

# Download params file
aws s3 cp s3://nextflow-input/params.yml .
```

### Creating Samplesheet and Params File Directly in the Script

* Alternatively, users can create these files directly within the script using the `cat` command as shown below.

```bash
# Create samplesheet
cat > samplesheet.csv << EOF
sample,fastq_1,fastq_2,strandedness
AL_TO_rep01,s3://nextflow-input/Sample_1_L007_R1_001.fastq.gz,s3://nextflow-input/Sample_1_L007_R2_001.fastq.gz,auto
AL_TO_rep01,s3://nextflow-input/Sample_2_L008_R1_001.fastq.gz,s3://nextflow-input/Sample_2_L008_R2_001.fastq.gz,auto
AL_TO_rep01,s3://nextflow-input/Sample_3_L014_R1_001.fastq.gz,s3://nextflow-input/Sample_3_L014_R2_001.fastq.gz,auto
AL_TO_rep01,s3://nextflow-input/Sample_4_L009_R1_001.fastq.gz,s3://nextflow-input/Sample_4_L009_R2_001.fastq.gz,auto
EOF

# Create params file
cat > params.yml << EOF
multiqc_title: "rnaseq_multiqc"
fasta: "s3://nextflow-input/reference/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz"
gtf: "s3://nextflow-input/reference/Caenorhabditis_elegans.WBcel235.111.gtf.gz"
save_reference: true
remove_ribo_rna: true
skip_alignment: true
pseudo_aligner: "salmon"
EOF
```

---

* Download your nextflow code repository using `git clone` or copying from S3

> NOTE: if using private git repositories you can export your GITHUB_TOKEN before the nextflow_command

* Finally, ensure you customize your `nextflow_command` with specific pipeline requirements and save the changes:

```bash
nextflow_command='nextflow run <nextflow-pipeline> \
-r <revision-number> \
-c mmc.config \
-params-file params.yml \
--input samplesheet.csv \
--outdir 's3://nextflow-output/rnaseq/' \
-resume '
```

> Remember to replace placeholders with your specific pipeline details. 

---

## Float Submit

* Login to your MMCloud opcenter:

```bash!
float login -a <opcenter-ip-address> -u <user>
```
> Enter your password in the next prompt and you should see `Login succeded`

* Ensure you are using the latest version of OpCenter & float CLI:

```bash
float release upgrade --sync
```

* Make sure you have the following variables set as-is for `float secret`'s:

```bash
+-----------------------+
| NAME |
+-----------------------+
| OPCENTER_IP_ADDRESS |
| OPCENTER_USERNAME |
| OPCENTER_PASSWORD |
| AWS_BUCKET_ACCESS_KEY |
| AWS_BUCKET_SECRET_KEY |
+-----------------------+
```

* Useful `float secret` commands:

```bash
# to list stored secrets
float secret ls
# to set a secret
float secret set OPCENTER_IP_ADDRESS 192.0.1.2
# to unset a secret
float secret unset OPCENTER_IP_ADDRESS
```

### Float Submit Command

* Replace the placeholders `<work-bucket>`, `<region>`, `<security-group>`, and `<job-name>` with your specific values and execute the float submit command:

```bash!
float submit \
--hostInit transient_JFS_AWS.sh \
--hostTerminate hostTerminate_AWS.sh \
-i docker.io/memverge/juiceflow \
--vmPolicy '[onDemand=true]' \
--migratePolicy '[disable=true]' \
--dataVolume '[size=60]:/mnt/jfs_cache' \
--dirMap /mnt/jfs:/mnt/jfs \
-c 2 -m 4 \
-n <job-name> \
--securityGroup <security-group> \
--env BUCKET=https://<work-bucket>.s3.<region>.amazonaws.com \
-j job_submit_AWS.sh
```

Here's a brief explanation of the parameters used in the `float submit` command:

| Parameter | Brief Description |
|-----------|-------------------|
| `--hostInit transient_JFS_AWS.sh` | Shell script to run on the host before the job starts. |
| `--hostTerminate hostTerminate_AWS.sh` | Shell script to run on the host after the job has been cancelled |
| `-i docker.io/memverge/juiceflow` | Docker image for the job's software environment. |
| `--vmPolicy '[onDemand=true]'` | Uses on-demand VM instance for head-node execution. |
| `--migratePolicy '[disable=true]'` | Disables head-node migration to different hosts/VMs. |
| `--dataVolume '[size=60]:/mnt/jfs_cache'` | Attaches a 60GB data volume at `/mnt/jfs_cache` in the container. |
| `--dirMap /mnt/jfs:/mnt/jfs` | Maps a host directory to a container directory for data sharing. |
| `-c 2` | Allocates 2 CPU cores to the job. |
| `-m 4` | Allocates 4GB of memory to the job. |
| `-n <job-name>` | Assigns a name to the job for identification. |
| `--securityGroup <security-group>` | Applies a security group to the job's VM for network rules. |
| `--env BUCKET=https://<work-bucket>.s3.<region>.amazonaws.com` | Sets an environment variable for the S3 bucket URL. |
| `-j job_submit_AWS.sh` | Specifies the job script or command to run inside the container. |

---

### Mounting S3 buckets as Data Volumes

* you can mount the input data bucket using S3FS as a data volume on the Nextflow head node and worker nodes as follows:

```
--storage <storage-name>
```

* Read more on [Juiceflow Performance Optimization](https://www.mmcloud.io/resources/docs/juiceflow-performance-optimization-aws)


---

## Important Considerations

- **JuiceFS Bucket Requirements**: JuiceFS requires formatting at the root level of a storage bucket. It cannot be formatted on a sub-directory within a bucket. Ensure the root directory of the bucket, or the entire bucket, is specified in the command line interface (CLI) command.

---

### Cancelling a Running Workflow to Resume Later

There are many circumstances where you might need to cancel a running Nextflow workflow and resume it later using Nextflow's `-resume` feature, such as changes to the configuration or resource specifications.

* To cancel a workflow in the JuiceFlow setup, cancel the head node job. This will cause the head node host to generate a `<JOB_ID>.meta.json.gz`, where the JOB_ID represents the job-id of the head node file in the S3 work bucket. 

* You can find the `<JOB_ID>.meta.json.gz` and the status of metadata dump in the `job.events` log in `Attachments` of the head node job

* This `<JOB_ID>.meta.json.gz` file is critical for restoring the work directory for subsequent attempts using the environmental variable `PREVIOUS_JOB_ID`.

---

### Resuming Workflows with the Job-Submit Script

JuiceFlow supports the `-resume` option with Nextflow. Each workflow execution generates a `<JOB_ID>.meta.json.gz` file in the S3 work bucket. This file is critical for restoring the work directory for subsequent attempts using the environmental variable `PREVIOUS_JOB_ID` .

> Example command
```
float submit \
--hostInit transient_JFS_AWS.sh \
--hostTerminate hostTerminate_AWS.sh \
-i docker.io/memverge/juiceflow \
--vmPolicy '[onDemand=true]' \
--migratePolicy '[disable=true]' \
--dataVolume '[size=60]:/mnt/jfs_cache' \
--dirMap /mnt/jfs:/mnt/jfs \
-c 2 -m 4 \
-n <job-name> \
--securityGroup <security-group> \
--env PREVIOUS_JOB_ID=<PREVIOUS_JOB_ID> \
--env BUCKET=https://<work-bucket>.s3.<region>.amazonaws.com \
-j job_submit_AWS.sh
```
 
---


## Monitoring on OpCenter

### Workflow Execution Log

To monitor workflow execution and get a detailed view of each process in the Nextflow workflow:

* Click on the `Workflows` dashboard in the OpCenter to monitor workflow execution and get a detailed view for each process in the Nextflow workflow:

![](https://baywatch-api.memverge.com/media/upload/202406/240626214812100945.png)

* Click on the workflow name, and you can monitor the jobs running in this workflow in this consolidated view:

![](https://baywatch-api.memverge.com/media/upload/202406/240626214854210539.png)

* Once the head-node job starts `Executing`, you can monitor the Nextflow stdout by clicking on the job -> `Attachments` -> `stdout.autosave`:

![](https://baywatch-api.memverge.com/media/upload/202406/240626214616545791.png)

### Individual Job Logs

To view the execution log of any particular job:

* Click on the Job-ID.
* Navigate to the `Attachments` tab.
* Click `view` (eye-icon) on the `stdout.autosave` file.

![](https://baywatch-api.memverge.com/media/upload/202406/240626215440890739.png)

![](https://baywatch-api.memverge.com/media/upload/202406/240626215520824360.png)

---

## Create Job Templates to Launch via MMCLoud GUI

Job Templates allow for the ease and customaization of runs that follow a similar format, without having the need to manually set up a command every time. It requires the submission of one job first.

* From the `Jobs` dashboard, select the head node job previously submitted above and click on `More Actions` -> `Save as Template`:

![](https://baywatch-api.memverge.com/media/upload/202406/240626215758509556.png)

* Provide a name and tag for the private template:

![](https://baywatch-api.memverge.com/media/upload/202406/240626215842168590.png)

* Navigate to the `Job Templates` dashboard and click on `Private` templates:

![](https://baywatch-api.memverge.com/media/upload/202406/240626220028284894.png)

* You can click on any job template, edit/change samplesheet, variables etc., and submit new jobs from the GUI.

---

> * Users can also submit jobs from templates via CLI
>```bash!
>float submit --template private::<template-name>:<template-tag> \
>-e BUCKET=s3://<aws-jfs-bucket>
>```
> Addtionally, please keep in mind the features that need to be updated with every run if they deviate from the default values provided in the private template. This will mainly include:
> * S3 Bucket URL
> * Updating of the nextflow run command in the job script

---

# Persistent Nextflow JuiceFS

## Introduction to JuiceFS

[JuiceFS](https://juicefs.com/docs/community/introduction/) is an open-source, high-performance distributed file system designed specifically for cloud environments. It offers unique features, such as:

- **Separation of Data and Metadata**: JuiceFS stores files in chunks within object storage like Amazon S3, while metadata can be stored in various databases, including Redis.
- **Performance**: Achieves millisecond-level latency and nearly unlimited throughput, depending on the object storage scale.
- **Easy Integration with MMCloud**: MMCloud provides pre-configured nextflow head node templates with JuiceFS setup, simplifying deployment.
- **Comparison with S3FS**: For a detailed comparison between JuiceFS and S3FS, see [JuiceFS vs. S3FS](https://juicefs.com/docs/community/comparison/juicefs_vs_s3fs/). JuiceFS typically offers better performance and scalability.

---

## Pre-requisites for Using JuiceFS with Nextflow

Before you begin, ensure you meet the following pre-requisites:

- A VPC Security Group with inbound-rule for port `6868`.
 - **Navigation:** `AWS EC2 console -> Network & Security -> Security Groups`

![](https://baywatch-api.memverge.com/media/upload/202406/240626212749399541.png)

> **Inbound rules should include:**
 - Custom TCP over TCP on port `6868`, used by the Redis server in this juiceflow setup.

2. **Create a New S3 Bucket**:
 - Cloud storage is required for JuiceFS's backend storage.
 - Create a new S3 bucket and note its region, as this will be needed later.
 - Path: *AWS S3 console -> Create Bucket*
 - ![S3 Bucket Creation](https://baywatch-api.memverge.com/media/upload/202312/231208005848879261.png)

---

# Deployment Steps for Individual Users on MMCloud

## Float Login

Ensure you are using the latest version of `float` CLI:

```bash
sudo float release sync
```

Login to your MMCloud opcenter:

```bash
float login -a <opcenter-ip-address> -u <user>
```

> After entering your password, verify that you see `Login succeeded!`

## Float Secret

Set your AWS credentials as secrets in `float`:

```bash
float secret set AWS_BUCKET_ACCESS_KEY <BUCKET_ACCESS_KEY>
float secret set AWS_BUCKET_SECRET_KEY <BUCKET_SECRET_KEY>
```

To verify the secrets:

```bash
float secret ls
```

> **Expected output**:
>
> ```bash
> +-----------------------+
> | NAME |
> +-----------------------+
> | AWS_BUCKET_ACCESS_KEY |
> | AWS_BUCKET_SECRET_KEY |
> +-----------------------+
> ```

## Deploy Nextflow Head Node

Deploy the Nextflow head node using the `nextflow:jfs` template:

```bash
float submit -n <head-node-name> \
--template nextflow:jfs \
--securityGroup <sg-00000000000a> \
-e BUCKET=https://<bucket-name>.s3.<bucket-region>.amazonaws.com \
--env BUCKET_ACCESS_KEY={secret:AWS_BUCKET_ACCESS_KEY} \
--env BUCKET_SECRET_KEY={secret:AWS_BUCKET_SECRET_KEY}
```

> **Note**: Replace `<head-node-name>`, `<security-group>`, `<bucket-name>`, and `<bucket-region>` with your specific details. The `nextflow:jfs` template comes pre-configured with JFS setup.

---

### Overriding Template Defaults

#### Customizing CPU and Memory
To override default CPU and memory settings:

```bash
--overwriteTemplate "*" -c <number-of-cpus> -m <memory-in-gb>
```

> **Example**: To set 8 CPUs and 32GB memory
> ```bash
> --overwriteTemplate "*" -c 8 -m 32
> ```

#### Specifying a Subnet
For deploying in a specific AWS [subnet](https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html):

```bash
--overwriteTemplate "*" --subnet <SUBNET-ID>
```
---

### Mounting S3 buckets as Data Volumes

* you can mount the input data bucket using S3FS as a data volume on the Nextflow head node and worker nodes as follows:

```
--dataVolume [mode=r,accesskey=xxx,secret=xxx,endpoint=s3.REGION.amazonaws.com]s3://BUCKET_NAME:/staged-files
```

* Read more on [Juiceflow Performance Optimization](https://www.mmcloud.io/resources/docs/juiceflow-performance-optimization-aws)

---

#### Incremental Snapshot Feature (From v2.4)
Enables faster checkpointing and requires larger storage:

```bash
--overwriteTemplate "*" --dumpMode incremental
```

> **Advantages**:
> - Supports larger workloads
> - Lower impact on job running time
> - No need for periodic snapshot interval configuration
>
> **Disadvantages**:
> - Requires larger storage for delta saves
> - Final snapshot is necessary for restore

#### Checking Head Node Deployment Status

```bash
float list
```

> **Example Output**:
> ```bash
> +-----------------------+----------------+------------------------------------+---------+-----------+----------------------+------------+-------------+
> | ID | NAME | WORKING HOST | USER | STATUS | SUBMIT TIME | DURATION | COST |
> | NlygkM1dA0qIucncPwjgD | jfs-head-1 | 54.211.194.151 (2Core4GB/OnDemand) | sateesh | Executing | 2023-11-02T17:40:06Z | 6m54s | 0.0049 USD |
> +-----------------------+----------------+------------------------------------+---------+-----------+----------------------+------------+-------------+
> ```

#### SSH into Head Node

- Locate the public IP address of the head node in the `Working Host` column.
- Retrieve the SSH key from Float's secret manager:

```bash
float secret get <job-id>_SSHKEY > <head-node-name>-ssh.key
```

> **Note**: If you encounter a `Resource not found` error, wait a few more minutes for the head node and SSH key to initialize.

- Set the appropriate permissions for the SSH key:

```bash
chmod 600 <head-node-name>-ssh.key
```

---

## SSH to Nextflow JFS Head Node

SSH into the Nextflow head node using the provided SSH key, username, and the head node's public IP address:

```bash
ssh -i <head-node-name>-ssh.key <user>@<head-node-public-ip-address>
```

> Note: Use the username `nextflow` to login as admin

## MMC nf-float configuration

### Editing the configuration file

1. Copy the template and edit the configuration file:

```bash
cp mmcloud.config.template mmc-jfs.config
vi mmc-jfs.config
```

> **Note**: If you're new to using `vi`, check out this [Beginner's Guide to Vi](https://www.howtogeek.com/102468/a-beginners-guide-to-editing-text-files-with-vi/) for basic instructions.

2. The `mmc-jfs.config` file copied from the template will be pre-filled with the OpCenter IP address, and the PRIVATE IP address of the Nextflow head node. **You only need to provide your OpCenter username, password and AWS access and secret keys in the config**.


```bash
plugins {
 id 'nf-float'
}

workDir = '/mnt/jfs/nextflow'

process {
 executor = 'float'
 errorStrategy = 'retry'
 extra =' --dataVolume [opts=" --cache-dir /mnt/jfs_cache "]jfs://172.31.41.331:6868/1:/mnt/jfs --dataVolume [size=120]:/mnt/jfs_cache'
 /*
 For some special tasks like Qualimap which generates very small IO request, using this -o writeback_cache can help with performance. Here's an example.
 withName: "QUALIMAP_RNASEQ" {
 extra =' --dataVolume [opts=" --cache-dir /mnt/jfs_cache -o writeback_cache"]jfs://172.31.41.331:6868/1:/mnt/jfs --dataVolume [size=120]:/mnt/jfs_cache'
 }
 */
}

podman.registry = 'quay.io'

float {
 address = '172.31.42.11:443'
 username = '<your_user_name>'
 password = '<your_password>'
}

// AWS access info if needed
aws {
 client {
 maxConnections = 20
 connectionTimeout = 300000
 }
 accessKey = '<bucket_access_key>'
 secretKey = '<bucket_secret_key>'
}
```

### Using Tmux

Start a tmux session named `nextflow`:

```bash
tmux new -s nextflow
```

> To attach to an existing tmux session:
>
> ```bash
> tmux attach -t nextflow
> ```

> **Tip**: If you're new to `tmux`, here's a handy [Tmux Cheat Sheet](https://tmuxcheatsheet.com/).

### Nextflow Version Check

Check the Nextflow version and update if necessary:

```bash
nextflow -v
```

> **Example Output**:
>
> ```bash
> nextflow version 23.10.0.5889
> ```

---

## Launch Nextflow

Launch a Nextflow or `nf-core/<pipeline>` by providing the MMC config file:

```bash
nextflow run nf-core/<pipeline> \
-profile test_full \
-c mmc-jfs.config \
--outdir s3://nextflow-work-dir/<pipeline>
```

---

## Head Node Management

- In this persistent head node setup, user is responsible to dispose off the head node. To cancel the head node job click on the cancel button for the job as shown below

![](https://baywatch-api.memverge.com/media/upload/202406/240626221259289772.png)

![](https://baywatch-api.memverge.com/media/upload/202406/240626221324725332.png)


---

## FAQ

**Q**: Why does `nf-core/rnaseq` fail at the QUALIMAP step?

**A**: The QUALIMAP process in `nf-core/rnaseq` often fails due to its high-frequency, small-size write requests, leading to timeouts. Enabling `-o writeback_cache` consolidates these requests and improves performance significantly. However, it turns sequential writes into random writes, affecting sequential write performance. Use this setting only in scenarios with intensive random writes.

Add the following in the `process {}` scope of your config:

```bash
withName: "QUALIMAP_RNASEQ" {
 extra =' --dataVolume [opts=" --cache-dir /mnt/jfs_cache -o writeback_cache"]jfs://<head-node-private-ip>:6868/1:/mnt/jfs --dataVolume [size=120]:/mnt/jfs_cache'
 }
```

---

## Additional Reading

### Data Volumes

For jobs that generate file system I/O, specifying data volumes is essential. The OpCenter supports a variety of data volume types. Learn more about configuring data volumes in MMCloud in the [MMCloud Data Volumes Guide](https://docs.memverge.com/MMCloud/latest/User%20Guide/Working%20with%20OpCenter/datavolumes/).

### Allow / Deny Instance Types

Configure which instance types are allowed or denied in your setup. 

![Allow/Deny Instance Types](https://baywatch-api.memverge.com/media/upload/202312/231208013813787510.png)

See detailed docs on [Controlling Virtual Machine Types](https://docs.memverge.com/MMCloud/latest/User%20Guide/Working%20with%20OpCenter/allowlistforvm/)

---

# Instance Setup

1. Launch EC2 Instance. From the EC2 console, click on Launch Instance
![](https://baywatch-api.memverge.com/media/upload/202408/240802211531904303.png)

2. For AMI, search for amzn2-ami-kernel-5.10-hvm-2.0.20230628.0-x86_64-gp2
![](https://baywatch-api.memverge.com/media/upload/202408/240802211541058090.png)

3. For Instance type, choose t2.medium and provide key-pair
![](https://baywatch-api.memverge.com/media/upload/202408/240802211549358643.png)

4. For Network Setting select existing default security-group
![](https://baywatch-api.memverge.com/media/upload/202408/240802211556479325.png)

5. For Configure Storage, 8Gb of root volume is sufficient. Proceed to click on Launch Instance
![](https://baywatch-api.memverge.com/media/upload/202408/240802211604440390.png)



# MMC CLI Setup
Assuming your OpCenter is setup, to install the CLI:
```
wget https://<op_center_ip_address>/float --no-check-certificate
sudo mv float /usr/local/bin/
sudo chmod +x /usr/local/bin/float
```

Connect to your opcenter by loggin in:
```
float login -a <op_center_ip_address> -u <username> -p <password>
```

# Cromwell Setup
### Install Java

```
curl -s "https://get.sdkman.io" | bash
source "/home/ec2-user/.sdkman/bin/sdkman-init.sh"
sdk install java 17.0.6-tem
java -version
```
### Install Cromwell
```
wget https://github.com/broadinstitute/cromwell/releases/download/84/cromwell-84.jar

# Check version
java -jar cromwell-84.jar --version
```

# Config file
Name your file `cromwell-float.conf`. Make sure you update the address to your OpCenter address
```
# This is an example of how you can use Cromwell to interact with float.

backend {
 default = float

 providers {
 float {
 actor-factory = "cromwell.backend.impl.sfs.config.ConfigBackendLifecycleActorFactory"
 config {
 runtime-attributes="""
 String f_cpu = "2"
 String f_memory = "4"
 String f_docker = ""
 String f_extra = ""
 """

 # If an 'exit-code-timeout-seconds' value is specified:
 # - check-alive will be run at this interval for every job
 # - if a job is found to be not alive, and no RC file appears after this interval
 # - Then it will be marked as Failed.
 # Warning: If set, Cromwell will run 'check-alive' for every job at this interval
 exit-code-timeout-seconds = 30

 submit = """
 mkdir -p ${cwd}/execution
 echo "set -e" > ${cwd}/execution/float-script.sh
 echo "cd ${cwd}/execution" >> ${cwd}/execution/float-script.sh
 tail -n +22 ${script} > ${cwd}/execution/no-header.sh
 head -n $(($(wc -l < ${cwd}/execution/no-header.sh) - 14)) ${cwd}/execution/no-header.sh >> ${cwd}/execution/float-script.sh

 float submit -i ${f_docker} -j ${cwd}/execution/float-script.sh --cpu ${f_cpu} --mem ${f_memory} ${f_extra} > ${cwd}/execution/sbatch.out 2>&1
 cat ${cwd}/execution/sbatch.out | sed -n 's/id: $.*$/\1/p' > ${cwd}/execution/job_id.txt
 echo "receive float job id: "
 cat ${cwd}/execution/job_id.txt

 JOB_SCRIPT_DIR=float-jobs/$(cat ${cwd}/execution/job_id.txt)
 mkdir -p $JOB_SCRIPT_DIR
 cd $JOB_SCRIPT_DIR
 
# create the check alive script
cat <<EOF > float-check-alive.sh
SCRIPT_DIR=$(pwd)
cd ${cwd}/execution
float show -j \$1 --runningOnly > job-status.yaml
if [[ -s job-status.yaml ]]; then
 cat job-status.yaml
else
 float show -j \$1 | grep rc: | tr -cd '[:digit:]' > rc
 float log cat -j \$1 stdout.autosave > stdout
 float log cat -j \$1 stderr.autosave > stderr
fi
cd $SCRIPT_DIR
EOF

# create the kill script
cat <<EOF > float-kill.sh
SCRIPT_DIR=$(pwd)
cd ${cwd}/execution
float scancel -f -j \$1
cd $SCRIPT_DIR
EOF

 cat ${cwd}/execution/sbatch.out
 """

 kill = """
 source float-jobs/${job_id}/float-kill.sh ${job_id}
 """

 check-alive = """
 source float-jobs/${job_id}/float-check-alive.sh ${job_id}
 """
 
 job-id-regex = "id: (\\w+)\\n"
 }
 }
 }
}

```
# S3 Bucket Setup
Follow the directions in this section: [Setup Nextflow host on AWS - HackMD](https://hackmd.io/@speri/rkmPkmP52#3-Create-Nextflow-work-directory-S3-bucket)

Once you made your bucket and created your access keys, install s3fs:
```
sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh
./configure --prefix=/usr --with-openssl
make
sudo make install
```
Create a password file with your access key and secret key in the form of
```
access_key:secret_key
```
Change mode your file to 600
```
chmod 600 ./passwd-s3fs
```
Mount your bucket to your designated mountpoint. If you are mounting to a directory that requires root privileges to access, you will need to use `sudo` to mount
```
s3fs BUCKET /MOUNTPOINT -o rw,allow_other -o multipart_size=52 -o parallel_count=30 -o passwd_file=~/.passwd-s3fs
```
If you plan on using an s3 bucket with your workflow, please update the `f_extra` line in the config:
```
String f_extra = "--dataVolume [accesskey=XXX,secret=XXX,mode=rw]s3://BUCKET:/MOUNTPOINT"
```

# Hello World (Read from Bucket)
### Create `hello.wdl`
```
workflow helloWorld {
 String name
 call sayHello { input: name=name }
}

task sayHello {
 String name

 command {
 printf "[cromwell-say-hello] hello to ${name} on $(date)\n"
 sleep 30
 }
 output {
 String out = read_string(stdout())
 }
 runtime {
 f_docker: "cactus"
 }
}
```
### Create `hello.json` in your bucket
```
{
 "helloWorld.name": "Developer"
}
```
### Run command (edit for your corresponding mountpoint)
```
java -Dconfig.file=cromwell-float.conf -jar \
 cromwell-84.jar run hello.wdl \
 --inputs /MOUNTPOINT/hello.json
```
A successful workflow will end in something similar to the snippet below. You may ignore the text that appear afterwards. The most important part is the “Succeeded”
```
[INFO] [11/27/2023 18:15:23.336] [cromwell-system-akka.dispatchers.engine-dispatcher-30] [akka://cromwell-system/user/SingleWorkflowRunnerActor] SingleWorkflowRunnerActor workflow finished with status 'Succeeded'.
{
 "outputs": {
 "helloWorld.sayHello.out": "[cromwell-say-hello] hello to Developer on Mon Nov 27 18:12:55 UTC 2023"
 },
 "id": "a1f0606e-367f-43a5-9381-e4ebe09ffbcf"
}
[2023-11-27 18:15:24,91] [info] Workflow polling stopped
```

# Sequence Workflow (Write to Bucket)
### Create `seq.wdl` (edit for your corresponding mountpoint)
```
workflow myWorkflow {
 call sayHello
 call writeReadFile { input: s=sayHello.out }
}

task sayHello {
 command {
 printf "[cromwell-say-hello] hello from $(whoami) on $(date)"
 sleep 30
 }
 output {
 String out = read_string(stdout())
 }
 runtime {
 f_docker: "cactus"
 f_cpu: "2"
 f_memory: "4"
 }
}

task writeReadFile {
 String s

 command {
 printf "[cromwell-write-read-file] write input to a file: ${s}\n" > /MOUNTPOINT/my_file.txt
 cat /MOUNTPOINT/my_file.txt
 }
 output {
 String out = read_string(stdout())
 }
 runtime {
 f_docker: "cactus"
 }
}
```
### Command
```
java -Dconfig.file=cromwell-float.conf -jar cromwell-84.jar run seq.wdl
```
You should expect to see the `my_file.txt` created and populated in your bucket

**Deploying MMCloud OpCenter in AWS Non-default VPC**
By Hui Chen, Yuchen Liu, Jing Gong

When EC2 resources are provisioned for the first time in an AWS account, a Default VPC is created automatically. In the default VPC, an Internet gateway and public subnets with corresponding routing tables are pre-configured; a public IP address are assigned to VMs by default; and access to the Internet is enabled automatically. As such, deploying MMCloud OpCenter in AWS default VPC is straightforward if the security posture of the default VPC is acceptable. However, in enterprise environments, it is often advantageous and/or necessary to run MMCloud workloads in a dedicated non-default VPC. Such set up allows granular security enforcement and ensures that MMCloud does not affect production environments. 

Deploying MMCloud OpCenter in a non-default VPC requires careful planning so as to ensure proper communication between the various MMCloud components. In a non-default VPC, customers may have both public IP subnets and private IP subnets. Public IP assignment normally is disabled for VMs, and most likely access to the internet will be disabled in the private subnets. For this kind of environment some extra cloud services need to be configured for MMCloud to work properly.

This document presents the procedure on how to deploy MMCloud OpCenter in AWS using a dedicated non-default VPC with typical enterprise security restrictions:

Step 1: Create VPC for MMCloud

From VPC dashboard, click on VPCs, then select Create VPC. Configure VPC settings as follows:
1.1	Resources to create: select VPC and more.
1.2	Name tag auto-generation: keep Auto-generate default set to be enabled; input a customized name tag.
1.3	IPv4 CIDR block: set the CIDR block addresses for the VPC.
1.4	IPv6 CIDR block: keep the default selection for No IPv6 CIDR block.
1.5	Tenancy: keep Default.
1.6	Number of Availability Zones (AZs): select 2 so as for the high availability.
1.7	Number of public subnets: select 2.
1.8	Number of private subnets: select 2 or 4 according to your needs.
1.9	NAT gateways ($): make selection according to your needs, None by default.
1.10	VPC endpoints: select S3Gateway.
1.11	DNS options: keep the default settings for Enable DNS hostnames and Enable DNS resolution. 
1.12	Click on Create VPC to create a new nondefault VPC.
 
![](https://baywatch-api.memverge.com/media/upload/202402/240227005041509145.png)
 
Step 2: Deploy MMCloud OpCenter in a public subnet of the newly created VPC. 

Follow Quick Start Deploy MMCloud OpCenter on AWS to set up the MMCloud in a public subnet of the VPC created at Step 1. 

This step allows users to access OpCenter from the public Internet, and allows OpCenter to communicate with the MMCloud License Server over the Internet. 

Step 3: Create EC2 endpoints for private subnets.

This step allows the OpCenter as well as the MMCloud worker nodes to access AWS services. 

From VPC dashboard, click on Endpoints, then select Create endpoint. Configure Endpoint settings as follows:
3.1 Name tag: input a customized name tag.
3.2 Service category: click on AWS services; search AWS service name of EC2 service, select it for the endpoint service.
3.3 VPC: select the VPC created at Step 1 as the VPC in which to create your endpoint.
3.4 Subnets: for all the availability zones in the VPC, select the private subnet IDs.
3.5 IP address type: keep the default IPv4 setting.
3.6 Security groups: select default group and two more security groups created for your MMCloud OpCenter and WorkerNodes.
3.7 Policy: Keep default Full access selection.
3.8 Tags: keep the default setting.
3.9 Click on Create endpoints to create a new EC2 instance endpoint. Wait until the status of the endpoint be fully ready.
3.10 Open OpCenter GUI. ((It may take several minutes for opcenter to be ready.)
 
![](https://baywatch-api.memverge.com/media/upload/202402/240227005055368761.png)

If a worker node needs to access Internet resources, you may need to create a NAT gateway. This is not necessary if a worker node only needs to access AWS services.

Below are the IAM roles and policies created when deploying Memory Machine Cloud through AWS Marketplace, via the CloudFormation script. 

There are two types of objects, OpCenter and WorkerNode, and Memory Machine Cloud will create different IAM roles for them.

### For OpCenter
- license-manager:*  (to query the license in license manager)
- pricing:* (to query the cloud resource price)
- IAM roles: (these are limited in the local AWS account)
iam:GetRole
iam:AttachRolePolicy
iam:CreateRole
iam:PutRolePolicy
iam:PassRole
iam:CreateServiceLinkedRole
- EC2 roles: (to allow Memory Machine Cloud to create and control an EC2 instance, for those delete related action, Memory Machine Cloud limited to the resource which created by Memory Machine Cloud)
ec2:Get*
ec2:Describe*
ec2:RunInstances
ec2:StartInstances
ec2:StopInstances
ec2:Create*
ec2:Modify*
ec2:AttachVolume
ec2:DetachVolume
ec2:DeleteVolume
ec2:DeleteSecurityGroup
ec2:TerminateInstances
ec2:DeleteSnapshot
- S3 roles: (to allow Memory Machine Cloud to access the customer data which is in the local AWS account’s s3 bucket or public buckets)
s3:DeleteObject
s3:Put*
s3:Replicate*
s3:Restore*
s3:CreateBucket
s3:DeleteBucket
s3:Update*
s3:List*
s3:Get*
s3:Describe*
- ECR roles: (to allow Memory Machine Cloud to access the customer’s ECR repo)
ecr:GetDownloadUrlForLayer
ecr:BatchGetImage
ecr:DescribeImages
ecr:ListImages
ecr:GetAuthorizationToken
ecr:BatchCheckLayerAvailability
- Marketplace roles: (to allow Memory Machine Cloud to push the Metering Usage)
aws-marketplace:MeterUsage

### For WorkerNode, the role is the subset of the OpCenter
- IAM roles:
iam:PassRole
- EC2 roles:
ec2:Get*
ec2:Describe*
ec2:StopInstances
ec2:Create*
ec2:Modify*
ec2:AttachVolume
ec2:DetachVolume
ec2:DeleteVolume
ec2:TerminateInstances
ec2:DeleteSnapshot
- S3 roles: (to allow Memory Machine Cloud to access the customer data which is in the local AWS account’s s3 bucket or public buckets)
s3:DeleteObject
s3:Put*
s3:Replicate*
s3:Restore*
s3:Update*
s3:List*
s3:Get*
s3:Describe*
- ECR roles: (to allow Memory Machine Cloud to access the customer’s ECR repo)
ecr:GetDownloadUrlForLayer
ecr:BatchGetImage
ecr:DescribeImages
ecr:ListImages
ecr:GetAuthorizationToken
ecr:BatchCheckLayerAvailability

## Introduction

MMcloud provides a checkpoint solution that: 
* captures the complete application state, both memory and file system
* is transparent to the application, ie. no application change is needed
* causes minimal overhead, ie. second-level wall clock time per checkpoint
* is automated after integration with the customer orchestration environment.

MM checkpoint protects the application progress, which enables applications to be migrated across both compute nodes and execution time. The specific use cases include:
* Reduced operational costs, by replacing expensive On-demand VM instances with cheaper but less reliable Spot VM instances
* Faster time to result, by capturing the application states in execution and sharing progress in later pipeline stages or with the team members
* Higher resource utilization, by dynamically migrating applications to the right-sized VM instances or scheduling to run only in off-peak times

![](https://baywatch-api.memverge.com/media/upload/202407/240719133501300304.png)

---

Types of file systems supported for storing checkpoint data:
* AWS EFS
* AWS FsX Lustre
* Juicefs with AWS S3

---

## Steps

### Nextflow Snapshot Engine AMI

* We will provide an AMI customized for nextflow execution on AWS Batch with MMC Snapshot Engine built in.

* We can also provide the snapshot engine release bin file to customize the AMI even further based on customer requirements

---

### Set up metadata database for JuiceFS

* There are a number of ways to set up a database server for JuiceFS, specifically for a distributed Juicefs. Options include Amazon Elastic Cache, MemoryDB, and Amazon RDS.

* This quick-guide uses Amazon RDS (if you are not using this option, skip to the Create Launch Template after you have created a metadata server with the other options). 

* For this option, you must create a database within your RDS before using. This is a one-time thing. You can spin up a new instance and follow these steps:
 * Install JuiceFS
 * [Install mysql locally](https://dev.to/aws-builders/installing-mysql-on-amazon-linux-2023-1512)
 * Connect to your metadata server in this format
```
mysql -h tutorial-database-1-instance-1.cepmeznhgaxt.us-east-1.rds.amazonaws.com -P 3306 -u admin -pPASSWORD
```
> There is no space between the -p and your password (from RDS setup)

 * Create a database. This example names their database juicefs
```
create database juicefs; 
```
 * Then, exit the mysql
```
exit 
```

---

### Create EC2 Launch Template

* After your AMI is created, create a launch template. The core components are:
 * New AMI
 * SSH Key
 * User Data
* Format and Mount JuiceFS, which requires
 * Bucket
 * Access Keys
 * Checkpoint directory (same one used in packer setup)
 * Metadata server address, username, password, and database name

* Below is an example of the User Data. Make sure it is in proper MIME format.

```
Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0

--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"

#cloud-config
cloud_final_modules:
- [scripts-user, always]

--//
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"

#!/bin/bash
juicefs format --storage s3 --bucket https://<BUCKET_NAME>.s3.us-east-1.amazonaws.com 'mysql://<USERNAME>:<PASSWORD>@(<METADATA_ADDRESS>:3306)/<DATABASE_NAME>' myjfs-rds --access-key xxx --secret-key yyy
sudo mkdir -p /memverge-mm-checkpoint
sudo chmod 777 /memverge-mm-checkpoint
juicefs mount 'mysql://<USERNAME>:<PASSWORD>@(<METADATA_ADDRESS>:3306)/<DATABASE_NAME>' /memverge-mm-checkpoint &
--//--
```

---

### AWS Batch Compute Environment


* Amazon EC2 configuration
* Instance role - ecsInstanceRole (should already exist in your account)
* Service role - AWSServiceRoleForBatch (should already exist in your account)
* Default network setup

![](https://baywatch-api.memverge.com/media/upload/202407/240718181446372712.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718181724633400.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718181842065561.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718182133866276.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718182259238061.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718182405885118.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718182547415064.png)




### Job Queue

* Amazon EC2 configuration
* Connect to previously-made compute environment

![](https://baywatch-api.memverge.com/media/upload/202407/240718182713516508.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718182856447540.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718183018644779.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718183108271338.png)

---

### Nextflow 

`aws.config`

```
plugins {
 id 'nf-amazon'
}
process {
 executor = 'awsbatch'
 queue = 'aws-spot-q'
 maxRetries = 2
}

process.containerOptions = '--env MMC_CHECKPOINT_INTERVAL=5m --env MMC_CHECKPOINT_IMAGE_SUBPATH=nextflow --env MMC_CHECKPOINT_MODE=true --env MMC_CHECKPOINT_IMAGE_PATH=/memverge-mm-checkpoint'

aws {
 accessKey = 'access
 secretKey = 'secret'
 region = 'us-east-1'
 client {
 maxConnections = 20
 connectionTimeout = 10000
 uploadStorageClass = 'INTELLIGENT_TIERING'
 storageEncryption = 'AES256'
 }
 batch {
 cliPath = '/nextflow_awscli/bin/aws'
 maxTransferAttempts = 3
 delayBetweenAttempts = '5 sec'
 }
}

```

## Environmental Variables

* `MMC_CHECKPOINT_INTERVAL = 5m` (default 15m)
* `MMC_CHECKPOINT_IMAGE_SUBPATH = checkpoint_dir`
> This is an optional variable that creates a subdirectory under your MMC_CHECKPOINT_IMAGE_PATH. Not having this variable will put all dumps into the MMC_CHECKPOINT_IMAGE_PATH.
* `MMC_CHECKPOINT_MODE = true`
* `MMC_CHECKPOINT_IMAGE_PATH = /memverge-mm-checkpoint`

---

`nextflow run command`

```
nextflow run sateeshperi/bwa-index -c aws.config -work-dir 's3://workdir-2'
```

---

### Run, Watch Logs, and Spot Interrupt

When the job is submitted, you are waiting for the Log stream name section to be populated, as well as your job to be in a Running state.


`Before spot Interrupt`
![](https://baywatch-api.memverge.com/media/upload/202407/240719133243470784.png)

`After spot Interrupt`
![](https://baywatch-api.memverge.com/media/upload/202407/240719133307081617.png)

* MMCloud's Juiceflow solution provides a high-performance method for running Nextflow pipelines on the cloud by leveraging JuiceFS to optimize cloud storage for the work directory.

> Read more about Juiceflow in a blog [here](https://www.mmcloud.io/blog/juiceflow-a-next-generation-solution-for-nextflow)

* [JuiceFS](https://juicefs.com/docs/community/introduction/) is a general-purpose, distributed file system compatible with any application. In the current MMCloud release, JuiceFS can only be used with a Nextflow host deployed using the OpCenter's built-in Nextflow job template.

* While cloud storage formatted with JuiceFS offers high performance for work directories, input data often needs to be staged in the work directory. 

* Nextflow natively supports staging or even streaming inputs from S3 to the executors of the respective process steps. By defining the input S3 location as a Channel, the Nextflow framework ensures the data is available for the task.

* However, with a large number of files, staging can become a bottleneck as the files must first be downloaded to the stage subdirectory in the work directory and then made available to the worker nodes for execution.

* To avoid this staging bottleneck, input data in cloud storage can be mounted as data volumes, making the files available locally and bypassing the staging process.

* Essentially, you can register storage and mount the storage bucket using S3FS as a data volume on the Nextflow head node and worker nodes as `--storage <storage-name>`

---

## Steps:

> See the official [Juiceflow Quick Guide](https://www.mmcloud.io/resources/docs/juiceflow-aws) for introduction on how to run nextflow pipelines on MMCloud

### Register Storage

> Available from float `v3.0.0-69ce0c9-Imperia` onwards

* In the OpCetner left navigation bar, click on `Storage`

![](https://baywatch-api.memverge.com/media/upload/202408/240812050212519294.png)

* Click on `Register Storage` button

![](https://baywatch-api.memverge.com/media/upload/202408/240812050106954736.png)


* Select `Storage Type`

| Volume | col | col |
| - | - | - |
| NFS | content | content |
| Lustre | content | content |
| S3 | content | content |
| OSS | content | content |
| GS | content | content |

![](https://baywatch-api.memverge.com/media/upload/202408/240812050418968825.png)

* Provide a `name` for the storage, `S3 URI`, `endpoint`, `access key`, `secret key`, `mount-point` of the bucket and choose `Access-Mode` - `Read` only or `Read Write`

![](https://baywatch-api.memverge.com/media/upload/202408/240812051142317705.png)

### Using storage arguments in JuiceFlow

1. In your job script, add the input bucket as `--storage <storage-name>` in the `process.extra` section of `mmc.config`:

```
process {
 executor = 'float'
 errorStrategy = 'retry'
 extra = '--dataVolume [opts=" --cache-dir /mnt/jfs_cache "]jfs://${jfs_private_ip}:6868/1:/mnt/jfs --dataVolume [size=120]:/mnt/jfs_cache --storage <storage-name-1> --storage <storage-name-2>'
}
```

2. Modify your sample sheet to now read from the storage mount point defined while registering storage `</storage-mount-point>`:

```
group_id,subject_id,sample_id,sample_type,sequence_type,filetype,filepath
COLO829_Full,COLO829,COLO829T,tumor,dna,bam,/storage-mount-point/COLO829v003T.bam
COLO829_Full,COLO829,COLO829T,tumor,dna,bai,/storage-mount-point/COLO829v003T.bam.bai
COLO829_Full,COLO829,COLO829R,normal,dna,bam,/storage-mount-point/COLO829v003R.bam
COLO829_Full,COLO829,COLO829R,normal,dna,bai,/storage-mount-point/COLO829v003R.bam.bai
```

3. For the head node, add the `--storage <storage-name>` variable to the float submit command:

```
float submit \
--hostInit transient_JFS_AWS.sh \
--hostTerminate hostTerminate_AWS.sh \
-i docker.io/memverge/juiceflow \
--vmPolicy '[onDemand=true]' \
--migratePolicy '[disable=true]' \
--dataVolume '[size=60]:/mnt/jfs_cache' \
--storage <storage-name-1> \
--storage <storage-name-2> \
--dirMap /mnt/jfs:/mnt/jfs \
-c 2 -m 4 \
-n <job-name> \
--securityGroup <security-group> \
--env BUCKET=https://<work-bucket>.s3.<region>.amazonaws.com \
-j job_submit_AWS.sh
```

This setup ensures efficient data handling by reading input data from input buckets using S3FS as staged data volume, while the work buckets leverage JuiceFS for high performance.

---

## Using JuiceFS SnapLocation for better efficiency of checkpoint/restore

* MMC can use either local EBS volumes or JuiceFS formatted S3 buckets for storing snapshot data

* From the left navigation bar of the opcenter, click on `System Settings`

![](https://baywatch-api.memverge.com/media/upload/202408/240812052246117983.png)

* In System Settings, click on `Cloud` settings and go the `Snapshot Location` field


![](https://baywatch-api.memverge.com/media/upload/202408/240812052423317064.png)

* Provide the S3 bucket URL in the following format making sure to include the `accesskey`, `secretkey` and `mode=rw` 

```
[accesskey=<access>,secret=<secret>,mode=rw]s3://preview-opcenter-jfs-snaplocation
```

> **NOTE: THE S3 bucket provided for snaplocation as above will be used to store the snapshot data of all jobs from all users in the opcenter and the data is cleared out automatically after a job finished**

AWS Known Issues

Known Issues-1: AWS Rebalance Recommendation Signal