# Setup MMC OpCenter on AWS


## Setup & Download EC2 Key-Pair for OpCenter Access

* Login to `AWS` & navigate to the `EC2` console and choose the appropriate region

![](https://baywatch-api.memverge.com/media/upload/202406/240626233908180703.png)

* Navigate to `Network & Security` from the left navigation bar and click on `Key Pairs`.

![](https://baywatch-api.memverge.com/media/upload/202406/240626234021897622.png)

* Create a key-pair if you don't have one and save this key. **NOTE: You will need this key only to SSH into the opcenter instance.** 

![](https://baywatch-api.memverge.com/media/upload/202408/240807081513238257.png)

---

## AWS Marketplace - Memory Machine Cloud Subscription

* Next, navigate to `AWS Marketplace` and search for `memory machine cloud` and click on `Continue to Subscribe`

![](https://baywatch-api.memverge.com/media/upload/202408/240807071354851639.png)

![](https://baywatch-api.memverge.com/media/upload/202408/240807071450108428.png)

* Next click on `Continue to Configuration`

![](https://baywatch-api.memverge.com/media/upload/202408/240807071628691257.png)

* Make sure the region is correct and leave the rest to defaults and click on `Continue to Launch`

![](https://baywatch-api.memverge.com/media/upload/202408/240807071801687942.png)

* Next choose `Launch CloudFormation` from the drop down box and click on the `Launch` button

![](https://baywatch-api.memverge.com/media/upload/202408/240807071930819017.png)

* You will be re-directed to a new window to create a stack. Use defaults on this page for tempaltes and click `Next`

![](https://baywatch-api.memverge.com/media/upload/202408/240807072107784401.png)

* Next, specify a stack name

![](https://baywatch-api.memverge.com/media/upload/202408/240807072307600844.png)

* Next, select an instance type for MMCE OpCenter under `OpCenterType`

![](https://baywatch-api.memverge.com/media/upload/202408/240807072442790376.png)

| Type | CPU | Memory(GB) | Price/hour | Guidance |
| - | - | - | - | - |
| POC | 2 | 2 | $0.0188 | All historic jobs < 13K| 
| Small | 2 | 4 | $0.0765 | All historic jobs < 26K |
| Medium | 4 | 8 | $0.1530 | All historic jobs < 80K |
| Large | 4 | 16 | $0.2016 | All historic jobs < 185K |

* If you prefer a `CustomizedInstanceType` specify other instance type for MMCE OpCenter. Note this field is **OPTIONAL**.

![](https://baywatch-api.memverge.com/media/upload/202408/240807072958088594.png)

* Next specify disk type for MMCE OpCenter. Available types are `gp2` & `gp3`. Choose default `gp3` for better performance.

![](https://baywatch-api.memverge.com/media/upload/202408/240807073248757580.png)


* Next, select an EC2 Key Pair for all instances

![](https://baywatch-api.memverge.com/media/upload/202408/240807073435594906.png)

* Next, select a web-accessible VPC to run MMCE service

![](https://baywatch-api.memverge.com/media/upload/202408/240807073559749009.png)

* Next, select a subnet in the VPC to run OpCenter instance

![](https://baywatch-api.memverge.com/media/upload/202408/240807073712437070.png)

* Next, select the Availability Zone in which the subnet resides

![](https://baywatch-api.memverge.com/media/upload/202408/240807073830333880.png)

* Next, select `True` if you want to assign public IP to MMCE service

![](https://baywatch-api.memverge.com/media/upload/202408/240807073938118039.png)

* Next, define the IP address range to access MMCE service under `ExternalAccessCidr` & `SshCidr`.

>  `0.0.0.0/0` to allow all access

![](https://baywatch-api.memverge.com/media/upload/202408/240807074118592777.png)

* Next, select `True` if you want to enable worker nodes to access other nodes internally. (Default is `False`)

![](https://baywatch-api.memverge.com/media/upload/202408/240807074355912386.png)

* Next, For `Configure stack options` options keep all defaults on this page and click `Next`

![](https://baywatch-api.memverge.com/media/upload/202408/240807074626143844.png)

* Next on the `Review and create` page, review all the parameters and finally select the check-box to allow CloudFormation to create resources and click on `Submit`

![](https://baywatch-api.memverge.com/media/upload/202408/240807074750728081.png)

* Next, you will see the stack creation in progress

![](https://baywatch-api.memverge.com/media/upload/202408/240807075129129926.png)

* Successful creation of stack will look as shown below with status `CREATE_COMPLETE`

![](https://baywatch-api.memverge.com/media/upload/202408/240807075426259908.png)

* Click on the `Resources` tab and click on the `mvOpCenter` id to re-direct to the EC2 instance assigned to the MMC OpCenter

![](https://baywatch-api.memverge.com/media/upload/202408/240807075508326039.png)

* Next, click on the `Instance ID` for more details of the instance

![](https://baywatch-api.memverge.com/media/upload/202408/240807075659076667.png)

* Next, click on the open address link of `Public IPv4 address` to be re-directed to the OpCenter

![](https://baywatch-api.memverge.com/media/upload/202408/240807075735012857.png)

> NOTE: When logging in for the first time to the opcenter, you might see a warning about invalid security certificate. Click on `Advanced` -> `Accept the Risk and Continue`

![](https://baywatch-api.memverge.com/media/upload/202408/240807075838231038.png)

* The MMCloud OpCenter landing page will look as below.

> The opcenter can now be accessed via its public IP address: `https://54.295.216.250/#/login`
> Bookmark this link for easy access

![](https://baywatch-api.memverge.com/media/upload/202408/240807080247340972.png)

> NOTE:
> default username: `admin`
> default password: `memverge`

---

# Change Admin Password

* We recommend changing the default username (`admin`) and password (`memverge`) for the opcenter after logging in for the first time.

* Click on `Users & Groups` from the left navigation menu and for the admin user click on `Actions` and change the password as necessary.

![](https://baywatch-api.memverge.com/media/upload/202408/240807080723259525.png)

![](https://baywatch-api.memverge.com/media/upload/202408/240807080855526736.png)

---

# Activate MMCloud License

* Next, activate your mmcloud license by clicking on the star icon in the top-right corner and enter your mmcloud account credentials

> Not registered? Visit www.mmcloud.io

![](https://baywatch-api.memverge.com/media/upload/202408/240807081016728068.png)

---

# Download FLOAT CLI

* The float CLI binary helps connect a user from any local machine to the MMCloud opcenter. To download the CLI, click on the terminal icon towards the top right corner of the opcenter and choose the version of CLI tool based on the operating system

![](https://baywatch-api.memverge.com/media/upload/202402/240207015753981576.png)

* From macOS 10.15 (Catalina) onwards, zsh is the default shell. In zsh, float is a reserved word. To use the float CLI with macOS, either change the shell to bash or alias the word float to the float binary, as follows:

```bash=
alias float = /path/to/float_binary/float
```
> where `/path/to/float_binary/` is the path to where you placed the float binary.

* To change the MAC-OS default shell from `zsh` to `bash` see the following [guide](https://phoenixnap.com/kb/change-zsh-to-bash-mac)

---

# Juiceflow - AWS

## Introduction

Juiceflow combines JuiceFS and Nextflow on MMCloud, offering a powerful, scalable solution for managing and executing workflows in the cloud.

<details>
<summary>Expand for more details on JuiceFS</summary>

[JuiceFS](https://juicefs.com/docs/community/introduction/) is an open-source, high-performance distributed file system designed specifically for cloud environments. It offers unique features, such as:

* Separation of Data and Metadata: JuiceFS stores files in chunks within object storage like Amazon S3, while metadata can be stored in various databases, including Redis.
* Performance: Achieves millisecond-level latency and nearly unlimited throughput, depending on the object storage scale.
* Easy Integration with MMCloud: MMCloud provides pre-configured nextflow head node templates with JuiceFS setup, simplifying deployment.
* Comparison with S3FS: For a detailed comparison between JuiceFS and S3FS, see [JuiceFS vs. S3FS](https://juicefs.com/docs/community/comparison/juicefs_vs_s3fs/). JuiceFS typically offers better performance and scalability.


</details>

---

## Pre-requisites

- A VPC Security Group with inbound-rule for port `6868`.
  - **Navigation:** `AWS EC2 console -> Network & Security -> Security Groups`

![](https://baywatch-api.memverge.com/media/upload/202406/240626212749399541.png)

> **Inbound rules should include:**
    - Custom TCP over TCP on port `6868`, used by the Redis server in this juiceflow setup.

- AWS S3 buckets for Nextflow work and output directories.
- AWS S3 access, secret keys & MMC OpCenter Credentials stored as float secrets

---

## Overview of the Setup

This solution leverages two scripts:
- `transient_JFS_AWS.sh`: Formats the work directory S3 bucket to JuiceFS format.
- `hostTerminate_AWS.sh`: Allows for graceful exit of workflow from nextflow head-node
- `job_submit_AWS.sh`: Contains Nextflow input parameters and MMC config.

---

## Steps

### Download Scripts

* Host-init script (you don't have to edit this script, but you'll need it later)

```bash!
wget https://mmce-data.s3.amazonaws.com/juiceflow/v1/aws/transient_JFS_AWS.sh
```

* Host-terminate script (you don't have to edit this script, but you'll need it later)

```bash!
wget https://mmce-data.s3.amazonaws.com/juiceflow/v1/aws/hostTerminate_AWS.sh
```

* Job-submit script (Download the template or create one locally based on the template below with your Nextflow inputs and run configurations)

```bash!
wget https://mmce-data.s3.amazonaws.com/juiceflow/v1/aws/job_submit_AWS.sh
```

<details>
<summary>Expand to view a sample `job_submit_AWS.sh` script</summary>

```bash!
#!/bin/bash

# ---- User Configuration Section ----
# These configurations must be set by the user before running the script.

# ---- Optional Configuration Section ----
# These configurations are optional and can be customized as needed.

# JFS (JuiceFS) Private IP: Retrieved from the WORKER_ADDR environment variable.
jfs_private_ip=$(echo $WORKER_ADDR)

# Work Directory: Defines the root directory for working files. Optional suffix can be added.
workDir_suffix=''
workDir='/mnt/jfs/'$workDir_suffix
mkdir -p $workDir  # Ensures the working directory exists.
cd $workDir  # Changes to the working directory.
export NXF_HOME=$workDir  # Sets the NXF_HOME environment variable to the working directory.

# ------------------------------------------
# ---- vvv DO NOT EDIT THIS SECTION vvv ----
# ------------------------------------------

function install_float {
  # Install float
  local address=$(echo "$FLOAT_ADDR" | cut -d':' -f1)
  wget https://$address/float --no-check-certificate --quiet
  chmod +x float
}

function get_secret {
  input_string=$1
  local address=$(echo "$FLOAT_ADDR" | cut -d':' -f1)
  secret_value=$(./float secret get $input_string -a $address)
  if [[ $? -eq 0 ]]; then
    # Have this secret, will use the secret value
    echo $secret_value
    return
  else
    # Don't have this secret, will still use the input string
    echo $1
  fi
}

# Set Opcenter credentials
install_float 
access_key=$(get_secret AWS_BUCKET_ACCESS_KEY)
secret_key=$(get_secret AWS_BUCKET_SECRET_KEY)
export AWS_ACCESS_KEY_ID=$access_key
export AWS_SECRET_ACCESS_KEY=$secret_key

opcenter_ip_address=$(get_secret OPCENTER_IP_ADDRESS)
opcenter_username=$(get_secret OPCENTER_USERNAME)
opcenter_password=$(get_secret OPCENTER_PASSWORD)

# ------------------------------------------
# ---- ^^^ DO NOT EDIT THIS SECTION ^^^ ----
# ------------------------------------------

# ---- Nextflow Configuration File Creation ----
# This section creates a Nextflow configuration file with various settings for the pipeline execution.

# Use cat to create or overwrite the mmc.config file with the desired Nextflow configurations.
# NOTE: S3 keys and OpCenter information will be concatted to the end of the config file. No need to add them now

# Additionally, please add your STAGE MOUNT BUCKETS here
cat > mmc.config << EOF
// enable nf-float plugin.
plugins {
    id 'nf-float'
}

// Process settings: Executor, error strategy, and resource allocation specifics.
process {
    executor = 'float'
    errorStrategy = 'retry'
    extra = '--dataVolume [opts=" --cache-dir /mnt/jfs_cache "]jfs://${jfs_private_ip}:6868/1:/mnt/jfs --dataVolume [size=120]:/mnt/jfs_cache --vmPolicy [spotOnly=true,retryLimit=10,retryInterval=300s]'
}

// Directories for Nextflow execution.
workDir = '${workDir}'
launchDir = '${workDir}'

// OpCenter connection settings.
float {
    address = '${opcenter_ip_address}'
    username = '${opcenter_username}'
    password = '${opcenter_password}'
}

// AWS S3 Client configuration.
aws {
  client {
    maxConnections = 20
    connectionTimeout = 300000
  }
  accessKey = '${access_key}'
  secretKey = '${secret_key}'
}

EOF

# ---- Data Preparation ----
# Use this section to copy essential files from S3 to the working directory.

# For example, copy the sample sheet and params.yml from S3 to the current working directory.
# aws s3 cp s3://nextflow-input/samplesheet.csv .
# aws s3 cp s3://nextflow-input/scripts/params.yml .

# ---- Nextflow Command Setup ----
# Important: The -c option appends the mmc config file and soft overrides the nextflow configuration.

# Assembles the Nextflow command with all necessary options and parameters.
nextflow_command='nextflow run <nextflow-pipeline> \
-r <revision-number> \
-c mmc.config \
-params-file params.yml \
--input samplesheet.csv \
--outdir 's3://nextflow-output/rnaseq/' \
-resume '

# ---------------------------------------------
# ---- vvv DO NOT EDIT BELOW THIS LINE vvv ----
# ---------------------------------------------
# The following section contains functions and commands that should not be modified by the user.

# Create side script to tag head node - exits when properly tagged
cat > tag_nextflow_head.sh << EOF
#!/bin/bash

runname="\$(cat .nextflow.log 2>/dev/null | grep nextflow-io-run-name | head -n 1 | grep -oP '(?<=nextflow-io-run-name:)[^ ]+')"
workflowname="\$(cat .nextflow.log 2>/dev/null | grep nextflow-io-project-name | head -n 1 | grep -oP '(?<=nextflow-io-project-name:)[^ ]+')"

while true; do

  # Runname and workflowname will be populated at the same time
  # If the variables are populated and not tagged it, tag the head node
  if [ ! -z \$runname ]; then
    ./float modify -j "$(echo $FLOAT_JOB_ID)" --addCustomTag run-name:\$runname 2>/dev/null
    ./float modify -j "$(echo $FLOAT_JOB_ID)" --addCustomTag workflow-name:\$workflowname 2>/dev/null
    break
  fi

  runname="\$(cat .nextflow.log 2>/dev/null | grep nextflow-io-run-name | head -n 1 | grep -oP '(?<=nextflow-io-run-name:)[^ ]+')"
  workflowname="\$(cat .nextflow.log 2>/dev/null | grep nextflow-io-project-name | head -n 1 | grep -oP '(?<=nextflow-io-project-name:)[^ ]+')"

  sleep 1s

done
EOF

# Start tagging side-script
chmod +x ./tag_nextflow_head.sh
./tag_nextflow_head.sh &

# Start Nextflow run
$nextflow_command

if [[ $? -ne 0 ]]; then
  echo $(date): "Nextflow command failed."
  exit 1
else 
  echo $(date): "Nextflow command succeeded."
  exit 0
fi

```

</details>

### Job-Submit Script Adjustments

* Modify the `process.extra` within the `mmc.config` section to customize the `vmPolicy` for individual nextflow processes (Default policy is `spotOnly` in the job submit script. Adjust as needed e.g., `onDemand`, `spotFirst`). You may find more options when calling `float submit -h`:

```bash
--vmPolicy [spotOnly=true,retryLimit=10,retryInterval=300s]
```

For handling nextflow code repository, samplesheets and params-file, you have two options: download them or create them directly in the script. Here’s how to do both:

### Downloading Samplesheet and Params File

* Provide download commands for users to obtain samplesheet and params file for Nextflow, ensuring you replace `<download-link>` with the actual URLs:

```bash
# Download samplesheet
aws s3 cp s3://nextflow-input/samplesheet.csv .

# Download params file
aws s3 cp s3://nextflow-input/params.yml .
```

### Creating Samplesheet and Params File Directly in the Script

* Alternatively, users can create these files directly within the script using the `cat` command as shown below.

```bash
# Create samplesheet
cat > samplesheet.csv << EOF
sample,fastq_1,fastq_2,strandedness
AL_TO_rep01,s3://nextflow-input/Sample_1_L007_R1_001.fastq.gz,s3://nextflow-input/Sample_1_L007_R2_001.fastq.gz,auto
AL_TO_rep01,s3://nextflow-input/Sample_2_L008_R1_001.fastq.gz,s3://nextflow-input/Sample_2_L008_R2_001.fastq.gz,auto
AL_TO_rep01,s3://nextflow-input/Sample_3_L014_R1_001.fastq.gz,s3://nextflow-input/Sample_3_L014_R2_001.fastq.gz,auto
AL_TO_rep01,s3://nextflow-input/Sample_4_L009_R1_001.fastq.gz,s3://nextflow-input/Sample_4_L009_R2_001.fastq.gz,auto
EOF

# Create params file
cat > params.yml << EOF
multiqc_title: "rnaseq_multiqc"
fasta: "s3://nextflow-input/reference/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz"
gtf: "s3://nextflow-input/reference/Caenorhabditis_elegans.WBcel235.111.gtf.gz"
save_reference: true
remove_ribo_rna: true
skip_alignment: true
pseudo_aligner: "salmon"
EOF
```

---

* Download your nextflow code repository using `git clone` or copying from S3

> NOTE: if using private git repositories you can export your GITHUB_TOKEN before the nextflow_command

* Finally, ensure you customize your `nextflow_command` with specific pipeline requirements and save the changes:

```bash
nextflow_command='nextflow run <nextflow-pipeline> \
-r <revision-number> \
-c mmc.config \
-params-file params.yml \
--input samplesheet.csv \
--outdir 's3://nextflow-output/rnaseq/' \
-resume '
```

> Remember to replace placeholders with your specific pipeline details. 

---

## Float Submit

* Login to your MMCloud opcenter:

```bash!
float login -a <opcenter-ip-address> -u <user>
```
> Enter your password in the next prompt and you should see `Login succeded`

* Ensure you are using the latest version of OpCenter & float CLI:

```bash
float release upgrade --sync
```

* Make sure you have the following variables set as-is for `float secret`'s:

```bash
+-----------------------+
|         NAME          |
+-----------------------+
| OPCENTER_IP_ADDRESS   |
| OPCENTER_USERNAME     |
| OPCENTER_PASSWORD     |
| AWS_BUCKET_ACCESS_KEY |
| AWS_BUCKET_SECRET_KEY |
+-----------------------+
```

* Useful `float secret` commands:

```bash
# to list stored secrets
float secret ls
# to set a secret
float secret set OPCENTER_IP_ADDRESS 192.0.1.2
# to unset a secret
float secret unset OPCENTER_IP_ADDRESS
```

### Float Submit Command

* Replace the placeholders `<work-bucket>`, `<region>`, `<security-group>`, and `<job-name>` with your specific values and execute the float submit command:

```bash!
float submit \
--hostInit transient_JFS_AWS.sh \
--hostTerminate hostTerminate_AWS.sh \
-i docker.io/memverge/juiceflow \
--vmPolicy '[onDemand=true]' \
--migratePolicy '[disable=true]' \
--dataVolume '[size=60]:/mnt/jfs_cache' \
--dirMap /mnt/jfs:/mnt/jfs \
-c 2 -m 4 \
-n <job-name> \
--securityGroup <security-group> \
--env BUCKET=https://<work-bucket>.s3.<region>.amazonaws.com \
-j job_submit_AWS.sh
```

Here's a brief explanation of the parameters used in the `float submit` command:

| Parameter | Brief Description |
|-----------|-------------------|
| `--hostInit transient_JFS_AWS.sh` | Shell script to run on the host before the job starts. |
| `--hostTerminate hostTerminate_AWS.sh` | Shell script to run on the host after the job has been cancelled |
| `-i docker.io/memverge/juiceflow` | Docker image for the job's software environment. |
| `--vmPolicy '[onDemand=true]'` | Uses on-demand VM instance for head-node execution. |
| `--migratePolicy '[disable=true]'` | Disables head-node migration to different hosts/VMs. |
| `--dataVolume '[size=60]:/mnt/jfs_cache'` | Attaches a 60GB data volume at `/mnt/jfs_cache` in the container. |
| `--dirMap /mnt/jfs:/mnt/jfs` | Maps a host directory to a container directory for data sharing. |
| `-c 2` | Allocates 2 CPU cores to the job. |
| `-m 4` | Allocates 4GB of memory to the job. |
| `-n <job-name>` | Assigns a name to the job for identification. |
| `--securityGroup <security-group>` | Applies a security group to the job's VM for network rules. |
| `--env BUCKET=https://<work-bucket>.s3.<region>.amazonaws.com` | Sets an environment variable for the S3 bucket URL. |
| `-j job_submit_AWS.sh` | Specifies the job script or command to run inside the container. |

---

### Mounting S3 buckets as Data Volumes

* you can mount the input data bucket using S3FS as a data volume on the Nextflow head node and worker nodes as follows:

```
--storage <storage-name>
```

* Read more on [Juiceflow Performance Optimization](https://www.mmcloud.io/resources/docs/juiceflow-performance-optimization-aws)


---

## Important Considerations

- **JuiceFS Bucket Requirements**: JuiceFS requires formatting at the root level of a storage bucket. It cannot be formatted on a sub-directory within a bucket. Ensure the root directory of the bucket, or the entire bucket, is specified in the command line interface (CLI) command.

---

### Cancelling a Running Workflow to Resume Later

There are many circumstances where you might need to cancel a running Nextflow workflow and resume it later using Nextflow's `-resume` feature, such as changes to the configuration or resource specifications.

* To cancel a workflow in the JuiceFlow setup, cancel the head node job. This will cause the head node host to generate a `<JOB_ID>.meta.json.gz`, where the JOB_ID represents the job-id of the head node file in the S3 work bucket. 

* You can find the `<JOB_ID>.meta.json.gz` and the status of metadata dump in the `job.events` log in `Attachments` of the head node job

* This `<JOB_ID>.meta.json.gz` file is critical for restoring the work directory for subsequent attempts using the environmental variable `PREVIOUS_JOB_ID`.

---

### Resuming Workflows with the Job-Submit Script

JuiceFlow supports the `-resume` option with Nextflow. Each workflow execution generates a `<JOB_ID>.meta.json.gz` file in the S3 work bucket. This file is critical for restoring the work directory for subsequent attempts using the environmental variable `PREVIOUS_JOB_ID` .

> Example command
```
float submit \
--hostInit transient_JFS_AWS.sh \
--hostTerminate hostTerminate_AWS.sh \
-i docker.io/memverge/juiceflow \
--vmPolicy '[onDemand=true]' \
--migratePolicy '[disable=true]' \
--dataVolume '[size=60]:/mnt/jfs_cache' \
--dirMap /mnt/jfs:/mnt/jfs \
-c 2 -m 4 \
-n <job-name> \
--securityGroup <security-group> \
--env PREVIOUS_JOB_ID=<PREVIOUS_JOB_ID> \
--env BUCKET=https://<work-bucket>.s3.<region>.amazonaws.com \
-j job_submit_AWS.sh
```
    
---


## Monitoring on OpCenter

### Workflow Execution Log

To monitor workflow execution and get a detailed view of each process in the Nextflow workflow:

* Click on the `Workflows` dashboard in the OpCenter to monitor workflow execution and get a detailed view for each process in the Nextflow workflow:

![](https://baywatch-api.memverge.com/media/upload/202406/240626214812100945.png)

* Click on the workflow name, and you can monitor the jobs running in this workflow in this consolidated view:

![](https://baywatch-api.memverge.com/media/upload/202406/240626214854210539.png)

* Once the head-node job starts `Executing`, you can monitor the Nextflow stdout by clicking on the job -> `Attachments` -> `stdout.autosave`:

![](https://baywatch-api.memverge.com/media/upload/202406/240626214616545791.png)

### Individual Job Logs

To view the execution log of any particular job:

* Click on the Job-ID.
* Navigate to the `Attachments` tab.
* Click `view` (eye-icon) on the `stdout.autosave` file.

![](https://baywatch-api.memverge.com/media/upload/202406/240626215440890739.png)

![](https://baywatch-api.memverge.com/media/upload/202406/240626215520824360.png)

---

## Create Job Templates to Launch via MMCLoud GUI

Job Templates allow for the ease and customaization of runs that follow a similar format, without having the need to manually set up a command every time. It requires the submission of one job first.

* From the `Jobs` dashboard, select the head node job previously submitted above and click on `More Actions` -> `Save as Template`:

![](https://baywatch-api.memverge.com/media/upload/202406/240626215758509556.png)

* Provide a name and tag for the private template:

![](https://baywatch-api.memverge.com/media/upload/202406/240626215842168590.png)

* Navigate to the `Job Templates` dashboard and click on `Private` templates:

![](https://baywatch-api.memverge.com/media/upload/202406/240626220028284894.png)

* You can click on any job template, edit/change samplesheet, variables etc., and submit new jobs from the GUI.

---

> * Users can also submit jobs from templates via CLI
>```bash!
>float submit --template private::<template-name>:<template-tag> \
>-e BUCKET=s3://<aws-jfs-bucket>
>```
> Addtionally, please keep in mind the features that need to be updated with every run if they deviate from the default values provided in the private template. This will mainly include:
> * S3 Bucket URL
> * Updating of the nextflow run command in the job script

---

# Persistent Nextflow JuiceFS

## Introduction to JuiceFS

[JuiceFS](https://juicefs.com/docs/community/introduction/) is an open-source, high-performance distributed file system designed specifically for cloud environments. It offers unique features, such as:

- **Separation of Data and Metadata**: JuiceFS stores files in chunks within object storage like Amazon S3, while metadata can be stored in various databases, including Redis.
- **Performance**: Achieves millisecond-level latency and nearly unlimited throughput, depending on the object storage scale.
- **Easy Integration with MMCloud**: MMCloud provides pre-configured nextflow head node templates with JuiceFS setup, simplifying deployment.
- **Comparison with S3FS**: For a detailed comparison between JuiceFS and S3FS, see [JuiceFS vs. S3FS](https://juicefs.com/docs/community/comparison/juicefs_vs_s3fs/). JuiceFS typically offers better performance and scalability.

---

## Pre-requisites for Using JuiceFS with Nextflow

Before you begin, ensure you meet the following pre-requisites:

- A VPC Security Group with inbound-rule for port `6868`.
  - **Navigation:** `AWS EC2 console -> Network & Security -> Security Groups`

![](https://baywatch-api.memverge.com/media/upload/202406/240626212749399541.png)

> **Inbound rules should include:**
    - Custom TCP over TCP on port `6868`, used by the Redis server in this juiceflow setup.

2. **Create a New S3 Bucket**:
   - Cloud storage is required for JuiceFS's backend storage.
   - Create a new S3 bucket and note its region, as this will be needed later.
   - Path: *AWS S3 console -> Create Bucket*
   - ![S3 Bucket Creation](https://baywatch-api.memverge.com/media/upload/202312/231208005848879261.png)

---

# Deployment Steps for Individual Users on MMCloud

## Float Login

Ensure you are using the latest version of `float` CLI:

```bash
sudo float release sync
```

Login to your MMCloud opcenter:

```bash
float login -a <opcenter-ip-address> -u <user>
```

> After entering your password, verify that you see `Login succeeded!`

## Float Secret

Set your AWS credentials as secrets in `float`:

```bash
float secret set AWS_BUCKET_ACCESS_KEY <BUCKET_ACCESS_KEY>
float secret set AWS_BUCKET_SECRET_KEY <BUCKET_SECRET_KEY>
```

To verify the secrets:

```bash
float secret ls
```

> **Expected output**:
>
> ```bash
> +-----------------------+
> |           NAME        |
> +-----------------------+
> | AWS_BUCKET_ACCESS_KEY |
> | AWS_BUCKET_SECRET_KEY |
> +-----------------------+
> ```

## Deploy Nextflow Head Node

Deploy the Nextflow head node using the `nextflow:jfs` template:

```bash
float submit -n <head-node-name> \
--template nextflow:jfs \
--securityGroup <sg-00000000000a> \
-e BUCKET=https://<bucket-name>.s3.<bucket-region>.amazonaws.com \
--env BUCKET_ACCESS_KEY={secret:AWS_BUCKET_ACCESS_KEY} \
--env BUCKET_SECRET_KEY={secret:AWS_BUCKET_SECRET_KEY}
```

> **Note**: Replace `<head-node-name>`, `<security-group>`, `<bucket-name>`, and `<bucket-region>` with your specific details. The `nextflow:jfs` template comes pre-configured with JFS setup.

---

### Overriding Template Defaults

#### Customizing CPU and Memory
To override default CPU and memory settings:

```bash
--overwriteTemplate "*" -c <number-of-cpus> -m <memory-in-gb>
```

> **Example**: To set 8 CPUs and 32GB memory
> ```bash
> --overwriteTemplate "*" -c 8 -m 32
> ```

#### Specifying a Subnet
For deploying in a specific AWS [subnet](https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html):

```bash
--overwriteTemplate "*" --subnet <SUBNET-ID>
```
---

### Mounting S3 buckets as Data Volumes

* you can mount the input data bucket using S3FS as a data volume on the Nextflow head node and worker nodes as follows:

```
--dataVolume [mode=r,accesskey=xxx,secret=xxx,endpoint=s3.REGION.amazonaws.com]s3://BUCKET_NAME:/staged-files
```

* Read more on [Juiceflow Performance Optimization](https://www.mmcloud.io/resources/docs/juiceflow-performance-optimization-aws)

---

#### Incremental Snapshot Feature (From v2.4)
Enables faster checkpointing and requires larger storage:

```bash
--overwriteTemplate "*" --dumpMode incremental
```

> **Advantages**:
> - Supports larger workloads
> - Lower impact on job running time
> - No need for periodic snapshot interval configuration
>
> **Disadvantages**:
> - Requires larger storage for delta saves
> - Final snapshot is necessary for restore

#### Checking Head Node Deployment Status

```bash
float list
```

> **Example Output**:
> ```bash
> +-----------------------+----------------+------------------------------------+---------+-----------+----------------------+------------+-------------+
> |          ID           |      NAME      |            WORKING HOST            |  USER   |  STATUS   |     SUBMIT TIME      |  DURATION  |    COST     |
> | NlygkM1dA0qIucncPwjgD | jfs-head-1     | 54.211.194.151 (2Core4GB/OnDemand) | sateesh | Executing | 2023-11-02T17:40:06Z | 6m54s      | 0.0049 USD  |
> +-----------------------+----------------+------------------------------------+---------+-----------+----------------------+------------+-------------+
> ```

#### SSH into Head Node

- Locate the public IP address of the head node in the `Working Host` column.
- Retrieve the SSH key from Float's secret manager:

```bash
float secret get <job-id>_SSHKEY > <head-node-name>-ssh.key
```

> **Note**: If you encounter a `Resource not found` error, wait a few more minutes for the head node and SSH key to initialize.

- Set the appropriate permissions for the SSH key:

```bash
chmod 600 <head-node-name>-ssh.key
```

---

## SSH to Nextflow JFS Head Node

SSH into the Nextflow head node using the provided SSH key, username, and the head node's public IP address:

```bash
ssh -i <head-node-name>-ssh.key <user>@<head-node-public-ip-address>
```

> Note: Use the username `nextflow` to login as admin

## MMC nf-float configuration

### Editing the configuration file

1. Copy the template and edit the configuration file:

```bash
cp mmcloud.config.template mmc-jfs.config
vi mmc-jfs.config
```

> **Note**: If you're new to using `vi`, check out this [Beginner's Guide to Vi](https://www.howtogeek.com/102468/a-beginners-guide-to-editing-text-files-with-vi/) for basic instructions.

2. The `mmc-jfs.config` file copied from the template will be pre-filled with the OpCenter IP address, and the PRIVATE IP address of the Nextflow head node. **You only need to provide your OpCenter username, password and AWS access and secret keys in the config**.


```bash
plugins {
  id 'nf-float'
}

workDir = '/mnt/jfs/nextflow'

process {
    executor = 'float'
    errorStrategy = 'retry'
    extra ='  --dataVolume [opts=" --cache-dir /mnt/jfs_cache "]jfs://172.31.41.331:6868/1:/mnt/jfs --dataVolume [size=120]:/mnt/jfs_cache'
    /*
    For some special tasks like Qualimap which generates very small IO request, using this -o writeback_cache can help with performance. Here's an example.
    withName: "QUALIMAP_RNASEQ" {
      extra ='  --dataVolume [opts=" --cache-dir /mnt/jfs_cache  -o writeback_cache"]jfs://172.31.41.331:6868/1:/mnt/jfs --dataVolume [size=120]:/mnt/jfs_cache'
    }
    */
}

podman.registry = 'quay.io'

float {
    address = '172.31.42.11:443'
    username = '<your_user_name>'
    password = '<your_password>'
}

// AWS access info if needed
aws {
  client {
    maxConnections = 20
    connectionTimeout = 300000
  }
  accessKey = '<bucket_access_key>'
  secretKey = '<bucket_secret_key>'
}
```

### Using Tmux

Start a tmux session named `nextflow`:

```bash
tmux new -s nextflow
```

> To attach to an existing tmux session:
>
> ```bash
> tmux attach -t nextflow
> ```

> **Tip**: If you're new to `tmux`, here's a handy [Tmux Cheat Sheet](https://tmuxcheatsheet.com/).

### Nextflow Version Check

Check the Nextflow version and update if necessary:

```bash
nextflow -v
```

> **Example Output**:
>
> ```bash
> nextflow version 23.10.0.5889
> ```

---

## Launch Nextflow

Launch a Nextflow or `nf-core/<pipeline>` by providing the MMC config file:

```bash
nextflow run nf-core/<pipeline> \
-profile test_full \
-c mmc-jfs.config \
--outdir s3://nextflow-work-dir/<pipeline>
```

---

## Head Node Management

- In this persistent head node setup, user is responsible to dispose off the head node. To cancel the head node job click on the cancel button for the job as shown below

![](https://baywatch-api.memverge.com/media/upload/202406/240626221259289772.png)

![](https://baywatch-api.memverge.com/media/upload/202406/240626221324725332.png)


---

## FAQ

**Q**: Why does `nf-core/rnaseq` fail at the QUALIMAP step?

**A**: The QUALIMAP process in `nf-core/rnaseq` often fails due to its high-frequency, small-size write requests, leading to timeouts. Enabling `-o writeback_cache` consolidates these requests and improves performance significantly. However, it turns sequential writes into random writes, affecting sequential write performance. Use this setting only in scenarios with intensive random writes.

Add the following in the `process {}` scope of your config:

```bash
withName: "QUALIMAP_RNASEQ" {
     extra ='  --dataVolume [opts=" --cache-dir /mnt/jfs_cache  -o writeback_cache"]jfs://<head-node-private-ip>:6868/1:/mnt/jfs --dataVolume [size=120]:/mnt/jfs_cache'
   }
```

---

## Additional Reading

### Data Volumes

For jobs that generate file system I/O, specifying data volumes is essential. The OpCenter supports a variety of data volume types. Learn more about configuring data volumes in MMCloud in the [MMCloud Data Volumes Guide](https://docs.memverge.com/MMCloud/latest/User%20Guide/Working%20with%20OpCenter/datavolumes/).

### Allow / Deny Instance Types

Configure which instance types are allowed or denied in your setup. 

![Allow/Deny Instance Types](https://baywatch-api.memverge.com/media/upload/202312/231208013813787510.png)

See detailed docs on [Controlling Virtual Machine Types](https://docs.memverge.com/MMCloud/latest/User%20Guide/Working%20with%20OpCenter/allowlistforvm/)

---

# Instance Setup

1. Launch EC2 Instance. From the EC2 console, click on Launch Instance
![](https://baywatch-api.memverge.com/media/upload/202408/240802211531904303.png)

2. For AMI, search for amzn2-ami-kernel-5.10-hvm-2.0.20230628.0-x86_64-gp2
![](https://baywatch-api.memverge.com/media/upload/202408/240802211541058090.png)

3. For Instance type, choose t2.medium and provide key-pair
![](https://baywatch-api.memverge.com/media/upload/202408/240802211549358643.png)

4. For Network Setting select existing default security-group
![](https://baywatch-api.memverge.com/media/upload/202408/240802211556479325.png)

5. For Configure Storage, 8Gb of root volume is sufficient. Proceed to click on Launch Instance
![](https://baywatch-api.memverge.com/media/upload/202408/240802211604440390.png)



# MMC CLI Setup
Assuming your OpCenter is setup, to install the CLI:
```
wget https://<op_center_ip_address>/float --no-check-certificate
sudo mv float /usr/local/bin/
sudo chmod +x /usr/local/bin/float
```

Connect to your opcenter by loggin in:
```
float login -a <op_center_ip_address> -u <username> -p <password>
```

# Cromwell Setup
### Install Java

```
curl -s "https://get.sdkman.io" | bash
source "/home/ec2-user/.sdkman/bin/sdkman-init.sh"
sdk install java 17.0.6-tem
java -version
```
### Install Cromwell
```
wget https://github.com/broadinstitute/cromwell/releases/download/84/cromwell-84.jar

# Check version
java -jar cromwell-84.jar --version
```

# Config file
Name your file `cromwell-float.conf`. Make sure you update the address to your OpCenter address
```
# This is an example of how you can use Cromwell to interact with float.

backend {
  default = float

  providers {
    float {
      actor-factory = "cromwell.backend.impl.sfs.config.ConfigBackendLifecycleActorFactory"
      config {
        runtime-attributes="""
                String f_cpu = "2"
                String f_memory = "4"
                String f_docker = ""
                String f_extra = ""
        """

        # If an 'exit-code-timeout-seconds' value is specified:
        # - check-alive will be run at this interval for every job
        # - if a job is found to be not alive, and no RC file appears after this interval
        # - Then it will be marked as Failed.
        # Warning: If set, Cromwell will run 'check-alive' for every job at this interval
        exit-code-timeout-seconds = 30

        submit = """
            mkdir -p ${cwd}/execution
            echo "set -e" > ${cwd}/execution/float-script.sh
            echo "cd ${cwd}/execution" >> ${cwd}/execution/float-script.sh
            tail -n +22 ${script} > ${cwd}/execution/no-header.sh
            head -n $(($(wc -l < ${cwd}/execution/no-header.sh) - 14)) ${cwd}/execution/no-header.sh >> ${cwd}/execution/float-script.sh

            float submit -i ${f_docker} -j ${cwd}/execution/float-script.sh --cpu ${f_cpu} --mem ${f_memory} ${f_extra} >  ${cwd}/execution/sbatch.out 2>&1
            cat ${cwd}/execution/sbatch.out | sed -n 's/id: $.*$/\1/p' > ${cwd}/execution/job_id.txt
            echo "receive float job id: "
            cat ${cwd}/execution/job_id.txt

            JOB_SCRIPT_DIR=float-jobs/$(cat ${cwd}/execution/job_id.txt)
            mkdir -p $JOB_SCRIPT_DIR
            cd $JOB_SCRIPT_DIR
            
# create the check alive script
cat <<EOF > float-check-alive.sh
SCRIPT_DIR=$(pwd)
cd ${cwd}/execution
float show -j \$1 --runningOnly > job-status.yaml
if [[ -s job-status.yaml ]]; then
    cat job-status.yaml
else
    float show -j \$1 | grep rc: | tr -cd '[:digit:]' > rc
    float log cat -j \$1 stdout.autosave > stdout
    float log cat -j \$1 stderr.autosave > stderr
fi
cd $SCRIPT_DIR
EOF

# create the kill script
cat <<EOF > float-kill.sh
SCRIPT_DIR=$(pwd)
cd ${cwd}/execution
float scancel -f -j \$1
cd $SCRIPT_DIR
EOF

            cat ${cwd}/execution/sbatch.out
        """

        kill = """
            source float-jobs/${job_id}/float-kill.sh ${job_id}
        """

        check-alive = """
            source float-jobs/${job_id}/float-check-alive.sh ${job_id}
        """
        
        job-id-regex = "id: (\\w+)\\n"
      }
    }
  }
}

```
# S3 Bucket Setup
Follow the directions in this section: [Setup Nextflow host on AWS - HackMD](https://hackmd.io/@speri/rkmPkmP52#3-Create-Nextflow-work-directory-S3-bucket)

Once you made your bucket and created your access keys, install s3fs:
```
sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh
./configure --prefix=/usr --with-openssl
make
sudo make install
```
Create a password file with your access key and secret key in the form of
```
access_key:secret_key
```
Change mode your file to 600
```
chmod 600 ./passwd-s3fs
```
Mount your bucket to your designated mountpoint. If you are mounting to a directory that requires root privileges to access, you will need to use `sudo` to mount
```
s3fs BUCKET /MOUNTPOINT -o rw,allow_other -o multipart_size=52 -o parallel_count=30 -o passwd_file=~/.passwd-s3fs
```
If you plan on using an s3 bucket with your workflow, please update the `f_extra` line in the config:
```
String f_extra = "--dataVolume [accesskey=XXX,secret=XXX,mode=rw]s3://BUCKET:/MOUNTPOINT"
```

# Hello World (Read from Bucket)
### Create `hello.wdl`
```
workflow helloWorld {
    String name
    call sayHello { input: name=name }
}

task sayHello {
    String name

    command {
        printf "[cromwell-say-hello] hello to ${name} on $(date)\n"
        sleep 30
    }
    output {
        String out = read_string(stdout())
    }
    runtime {
        f_docker: "cactus"
    }
}
```
### Create `hello.json` in your bucket
```
{
    "helloWorld.name": "Developer"
}
```
### Run command (edit for your corresponding mountpoint)
```
java -Dconfig.file=cromwell-float.conf -jar \
 cromwell-84.jar run hello.wdl \
 --inputs /MOUNTPOINT/hello.json
```
A successful workflow will end in something similar to the snippet below. You may ignore the text that appear afterwards. The most important part is the “Succeeded”
```
[INFO] [11/27/2023 18:15:23.336] [cromwell-system-akka.dispatchers.engine-dispatcher-30] [akka://cromwell-system/user/SingleWorkflowRunnerActor] SingleWorkflowRunnerActor workflow finished with status 'Succeeded'.
{
  "outputs": {
    "helloWorld.sayHello.out": "[cromwell-say-hello] hello to Developer on Mon Nov 27 18:12:55 UTC 2023"
  },
  "id": "a1f0606e-367f-43a5-9381-e4ebe09ffbcf"
}
[2023-11-27 18:15:24,91] [info] Workflow polling stopped
```

# Sequence Workflow (Write to Bucket)
### Create `seq.wdl` (edit for your corresponding mountpoint)
```
workflow myWorkflow {
    call sayHello
    call writeReadFile { input: s=sayHello.out }
}

task sayHello {
    command {
        printf "[cromwell-say-hello] hello from $(whoami) on $(date)"
        sleep 30
    }
    output {
        String out = read_string(stdout())
    }
    runtime {
        f_docker: "cactus"
        f_cpu: "2"
        f_memory: "4"
    }
}

task writeReadFile {
    String s

    command {
        printf "[cromwell-write-read-file] write input to a file: ${s}\n" > /MOUNTPOINT/my_file.txt
        cat /MOUNTPOINT/my_file.txt
    }
    output {
        String out = read_string(stdout())
    }
    runtime {
        f_docker: "cactus"
    }
}
```
### Command
```
java -Dconfig.file=cromwell-float.conf -jar cromwell-84.jar run seq.wdl
```
You should expect to see the `my_file.txt` created and populated in your bucket

**Deploying MMCloud OpCenter in AWS Non-default VPC**
By Hui Chen, Yuchen Liu, Jing Gong

When EC2 resources are provisioned for the first time in an AWS account, a Default VPC is created automatically.  In the default VPC, an Internet gateway and public subnets with corresponding routing tables are pre-configured; a public IP address are assigned to VMs by default; and access to the Internet is enabled automatically.  As such, deploying MMCloud OpCenter in AWS default VPC is straightforward if the security posture of the default VPC is acceptable. However, in enterprise environments, it is often advantageous and/or necessary to run MMCloud workloads in a dedicated non-default VPC.  Such set up allows granular security enforcement and ensures that MMCloud does not affect production environments. 

Deploying MMCloud OpCenter in a non-default VPC requires careful planning so as to ensure proper communication between the various MMCloud components. In a non-default VPC, customers may have both public IP subnets and private IP subnets.  Public IP assignment normally is disabled for VMs, and most likely access to the internet will be disabled in the private subnets. For this kind of environment some extra cloud services need to be configured for MMCloud to work properly.

This document presents the procedure on how to deploy MMCloud OpCenter in AWS using a dedicated non-default VPC with typical enterprise security restrictions:

Step 1: Create VPC for MMCloud

From VPC dashboard, click on VPCs, then select Create VPC. Configure VPC settings as follows:
1.1	Resources to create: select VPC and more.
1.2	Name tag auto-generation: keep Auto-generate default set to be enabled; input a customized name tag.
1.3	IPv4 CIDR block: set the CIDR block addresses for the VPC.
1.4	IPv6 CIDR block: keep the default selection for No IPv6 CIDR block.
1.5	Tenancy: keep Default.
1.6	Number of Availability Zones (AZs): select 2 so as for the high availability.
1.7	Number of public subnets: select 2.
1.8	Number of private subnets: select 2 or 4 according to your needs.
1.9	NAT gateways ($): make selection according to your needs, None by default.
1.10	VPC endpoints: select S3Gateway.
1.11	DNS options: keep the default settings for Enable DNS hostnames and Enable DNS resolution. 
1.12	Click on Create VPC to create a new nondefault VPC.
<br>
![](https://baywatch-api.memverge.com/media/upload/202402/240227005041509145.png)
<br>
Step 2: Deploy MMCloud OpCenter in a public subnet of the newly created VPC. 

Follow Quick Start Deploy MMCloud OpCenter on AWS to set up the MMCloud in a public subnet of the VPC created at Step 1.  

This step allows users to access OpCenter from the public Internet, and allows OpCenter to communicate with the MMCloud License Server over the Internet.  

Step 3: Create EC2 endpoints for private subnets.

This step allows the OpCenter as well as the MMCloud worker nodes to access AWS services.  

From VPC dashboard, click on Endpoints, then select Create endpoint. Configure Endpoint settings as follows:
3.1 Name tag: input a customized name tag.
3.2 Service category: click on AWS services; search AWS service name of EC2 service, select it for the endpoint service.
3.3 VPC: select the VPC created at Step 1 as the VPC  in which to create your endpoint.
3.4 Subnets: for all the availability zones in the VPC, select the private subnet IDs.
3.5 IP address type: keep the default IPv4 setting.
3.6 Security groups: select default group and two more security groups created for your MMCloud OpCenter and WorkerNodes.
3.7 Policy: Keep default Full access selection.
3.8 Tags: keep the default setting.
3.9 Click on Create endpoints to create a new EC2 instance endpoint. Wait until the status of the endpoint be fully ready.
3.10 Open OpCenter GUI. ((It may take several minutes for opcenter to be ready.)
<br>
![](https://baywatch-api.memverge.com/media/upload/202402/240227005055368761.png)

If a worker node needs to access Internet resources, you may need to create a NAT gateway. This is not necessary if a worker node only needs to access AWS services.

Below are the IAM roles and policies created when deploying Memory Machine Cloud through AWS Marketplace, via the CloudFormation script. 

There are two types of objects, OpCenter and WorkerNode, and Memory Machine Cloud will create different IAM roles for them.

### For OpCenter
- license-manager:*  (to query the license in license manager)
- pricing:* (to query the cloud resource price)
- IAM roles: (these are limited in the local AWS account)
iam:GetRole
iam:AttachRolePolicy
iam:CreateRole
iam:PutRolePolicy
iam:PassRole
iam:CreateServiceLinkedRole
- EC2 roles: (to allow Memory Machine Cloud to create and control an EC2 instance, for those delete related action, Memory Machine Cloud limited to the resource which created by Memory Machine Cloud)
ec2:Get*
ec2:Describe*
ec2:RunInstances
ec2:StartInstances
ec2:StopInstances
ec2:Create*
ec2:Modify*
ec2:AttachVolume
ec2:DetachVolume
ec2:DeleteVolume
ec2:DeleteSecurityGroup
ec2:TerminateInstances
ec2:DeleteSnapshot
- S3 roles: (to allow Memory Machine Cloud to access the customer data which is in the local AWS account’s s3 bucket or public buckets)
s3:DeleteObject
s3:Put*
s3:Replicate*
s3:Restore*
s3:CreateBucket
s3:DeleteBucket
s3:Update*
s3:List*
s3:Get*
s3:Describe*
- ECR roles: (to allow Memory Machine Cloud to access the customer’s ECR repo)
ecr:GetDownloadUrlForLayer
ecr:BatchGetImage
ecr:DescribeImages
ecr:ListImages
ecr:GetAuthorizationToken
ecr:BatchCheckLayerAvailability
- Marketplace roles: (to allow Memory Machine Cloud to push the Metering Usage)
aws-marketplace:MeterUsage

### For WorkerNode, the role is the subset of the OpCenter
- IAM roles:
iam:PassRole
- EC2 roles:
ec2:Get*
ec2:Describe*
ec2:StopInstances
ec2:Create*
ec2:Modify*
ec2:AttachVolume
ec2:DetachVolume
ec2:DeleteVolume
ec2:TerminateInstances
ec2:DeleteSnapshot
- S3 roles: (to allow Memory Machine Cloud to access the customer data which is in the local AWS account’s s3 bucket or public buckets)
s3:DeleteObject
s3:Put*
s3:Replicate*
s3:Restore*
s3:Update*
s3:List*
s3:Get*
s3:Describe*
- ECR roles: (to allow Memory Machine Cloud to access the customer’s ECR repo)
ecr:GetDownloadUrlForLayer
ecr:BatchGetImage
ecr:DescribeImages
ecr:ListImages
ecr:GetAuthorizationToken
ecr:BatchCheckLayerAvailability

# Deploy MMCloud OpCenter on Google Cloud Platform (GCP)

## Quick-Guide

1. Log in to the *Google Cloud Platform* at console.cloud.google.com

2. From the *Welcome* page, use the dropdown menu on the top navigation bar to select the project you are working on.
![image](https://hackmd.io/_uploads/SkKFgR7Ta.png)

3. From the *Navigation* Menu on the left side of the navigation bar, select **`Marketplace`**.
![image](https://hackmd.io/_uploads/rJ9MZAmaa.png)

4. In the search bar located in the middle of the page, type **`MemVerge`**. The search will return a single result named **`Memory Machine Cloud`**.
![image](https://hackmd.io/_uploads/BJ-wbAmpp.png)

    -  Click on **`Memory Machine Cloud`**
    -  Click **`Get Started`**
    -  If you agree to the terms and conditions, click **`Agree`**, then click **`Deploy`** in the pop-up window.

5. You would see a page displaying the steps required to install the OpCenter using a **Terraform** template.
![image](https://hackmd.io/_uploads/BkXU7RXpa.png)

    - Scroll all the way down and click **Download**. A zip file with a name in the format of **`xxx_terraform.zip`** will be downloaded to your local machine.
![image](https://hackmd.io/_uploads/r1rj7CXaa.png)

6.  At the top right-hand side of the navigation bar, click the *Activate Cloud Shell* icon. A Cloud Shell terminal session will open at the bottom part of your browser window. Resize the terminal as needed.

![image](https://hackmd.io/_uploads/HJMFDAmTp.png)

> You would also see message like below, please confirm again you are under the right project.

```!
Welcome to Cloud Shell! Type "help" to get started.
Your Cloud Platform project in this session is set to mmce-project.
Use “gcloud config set project [PROJECT_ID]” to change to a different project.
```

7. In the Cloud shell terminal, go to the right-hand side of the navigation bar, click the **More** icon (three dots), and then click Upload.
![image](https://hackmd.io/_uploads/r1ck_0QaT.png)

8. In the pop-up window, click Choose Files to locate and select the terraform module (zip file) just downloaded on your local machine, and then click UPLOAD. You can also determine where to save the files.
![image](https://hackmd.io/_uploads/BkGuuRX6p.png)

9. Navigate to the directory where you uploaded the file, and unzip the module.

```!
yiwei_zhang@cloudshell:~ (mmce-project)$ ls
96d6513f-2283-4218-8b94-8a5a2dc2faf7_terraform.zip

yiwei_zhang@cloudshell:~ (mmce-project)$ unzip *terraform.zip
Archive:  96d6513f-2283-4218-8b94-8a5a2dc2faf7_terraform.zip
  inflating: README.md               
  inflating: backend.tf              
   creating: examples/
   creating: examples/marketplace-test/
  inflating: examples/marketplace-test/main.tf  
  inflating: main.tf                 
   creating: modules/
   creating: modules/opcenter-vpc/
  inflating: modules/opcenter-vpc/outputs.tf  
  inflating: modules/opcenter-vpc/README.md  
  inflating: modules/opcenter-vpc/variables.tf  
  inflating: modules/opcenter-vpc/main.tf  
   creating: modules/opcenter-gce/
  inflating: modules/opcenter-gce/outputs.tf  
  inflating: modules/opcenter-gce/README.md  
  inflating: modules/opcenter-gce/variables.tf  
  inflating: modules/opcenter-gce/main.tf  
   creating: modules/apis/
  inflating: modules/apis/README.md  
  inflating: modules/apis/main.tf    
  inflating: outputs.tf              
  inflating: providers.tf            
  inflating: terraform.tfvars        
  inflating: variables.tf        
```

10. Edit the **`terraform.tfvars`** file (by typing `vi terraform.tfvars`, for example) to configure the project ID and a name for the OpCenter instance as follows.

```
mmc_project = "[PROJECT_ID]"
mmc_name = "[OP_CENTER_NAME]"
```
> For example
> ![image](https://hackmd.io/_uploads/S1ns5R7pT.png)

> NOTE: Please make sure that the project_ID is the current project you are in. 

11. **(Optional)** Change any parameter values from their default values by editing the relevant entries in *terraform.tfvars*, for example, `mmce_region`.

12. Initialize the terraform environment by executing the command `terraform init`.

```!
yiwei_zhang@cloudshell:~ (mmce-project)$ terraform init

Initializing the backend...
Initializing modules...

Initializing provider plugins...
- Reusing previous version of hashicorp/random from the dependency lock file
- Reusing previous version of hashicorp/google-beta from the dependency lock file
- Reusing previous version of hashicorp/google from the dependency lock file
- Using previously-installed hashicorp/random v3.6.0
- Using previously-installed hashicorp/google-beta v5.10.0
- Using previously-installed hashicorp/google v5.10.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
```

13. Apply the terraform module by entering the following command and enter **`yes`** when prompted.

```
terraform apply
```

14. Wait a few minutes, and you will see information about OpCenter upon completion.

```!
Apply complete! Resources: 22 added, 0 changed, 0 destroyed.

Outputs:

mmc_project = "mmce-project"
opcenter_db_disk_name = "test-opcenter-302e0e-db"
opcenter_instance_name = "test-opcenter-302e0e"
opcenter_ip_addr = "34.132.126.70"
opcenter_nfs_disk_name = "test-opcenter-302e0e-nfs"
opcenter_website = "https://34.132.126.70"
vpc_name = "test-default"
```

15. Click on the `opcenter_website` link or copy and paste the address into a new tab to open the MMC OpCenter, and here begins your journey with Memory Machine Cloud on GCP!

---

# MMCloud OpCenter

![](https://hackmd.io/_uploads/SyqXt2Uo2.png)

---

# Change Admin Password

We recommend changing the default username (**`admin`**) and password (**`memverge`**) for the opcenter after logging in for the first time.

* Click on **`Users & Groups`** from the left navigation menu and for the admin user click on **`Actions`** and change the password as necessary.

![](https://baywatch-api.memverge.com/media/upload/202402/240207013630883120.png)

![](https://baywatch-api.memverge.com/media/upload/202402/240207013901726047.png)

---

# Activate MMCloud License

* Log into your MMCloud OpCenter and activate license by clicking on the star icon in the top-right corner and enter your credentials

![](https://hackmd.io/_uploads/Skv7PSLyp.png)

---

# Download FLOAT CLI

* The float CLI binary helps connect a user from any local machine to the MMCloud opcenter. To download the CLI, click on the terminal icon towards the top right corner of the opcenter and choose the version of CLI tool based on the operating system

![](https://baywatch-api.memverge.com/media/upload/202402/240207015753981576.png)

* From macOS 10.15 (Catalina) onwards, zsh is the default shell. In zsh, float is a reserved word. To use the float CLI with macOS, either change the shell to bash or alias the word float to the float binary, as follows:

```bash=
alias float = /path/to/float_binary/float
```
> where `/path/to/float_binary/` is the path to where you placed the float binary.

* To change the default shell from `zsh` to `bash` see the following guide:
https://phoenixnap.com/kb/change-zsh-to-bash-mac

---

## Known Issues-1: AWS Rebalance Recommendation Signal

1. **Frequent Rebalance Recommendation Signals**:

**<u>Issue</u>**: AWS may send a [Rebalance Recommendation signal](https://docs.memverge.com/MMCloud/latest/User%20Guide/Exploring%20MMCloud%20Features/SpotSurfer/?h=rebala#aws-rebalance-recommendation-signal) if it determines that a Spot Instance has an increased likelihood of being reclaimed. This signal can be intercepted by the OpCenter, which uses a rules-based approach to decide whether to maintain the current Spot Instance or proactively capture a snapshot and transition to a new instance.
    
**<u>Impact</u>**: Frequent rebalance signals may lead to numerous interruptions, which can cause jobs to fail with checkpoint errors due to multiple spot reclaims.
    
**<u>Example</u>**: In the job example screenshot below, the OpCenter encountered numerous rebalance signals, resulting in 17 interruptions over a short period. These interruptions reset the buffer cache, causing processing delays and timeouts.

![](https://baywatch-api.memverge.com/media/upload/202408/240809162426315505.png)


2. **<u>Default Rebalance Threshold Settings</u>**:

**<u>Issue</u>**: The rebalance signal threshold on a new OpCenter is set to 64GB by default. This setting might trigger unnecessary rebalance actions that don't lead to actual spot reclaims.

**<u>Impact</u>**: This can cause jobs that require substantial buffer cache to fail or slow down significantly due to frequent cache resets and timeouts.
    
**<u>Resolution</u>**: Increasing the rebalance threshold to 125GB can help by ensuring that jobs are only checkpointed on actual spot reclaim signals rather than on every rebalance signal.
    
> **Note**: Changes to the rebalance threshold will only apply to newly launched jobs and not to those already in progress.

## Introduction

MMcloud provides a checkpoint solution that: 
* captures the complete application state, both memory and file system
* is transparent to the application, ie. no application change is needed
* causes minimal overhead, ie. second-level wall clock time per checkpoint
* is automated after integration with the customer orchestration environment.

MM checkpoint protects the application progress, which enables applications to be migrated across both compute nodes and execution time. The specific use cases include:
* Reduced operational costs, by replacing expensive On-demand VM instances with cheaper but less reliable Spot VM instances
* Faster time to result, by capturing the application states in execution and sharing progress in later pipeline stages or with the team members
* Higher resource utilization, by dynamically migrating applications to the right-sized VM instances or scheduling to run only in off-peak times

![](https://baywatch-api.memverge.com/media/upload/202407/240719133501300304.png)

---

Types of file systems supported for storing checkpoint data:
* AWS EFS
* AWS FsX Lustre
* Juicefs with AWS S3

---

## Steps

### Nextflow Snapshot Engine AMI

* We will provide an AMI customized for nextflow execution on AWS Batch with MMC Snapshot Engine built in.

* We can also provide the snapshot engine release bin file to customize the AMI even further based on customer requirements

---

### Set up metadata database for JuiceFS

* There are a number of ways to set up a database server for JuiceFS, specifically for a distributed Juicefs. Options include Amazon Elastic Cache, MemoryDB, and Amazon RDS.

* This quick-guide uses Amazon RDS (if you are not using this option, skip to the Create Launch Template after you have created a metadata server with the other options). 

* For this option, you must create a database within your RDS before using. This is a one-time thing. You can spin up a new instance and follow these steps:
  * Install JuiceFS
  * [Install mysql locally](https://dev.to/aws-builders/installing-mysql-on-amazon-linux-2023-1512)
  * Connect to your metadata server in this format
```
mysql -h tutorial-database-1-instance-1.cepmeznhgaxt.us-east-1.rds.amazonaws.com -P 3306 -u admin -pPASSWORD
```
> There is no space between the -p and your password (from RDS setup)

  * Create a database. This example names their database juicefs
```
create database juicefs;    
```
  * Then, exit the mysql
```
exit 
```

---

### Create EC2 Launch Template

* After your AMI is created, create a launch template. The core components are:
  * New AMI
  * SSH Key
  * User Data
* Format and Mount JuiceFS, which requires
  * Bucket
  * Access Keys
  * Checkpoint directory (same one used in packer setup)
  * Metadata server address, username, password, and database name

* Below is an example of the User Data. Make sure it is in proper MIME format.

```
Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0

--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"

#cloud-config
cloud_final_modules:
- [scripts-user, always]

--//
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"

#!/bin/bash
juicefs format --storage s3 --bucket https://<BUCKET_NAME>.s3.us-east-1.amazonaws.com 'mysql://<USERNAME>:<PASSWORD>@(<METADATA_ADDRESS>:3306)/<DATABASE_NAME>' myjfs-rds --access-key xxx --secret-key yyy
sudo mkdir -p /memverge-mm-checkpoint
sudo chmod 777 /memverge-mm-checkpoint
juicefs mount 'mysql://<USERNAME>:<PASSWORD>@(<METADATA_ADDRESS>:3306)/<DATABASE_NAME>' /memverge-mm-checkpoint &
--//--
```

---

### AWS Batch Compute Environment


* Amazon EC2 configuration
* Instance role - ecsInstanceRole (should already exist in your account)
* Service role - AWSServiceRoleForBatch (should already exist in your account)
* Default network setup

![](https://baywatch-api.memverge.com/media/upload/202407/240718181446372712.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718181724633400.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718181842065561.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718182133866276.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718182259238061.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718182405885118.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718182547415064.png)




### Job Queue

* Amazon EC2 configuration
* Connect to previously-made compute environment

![](https://baywatch-api.memverge.com/media/upload/202407/240718182713516508.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718182856447540.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718183018644779.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240718183108271338.png)

---

### Nextflow 

`aws.config`

```
plugins {
    id 'nf-amazon'
}
process {
    executor = 'awsbatch'
    queue = 'aws-spot-q'
    maxRetries = 2
}

process.containerOptions = '--env MMC_CHECKPOINT_INTERVAL=5m --env MMC_CHECKPOINT_IMAGE_SUBPATH=nextflow --env MMC_CHECKPOINT_MODE=true --env MMC_CHECKPOINT_IMAGE_PATH=/memverge-mm-checkpoint'

aws {
    accessKey = 'access
    secretKey = 'secret'
    region = 'us-east-1'
    client {
        maxConnections = 20
        connectionTimeout = 10000
        uploadStorageClass = 'INTELLIGENT_TIERING'
        storageEncryption = 'AES256'
    }
    batch {
        cliPath = '/nextflow_awscli/bin/aws'
        maxTransferAttempts = 3
        delayBetweenAttempts = '5 sec'
    }
}

```

## Environmental Variables

* `MMC_CHECKPOINT_INTERVAL = 5m` (default 15m)
* `MMC_CHECKPOINT_IMAGE_SUBPATH = checkpoint_dir`
> This is an optional variable that creates a subdirectory under your MMC_CHECKPOINT_IMAGE_PATH. Not having this variable will put all dumps into the MMC_CHECKPOINT_IMAGE_PATH.
* `MMC_CHECKPOINT_MODE = true`
* `MMC_CHECKPOINT_IMAGE_PATH = /memverge-mm-checkpoint`

---

`nextflow run command`

```
nextflow run sateeshperi/bwa-index -c aws.config -work-dir 's3://workdir-2'
```

---

### Run, Watch Logs, and Spot Interrupt

When the job is submitted, you are waiting for the Log stream name section to be populated, as well as your job to be in a Running state.


`Before spot Interrupt`
![](https://baywatch-api.memverge.com/media/upload/202407/240719133243470784.png)

`After spot Interrupt`
![](https://baywatch-api.memverge.com/media/upload/202407/240719133307081617.png)

## Introduction

* MMCloud's Juiceflow solution provides a high-performance method for running Nextflow pipelines on the cloud by leveraging JuiceFS to optimize cloud storage for the work directory.

> See the official [Juiceflow Quick Guide](https://www.mmcloud.io/resources/docs/juiceflow-aws) for introduction on how to run nextflow pipelines on MMCloud

* This quick guide is aimed towards easy submission of nextflow workflows to your mmcloud opcenter via the air interface. No command line interface is needed via this method.

---

## Admin Steps

* Login to MMCloud's Air interface available publicly at **[this link](https://air.mmcloud.io/)**

---

* Click on `Launch Air` towards the top navigation bar

![](https://baywatch-api.memverge.com/media/upload/202407/240714175026945534.png)

---

* Click on the user icon(top-right) to `Sign In` to your MMCloud account

![](https://baywatch-api.memverge.com/media/upload/202407/240714175133707352.png)

---

* Provide your MMCloud user credentials and click on `Sign In`

![](https://baywatch-api.memverge.com/media/upload/202407/240714175155857857.png)

---

* Once logged in, Click on `Execute through` -> `Private MMCloud`

![](https://baywatch-api.memverge.com/media/upload/202407/240714175214738348.png)

---

* Once confirmed that the execution mode is via private mmcloud, click on `Admin` tab from the top navigation bar

![](https://baywatch-api.memverge.com/media/upload/202407/240714175239118622.png)

---

* Under the `Admin` page, **first** add the `Security Group` required for juiceflow with port open on `6868`

![](https://baywatch-api.memverge.com/media/upload/202407/240714175306072987.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240714175318155089.png)

---

* Next, under `Storage`, add cloud storage bucket that will be used as the work directory for nextflow

![](https://baywatch-api.memverge.com/media/upload/202407/240714175332297944.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240714175357823013.png)

---

* Next, `Add Private MMCloud OpCenter`, by providing the IP address and user credentials.

* Add on any `Storage` buckets & `Security Group` previously added to connect to this opcenter

![](https://baywatch-api.memverge.com/media/upload/202407/240714175414020293.png)

![](https://baywatch-api.memverge.com/media/upload/202407/240714175437464958.png)

> These `Admin` steps are to be setup once and will be stored on air interface securely

---

## Launch Steps

* Click on `Launch` tab from the top navigation bar and further click on launch of the `Nextflow Generic Workflow` template.

![](https://baywatch-api.memverge.com/media/upload/202407/240714175451958648.png)

---

* Provide a `Job Name` and select the `MMCloud OpCenter` executor, `Storage` and `Security Group` for execution and click on `Next`

![](https://baywatch-api.memverge.com/media/upload/202407/240714175507104849.png)

---

* Under parameters, provide the repository for nextflow run execution including the revision/branch.
> **NOTE: Do NOT include --input, -params-file, -c, -C, nor --outdir.**

![](https://baywatch-api.memverge.com/media/upload/202407/240714183930917089.png)

---

* Next, provide the VM selection policy for worker nodes along with additional settings like `retryLimit` & `retryInterval`

> Available policies: `SpotFirst`, `SpotOnly`, `OnDemand`

![](https://baywatch-api.memverge.com/media/upload/202407/240714184046609493.png)

---

* **Pre-Run Command**: Use this field to run any bash commands that need to be executed before the main nextflow run command

* If using private git repositories, use this pre-run field to export your `GITHUB_TOKEN` (ignore, if repository is publicly accessible)

![](https://baywatch-api.memverge.com/media/upload/202407/240716133735256369.png)

---

* **Staged Mount**: Provide one or more cloud storage bucket urls in the form of `https://BUCKET.s3.REGION.amazonaws.com` to be mounted as data volumes.

> **Staging buckets with multiple input data sources such as raw-data, references as data volumes, will make the files available locally and will bypass any long staging times.**

> Note: This step is optional and can be skipped if you wish nextflow to stage the files to the work directory instead

![](https://baywatch-api.memverge.com/media/upload/202407/240719063618938495.png)

---

* Provide the output cloud storage bucket url for nextflow output

![](https://baywatch-api.memverge.com/media/upload/202407/240714185524886068.png)

---

* Paste, upload or provide an s3 URL for your samplesheet with the input files

> NOTE: If using the `Staged Mount`, update the location of your all input files to have `/staged-files-#` as your root directory, where `#` corresponds to your respective mount, starting at 1.
> > For example, if your FIRST input files (example: raw-data) are under `s3://input-bucket/input-folder/input.tgz`, your file path should now be `/staged-files-1/input-folder/input.tgz`.
> > If you have a SECOND mount (example: references), it would be under `/staged-files-2/input-folder/input.tgz`

`Pasting the samplesheet in plain text`
![](https://baywatch-api.memverge.com/media/upload/202407/240719062423568646.png)

`Providing an S3 URL for samplesheet`
![](https://baywatch-api.memverge.com/media/upload/202407/240719062710685473.png)



---

* Paste, upload or provide an s3 URL the nextflow params in a `yml` format 

`Pasting the params.yml in plain text`
![](https://baywatch-api.memverge.com/media/upload/202407/240719062830606601.png)

`Providing an S3 URL for params.yml`
![](https://baywatch-api.memverge.com/media/upload/202407/240719062945029191.png)


---

* Provide any additional nextflow config values that will be appended to existing nextflow config

> NOTE: providing this additional config is **OPTIONAL**

* Click on `Submit` when ready

![](https://baywatch-api.memverge.com/media/upload/202407/240714175939607327.png)

---

* Read the following disclaimer and click on `Submit`

![](https://baywatch-api.memverge.com/media/upload/202407/240714180001970113.png)

---

* Upon confirmation that the job has been submitted, Click on the `Runs` tab on the top navigation bar to monitor the workflow execution state

![](https://baywatch-api.memverge.com/media/upload/202407/240714180140933270.png)

---

* For more detailed workflow run & log monitoring, navigate and login to your MMCloud OpCenter

![](https://baywatch-api.memverge.com/media/upload/202407/240714180058174327.png)

---

* Under `Jobs` dashboard, monitor the status of head node execution

![](https://baywatch-api.memverge.com/media/upload/202407/240714180238716411.png)

---

* Under `Workflows` dashboard, monitor the status of workflow execution

![](https://baywatch-api.memverge.com/media/upload/202407/240714180351575201.png)

---

* MMCloud's Juiceflow solution provides a high-performance method for running Nextflow pipelines on the cloud by leveraging JuiceFS to optimize cloud storage for the work directory.

> Read more about Juiceflow in a blog [here](https://www.mmcloud.io/blog/juiceflow-a-next-generation-solution-for-nextflow)

* [JuiceFS](https://juicefs.com/docs/community/introduction/) is a general-purpose, distributed file system compatible with any application. In the current MMCloud release, JuiceFS can only be used with a Nextflow host deployed using the OpCenter's built-in Nextflow job template.

* While cloud storage formatted with JuiceFS offers high performance for work directories, input data often needs to be staged in the work directory. 

* Nextflow natively supports staging or even streaming inputs from S3 to the executors of the respective process steps. By defining the input S3 location as a Channel, the Nextflow framework ensures the data is available for the task.

* However, with a large number of files, staging can become a bottleneck as the files must first be downloaded to the stage subdirectory in the work directory and then made available to the worker nodes for execution.

* To avoid this staging bottleneck, input data in cloud storage can be mounted as data volumes, making the files available locally and bypassing the staging process.

* Essentially, you can register storage and mount the storage bucket using S3FS as a data volume on the Nextflow head node and worker nodes as `--storage <storage-name>`

---

## Steps:

> See the official [Juiceflow Quick Guide](https://www.mmcloud.io/resources/docs/juiceflow-aws) for introduction on how to run nextflow pipelines on MMCloud

### Register Storage

> Available from float `v3.0.0-69ce0c9-Imperia` onwards

* In the OpCetner left navigation bar, click on `Storage`

![](https://baywatch-api.memverge.com/media/upload/202408/240812050212519294.png)

* Click on `Register Storage` button

![](https://baywatch-api.memverge.com/media/upload/202408/240812050106954736.png)


* Select `Storage Type`

| Volume | col | col |
| - | - | - |
| NFS | content | content |
| Lustre | content | content |
| S3 | content | content |
| OSS | content | content |
| GS | content | content |

![](https://baywatch-api.memverge.com/media/upload/202408/240812050418968825.png)

* Provide a `name` for the storage, `S3 URI`, `endpoint`, `access key`, `secret key`, `mount-point` of the bucket and choose `Access-Mode` - `Read` only or `Read Write`

![](https://baywatch-api.memverge.com/media/upload/202408/240812051142317705.png)

### Using storage arguments in JuiceFlow

1. In your job script, add the input bucket as `--storage <storage-name>` in the `process.extra` section of `mmc.config`:

```
process {
    executor = 'float'
    errorStrategy = 'retry'
    extra = '--dataVolume [opts=" --cache-dir /mnt/jfs_cache "]jfs://${jfs_private_ip}:6868/1:/mnt/jfs --dataVolume [size=120]:/mnt/jfs_cache --storage <storage-name-1> --storage <storage-name-2>'
}
```

2. Modify your sample sheet to now read from the storage mount point defined while registering storage `</storage-mount-point>`:

```
group_id,subject_id,sample_id,sample_type,sequence_type,filetype,filepath
COLO829_Full,COLO829,COLO829T,tumor,dna,bam,/storage-mount-point/COLO829v003T.bam
COLO829_Full,COLO829,COLO829T,tumor,dna,bai,/storage-mount-point/COLO829v003T.bam.bai
COLO829_Full,COLO829,COLO829R,normal,dna,bam,/storage-mount-point/COLO829v003R.bam
COLO829_Full,COLO829,COLO829R,normal,dna,bai,/storage-mount-point/COLO829v003R.bam.bai
```

3. For the head node, add the `--storage <storage-name>` variable to the float submit command:

```
float submit \
--hostInit transient_JFS_AWS.sh \
--hostTerminate hostTerminate_AWS.sh \
-i docker.io/memverge/juiceflow \
--vmPolicy '[onDemand=true]' \
--migratePolicy '[disable=true]' \
--dataVolume '[size=60]:/mnt/jfs_cache' \
--storage <storage-name-1> \
--storage <storage-name-2> \
--dirMap /mnt/jfs:/mnt/jfs \
-c 2 -m 4 \
-n <job-name> \
--securityGroup <security-group> \
--env BUCKET=https://<work-bucket>.s3.<region>.amazonaws.com \
-j job_submit_AWS.sh
```

This setup ensures efficient data handling by reading input data from input buckets using S3FS as staged data volume, while the work buckets leverage JuiceFS for high performance.

---

## Using JuiceFS SnapLocation for better efficiency of checkpoint/restore

* MMC can use either local EBS volumes or JuiceFS formatted S3 buckets for storing snapshot data

* From the left navigation bar of the opcenter, click on `System Settings`

![](https://baywatch-api.memverge.com/media/upload/202408/240812052246117983.png)

* In System Settings, click on `Cloud` settings and go the `Snapshot Location` field


![](https://baywatch-api.memverge.com/media/upload/202408/240812052423317064.png)

* Provide the S3 bucket URL in the following format making sure to include the `accesskey`, `secretkey` and `mode=rw` 

```
[accesskey=<access>,secret=<secret>,mode=rw]s3://preview-opcenter-jfs-snaplocation
```

> **NOTE: THE S3 bucket provided for snaplocation as above will be used to store the snapshot data of all jobs from all users in the opcenter and the data is cleared out automatically after a job finished**

# Persistent Nextflow JuiceFS

## Introduction to JuiceFS

[JuiceFS](https://juicefs.com/docs/community/introduction/) is an open-source, high-performance distributed file system designed specifically for cloud environments. It offers unique features, such as:

- **Separation of Data and Metadata**: JuiceFS stores files in chunks within object storage like Amazon S3, while metadata can be stored in various databases, including Redis.
- **Performance**: Achieves millisecond-level latency and nearly unlimited throughput, depending on the object storage scale.
- **Easy Integration with MMCloud**: MMCloud provides pre-configured nextflow head node templates with JuiceFS setup, simplifying deployment.
- **Comparison with S3FS**: For a detailed comparison between JuiceFS and S3FS, see [JuiceFS vs. S3FS](https://juicefs.com/docs/community/comparison/juicefs_vs_s3fs/). JuiceFS typically offers better performance and scalability.

---

# Deployment Steps for Individual Users on MMCloud

## Float Login

Ensure you are using the latest version of `float` CLI:

```bash
sudo float release sync
```

Login to your MMCloud opcenter:

```bash
float login -a <opcenter-ip-address> -u <user>
```

> After entering your password, verify that you see `Login succeeded!`

## Deploy Nextflow Head Node

Deploy the Nextflow head node using the `nextflow:jfs` template:

```bash
float submit -n <head-node-name> \
--template nextflow:jfs \
-e BUCKET=gs://<bucket-name>
```

> **Note**: Replace `<bucket-name>` with your specific details. The `nextflow:jfs` template comes pre-configured with JFS setup.

---

### Overriding Template Defaults

#### Customizing CPU and Memory
To override default CPU and memory settings:

```bash
--overwriteTemplate "*" -c <number-of-cpus> -m <memory-in-gb>
```

> **Example**: To set 8 CPUs and 32GB memory
> ```bash
> --overwriteTemplate "*" -c 8 -m 32
> ```

#### Specifying a Subnet
For deploying in a specific AWS [subnet](https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html):

```bash
--overwriteTemplate "*" --subnet <SUBNET-ID>
```

#### Incremental Snapshot Feature (From v2.4)
Enables faster checkpointing and requires larger storage:

```bash
--overwriteTemplate "*" --dumpMode incremental
```

> **Advantages**:
> - Supports larger workloads
> - Lower impact on job running time
> - No need for periodic snapshot interval configuration
>
> **Disadvantages**:
> - Requires larger storage for delta saves
> - Final snapshot is necessary for restore

#### Checking Head Node Deployment Status

```bash
float list
```

> **Example Output**:
> ```bash
>+-----------------------+---------------------+-----------------------------------+---------+---------------+----------+----------------------+------------+
>|          ID           |        NAME         |           WORKING HOST            |  USER   |    STATUS     | DURATION |     SUBMIT TIME      |    COST    |
>+-----------------------+---------------------+-----------------------------------+---------+---------------+----------+----------------------+------------+
>| qcraofa3dydy6q6zjuarw | pers-gcp            | 35.224.90.211 (2Core4GB/OnDemand) | sateesh | Executing     | 2h23m38s | 2024-06-20T23:54:25Z | 0.1494 USD |
>+-----------------------+---------------------+-----------------------------------+---------+---------------+----------+----------------------+------------+
> ```

#### SSH into Head Node

- Locate the public IP address of the head node in the `Working Host` column.
- Retrieve the SSH key from Float's secret manager:

```bash
float secret get <job-id>_SSHKEY > <head-node-name>-ssh.key
```

> **Note**: If you encounter a `Resource not found` error, wait a few more minutes for the head node and SSH key to initialize.

- Set the appropriate permissions for the SSH key:

```bash
chmod 600 <head-node-name>-ssh.key
```

---

## SSH to Nextflow JFS Head Node

SSH into the Nextflow head node using the provided SSH key, username, and the head node's public IP address:

```bash
ssh -i <head-node-name>-ssh.key <user>@<head-node-public-ip-address>
```

> Note: Use the username `nextflow` to login as admin

## MMC nf-float configuration

### Editing the configuration file

1. Copy the template and edit the configuration file:

```bash
cp mmcloud.config.template mmc-jfs.config
vi mmc-jfs.config
```

> **Note**: If you're new to using `vi`, check out this [Beginner's Guide to Vi](https://www.howtogeek.com/102468/a-beginners-guide-to-editing-text-files-with-vi/) for basic instructions.

2. The `mmc-jfs.config` file copied from the template will be pre-filled with the OpCenter IP address, and the PRIVATE IP address of the Nextflow head node. You only need to provide your OpCenter username, password and AWS access and secret keys in the config.
   
   ![Head Node Private IP Address](https://baywatch-api.memverge.com/media/upload/202312/231208015119217441.png)

```bash
plugins {
  id 'nf-float'
}

workDir = '/mnt/jfs/nextflow'

process {
    executor = 'float'
    errorStrategy = 'retry'
    extra ='  --dataVolume [opts=" --cache-dir /mnt/jfs_cache "]jfs://10.128.0.44:6868/1:/mnt/jfs --dataVolume [size=120]:/mnt/jfs_cache'
    /*
    For some special tasks like Qualimap which generates very small IO request, using this -o writeback_cache can help with performance. Here's an example.
    withName: "QUALIMAP_RNASEQ" {
      extra ='  --dataVolume [opts=" --cache-dir /mnt/jfs_cache  -o writeback_cache"]jfs://10.128.0.44:6868/1:/mnt/jfs --dataVolume [size=120]:/mnt/jfs_cache'
    }
    */
}

podman.registry = 'quay.io'

float {
    address = '10.128.0.2:443'
    username = '<your_user_name>'
    password = '<your_password>'
}
```

### Using Tmux

Start a tmux session named `nextflow`:

```bash
tmux new -s nextflow
```

> To attach to an existing tmux session:
>
> ```bash
> tmux attach -t nextflow
> ```

> **Tip**: If you're new to `tmux`, here's a handy [Tmux Cheat Sheet](https://tmuxcheatsheet.com/).

### Nextflow Version Check

Check the Nextflow version and update if necessary:

```bash
nextflow -v
```

> **Example Output**:
>
> ```bash
> nextflow version 23.10.0.5889
> ```

---

## Launch Nextflow

Launch a Nextflow or `nf-core/<pipeline>` by providing the MMC config file:

```bash
nextflow run nf-core/<pipeline> \
-profile test_full \
-c mmc-jfs.config \
--outdir s3://nextflow-work-dir/<pipeline>
```

## Float Summary

Users can retrieve a summary report for a specific Nextflow run using the unique workflow name generated for each run

```bash
float costs -f tags=nextflow-io-run-name:stupefied-jones
```

> Example output
```bash
Job Number: 409
  Succeeded:    400 ( 97.80%)
  Failed:         9 (  2.20%)
  Running:        0 (  0.00%)
Cloud Resource:
  vCPU:            2922 Core(s)
  Core Hours:       708.97
  Memory:         16031 GB
  Storage:        51408 GB
  Extra Storage:   3850 GB
Host Count:       421
Floating Count:    13
Costs:       16.1840 USD
  Compute:       14.6383 USD
  Storage:       0.7978 USD
  Extra Storage: 0.7479 USD
Savings:     17.4906 USD
```

This summary provides an overview of resource utilization and costs associated with a particular Nextflow run, aiding in budget management and efficiency analysis for large-scale computational workflows.

---

## FAQ

**Q**: Why does `nf-core/rnaseq` fail at the QUALIMAP step?

**A**: The QUALIMAP process in `nf-core/rnaseq` often fails due to its high-frequency, small-size write requests, leading to timeouts. Enabling `-o writeback_cache` consolidates these requests and improves performance significantly. However, it turns sequential writes into random writes, affecting sequential write performance. Use this setting only in scenarios with intensive random writes.

Add the following in the `process {}` scope of your config:

```bash
withName: "QUALIMAP_RNASEQ" {
     extra ='  --dataVolume [opts=" --cache-dir /mnt/jfs_cache  -o writeback_cache"]jfs://<head-node-private-ip>:6868/1:/mnt/jfs --dataVolume [size=120]:/mnt/jfs_cache'
   }
```

---

## Additional Reading

### Data Volumes

For jobs that generate file system I/O, specifying data volumes is essential. The OpCenter supports a variety of data volume types. Learn more about configuring data volumes in MMCloud in the [MMCloud Data Volumes Guide](https://docs.memverge.com/mmce/current/userguide/olh/oxy_ex-1/topics/mmce_datavolumes.html).

### Allow / Deny Instance Types

Configure which instance types are allowed or denied in your setup.

![Allow/Deny Instance Types](https://baywatch-api.memverge.com/media/upload/202312/231208013813787510.png)

---

## Introduction
This how-to guide walks you through how to set up your AWS S3 bucket and then use the S3 bucket to provide input data files to MMCloud Air to run workload and MMCloud will save output result files to your S3 bucket. 

## Pre-requisites
1. An AWS account
2. An AWS user, either
  a. root user
  b. AWS IAM user that has the privilege to create AWS S3 bucket

## Overview of the Setup
1. Create an AWS S3 bucket for your input data files and output result files
2. Create an AWS IAM Policy which permits access your AWS S3 bucket
3. Create an AWS IAM User and attach the AWS IAM Policy to the AWS IAM User
4. Create access key and secret key for the AWS IAM User


By the end of the setup, you will be able to use the following information
- S3 Bucket URI
- S3 Region
- S3 Access Key
- S3 Secret Key

for input data files and output result files for MMCloud Air. 

Please see next section for more detailed steps. 

## Steps 
### Step 1 - Create an AWS S3 bucket for your input data files and output result files
Step 1.1 - Visit your AWS Console. Search for `s3`. Click on `Services - S3`. 
![](https://baywatch-api.memverge.com/media/upload/202405/240503152019489549.png)

Step 1.2 - Pick the region for your AWS S3 bucket from the right-upper corner. Below is an example of region as `us-west-1` US West (N. California) , and once selected, you can noticed the URL from your browser is [https://us-west-1.console.aws.amazon.com/s3/home?region=us-west-1](https://us-west-1.console.aws.amazon.com/s3/home?region=us-west-1), which indicates region as `us-west-1`. 
![](https://baywatch-api.memverge.com/media/upload/202405/240503152028635057.png)

Step 1.3 - Create an AWS S3 bucket. Use default settings pre-populated by AWS. Below is an example of bucket name as `s3-for-air`, created at region `us-west-1` US West (N. California). 
![](https://baywatch-api.memverge.com/media/upload/202405/240503152202680227.png)

Step 1.4 - Once created, you will be able to see a success message from your AWS Console and you can go into your AWS S3 bucket to create folder and upload input data files. Up until now, you have the following information ready - 

- S3 Bucket URI
  - `s3://[your-bucket-name]`
  - In the example above, it would be `s3://s3-for-air`
- S3 Region
  - region identifier
  - In the example above, it would be `us-west-1`
![](https://baywatch-api.memverge.com/media/upload/202405/240503152220897306.png)

### Step 2 - Create an AWS IAM Policy which permits access your AWS S3 bucket
Step 2.1 - Visit your AWS Console. Search for `iam`. Click on `Services - IAM`. 

Open image-20240425-173214.png
image-20240425-173214.png
Step 2.2 - Click on `Policies` on the left hand side, and click on `Create policy`. 
![](https://baywatch-api.memverge.com/media/upload/202405/240503152355961118.png)

Step 2.3 - Click on `JSON` on the right hand side to bring up the Policy editor in JSON format. Replace with the following policy which can read the input data files from your S3 bucket for MMCloud Air and write output result files to your S3 bucket from MMCloud Air. Which in this example, is `s3-for-air`. 
```language
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": [
				"s3:*"
			],
			"Resource": [
				"arn:aws:s3:::s3-for-air",
				"arn:aws:s3:::s3-for-air/*"
			]
		}
	]
}
```
![](https://baywatch-api.memverge.com/media/upload/202405/240503152434002296.png)

Step 2.4 - Give a policy name and click `Create policy`. In this example the policy name is `policy-for-air`. 
![](https://baywatch-api.memverge.com/media/upload/202405/240503152456111828.png)

Step 2.5 - Once created, you will be able to see a success message from your AWS Console and you can view your AWS IAM Policy. 
![](https://baywatch-api.memverge.com/media/upload/202405/240503152502141143.png)

### Step 3 - Create an AWS IAM User and attach the AWS IAM Policy to the AWS IAM User
Step 3.1 - Click on `Users` on the left hand side, and click on `Create user`. 
![](https://baywatch-api.memverge.com/media/upload/202405/240503152516144958.png)

Step 3.2 - Provide an AWS IAM User name. In this example, we use `user-for-air`. 
![](https://baywatch-api.memverge.com/media/upload/202405/240503152527355528.png)

Step 3.3 - Select `Attach policies directly` to attach the AWS IAM Policy to the AWS IAM user and search for the policy name just created. In this example, it would be AWS IAM Policy `policy-for-air` attached to AWS IAM user `user-for-air`. 
![](https://baywatch-api.memverge.com/media/upload/202405/240503152547772860.png)

Step 3.4 - Once created, you will be able to see a success message from your AWS Console and you can view your AWS IAM User and the AWS IAM Policy attached to this AWS IAM User. 
![](https://baywatch-api.memverge.com/media/upload/202405/240503152554194084.png)

### Step 4 - Create access key and secret key for the AWS IAM User
Step 4.1 - Go to the AWS IAM User just created. In this example, it would be `user-for-air`. Click on `Create access key`. 
![](https://baywatch-api.memverge.com/media/upload/202405/240503152609088290.png)

Step 4.2 - Select `Application running outside AWS` and create access key. 
![](https://baywatch-api.memverge.com/media/upload/202405/240503152619042763.png)

Step 4.3 - Record your access key (a string with 20 characters) and secret key (a string with 40 characters).  
![](https://baywatch-api.memverge.com/media/upload/202405/240503152625694701.png)

Step 4.4 - Up until now, you have additional following information ready - 

- S3 Access Key
  - a string with 20 characters
- S3 Secret Key
  - a string with 40 characters

## What’s Next
You will be able to use the information to set up your storage for MMCloud Air. Below is a screenshot of MMCloud Air adding an S3 bucket using the example we included in this how-to document. 
![](https://baywatch-api.memverge.com/media/upload/202405/240503152648023819.png)

Please visit [https://air.mmcloud.io](https://air.mmcloud.io) to start your first MMCloud Air workflow! 

For support, please send email to support-air@memverge.com or join slack group [to be updated].

# Juiceflow - GCP

## Introduction

Juiceflow combines JuiceFS and Nextflow on MMCloud, offering a powerful, scalable solution for managing and executing workflows in the cloud.

<details>
<summary>Expand for more details on JuiceFS</summary>

[JuiceFS](https://juicefs.com/docs/community/introduction/) is an open-source, high-performance distributed file system designed specifically for cloud environments. It offers unique features, such as:

* Separation of Data and Metadata: JuiceFS stores files in chunks within object storage like Google Cloud Storage, while metadata can be stored in various databases, including Redis.
* Performance: Achieves millisecond-level latency and nearly unlimited throughput, depending on the object storage scale.
* Easy Integration with MMCloud: MMCloud provides pre-configured nextflow head node templates with JuiceFS setup, simplifying deployment.
* Comparison with S3FS: For a detailed comparison between JuiceFS and S3FS, see [JuiceFS vs. S3FS](https://juicefs.com/docs/community/comparison/juicefs_vs_s3fs/). JuiceFS typically offers better performance and scalability.

</details>

---

## Pre-requisites

- GCP `gs://` buckets with proper permissions to your service account.
- GCP Credential key for a service account. To create one navigate to [Google Cloud Console](https://console.cloud.google.com)

- Navigate to `IAM` -> `Service Accounts`, click your corresponding account -> `KEYS` tab.

![](https://baywatch-api.memverge.com/media/upload/202405/240502224839738097.png)

- Create a JSON key and the key will be downloaded to your local machine upon creation. This key will be set as float secret and used later.

![](https://baywatch-api.memverge.com/media/upload/202405/240502224913652782.png)

---

## Overview of the Setup

This solution leverages two scripts:
- `transient_JFS_GCP.sh`: Formats the work directory bucket to JuiceFS format.
- `job_submit_GCP.sh`: Contains Nextflow input parameters and MMC config.

---

## Steps

### Download Scripts

* Host-init script (you don't have to edit this script, but you'll need it later)

```bash!
wget https://mmce-data.s3.amazonaws.com/juiceflow/v1/gcp/transient_JFS_GCP.sh
```

* Job-submit script (Download the template or create one locally based on the template below with your Nextflow inputs and run configurations)

```bash!
wget https://mmce-data.s3.amazonaws.com/juiceflow/v1/gcp/job_submit_GCP.sh
```

<details>
<summary>Expand to view a sample job-submit.sh script</summary>

```bash!
#!/bin/bash

# ---- User Configuration Section ----
# These configurations must be set by the user before running the script.

# ---- Optional Configuration Section ----
# These configurations are optional and can be customized as needed.

# JFS (JuiceFS) Private IP: Retrieved from the WORKER_ADDR environment variable.
jfs_private_ip=$(echo $WORKER_ADDR)

# Work Directory: Defines the root directory for working files. Optional suffix can be added.
workDir_suffix=''
workDir='/mnt/jfs/'$workDir_suffix
mkdir -p $workDir  # Ensures the working directory exists.
cd $workDir  # Changes to the working directory.
export NXF_HOME=$workDir  # Sets the NXF_HOME environment variable to the working directory.

# ---- Nextflow Configuration File Creation ----
# This section creates a Nextflow configuration file with various settings for the pipeline execution.

# Use cat to create or overwrite the mmc.config file with the desired Nextflow configurations.
# NOTE: GCS keys and OpCenter information will be concatted to the end of the config file. No need to add them now
cat > mmc.config << EOF
// enable nf-float plugin.
plugins {
    id 'nf-float'
}

// Process settings: Executor, error strategy, and resource allocation specifics.
process {
    executor = 'float'
    errorStrategy = 'retry'

    extra = '--dataVolume [opts=" --cache-dir /mnt/jfs_cache "]jfs://${jfs_private_ip}:6868/1:/mnt/jfs --dataVolume [size=120]:/mnt/jfs_cache --vmPolicy [spotOnly=true,retryLimit=10,retryInterval=300s]'
}

// Directories for Nextflow execution.
workDir = '${workDir}'
launchDir = '${workDir}'

EOF

# ---- Data Preparation ----
# Use this section to copy essential files from GCS to the working directory.

# For example, copy the sample sheet and params.yml from GCS to the current working directory.
# gcloud storage gs://nextflow-input/samplesheet.csv .
# gcloud storage gs://nextflow-input/scripts/params.yml .

# ---- Nextflow Command Setup ----
# Important: The -c option appends the mmc config file and soft overrides the nextflow configuration.

# Assembles the Nextflow command with all necessary options and parameters.
# The gs path needs to contain at least a sub-directory
# Bucket name MUST NOT contain underscores due to an internal GCS limitation
nextflow_command='nextflow run <nextflow-pipeline> \
-r <revision-number> \
-c mmc.config \
-params-file params.yml \
--input samplesheet.csv \
--outdir 'gs://nextflow-output/rnaseq/' \
-resume '

# -------------------------------------
# ---- DO NOT EDIT BELOW THIS LINE ----
# -------------------------------------
# The following section contains functions and commands that should not be modified by the user.

function install_float {
  # Install float
  local address=$(echo "$FLOAT_ADDR" | cut -d':' -f1)
  wget https://$address/float --no-check-certificate --quiet
  chmod +x float
}

function get_secret {
  input_string=$1
  local address=$(echo "$FLOAT_ADDR" | cut -d':' -f1)
  secret_value=$(./float secret get $input_string -a $address)
  if [[ $? -eq 0 ]]; then
    # Have this secret, will use the secret value
    echo $secret_value
    return
  else
    # Don't have this secret, will still use the input string
    echo $1
  fi
}

function gcloud_init() {
  # Login to gcloud with credential key
  gcloud auth login --quiet --cred-file=/mnt/jfs/gcp_cred.json
  if [[ $? -ne 0 ]]; then
    echo $(date): "gcloud authentication failed. Cannot dump JuiceFS metadata."
    exit 1
  fi
}

function remove_old_metadata () {
  echo $(date): "First finding and removing old metadata..."
  FOUND_METADATA=$(gcloud storage ls $BUCKET | grep ".meta.json.gz" |awk '{print $1}' | sed 's/gs:\/\///' | cut -d'/' -f2-)

  if [[ -z "$FOUND_METADATA" ]]; then
    echo $(date): "No previous metadata dump found. Continuing with dumping current JuiceFs"
  else
    echo $(date): "Previous metadata dump found! Removing $FOUND_METADATA"
    gcloud storage rm $BUCKET/$FOUND_METADATA

    echo $(date): "Previous metadata $FOUND_METADATA removed"
  fi

}

function dump_and_cp_metadata() {
  echo $(date): "Attempting to dump JuiceFS data"

  if [[ -z "$FOUND_METADATA" ]]; then
    # If no previous metadata was found, use the current job id
    juicefs dump redis://$(echo $WORKER_ADDR):6868/1 $(echo $FLOAT_JOB_ID).meta.json.gz --keep-secret-key
    echo $(date): "JuiceFS metadata $FLOAT_JOB_ID.meta.json.gz created. Copying to JuiceFS Bucket"
    gcloud storage cp "$(echo $FLOAT_JOB_ID).meta.json.gz" $BUCKET
  else
    # If previous metadata was found, use the id of the previous metadata
    metadata_name=$(echo $FOUND_METADATA | awk -F '.meta.json.gz' '{print $1}')
    juicefs dump redis://$(echo $WORKER_ADDR):6868/1 $(echo $metadata_name).meta.json.gz --keep-secret-key
    echo $(date): "JuiceFS metadata $metadata_name).meta.json.gz created. Copying to JuiceFS Bucket"
    gcloud storage cp "$(echo $metadata_name).meta.json.gz" $BUCKET
  fi

  echo $(date): "Copying to JuiceFS Bucket complete! Exiting..."
}

# Variables
FOUND_METADATA=""
export GOOGLE_APPLICATION_CREDENTIALS="/mnt/jfs/gcp_cred.json"

# Set Opcenter credentials
install_float
opcenter_ip_address=$(get_secret OPCENTER_IP_ADDRESS)
opcenter_username=$(get_secret OPCENTER_USERNAME)
opcenter_password=$(get_secret OPCENTER_PASSWORD)

# Append to config file
cat <<EOT >> mmc.config

// OpCenter connection settings.
float {
    address = '${opcenter_ip_address}'
    username = '${opcenter_username}'
    password = '${opcenter_password}'
}
EOT

# Initialize gcloud login
gcloud_init

# Start Nextflow run
$nextflow_command

# Error handling - JuiceFS dump and copy to gs bucket
if [[ $? -ne 0 ]]; then
  echo $(date): "Nextflow command failed."
  remove_old_metadata
  dump_and_cp_metadata
  exit 1
else 
  echo $(date): "Nextflow command succeeded."
  remove_old_metadata
  dump_and_cp_metadata
  exit 0
fi
```

</details>

### Job-Submit Script Adjustments

* Modify the `process.extra` within the `mmc.config` section to customize the `vmPolicy` for individual nextflow processes (Default policy is `spotOnly` in the job submit script. Adjust as needed e.g., `onDemand`, `spotFirst`). You may find more options when calling `float submit -h`:

```bash
--vmPolicy [spotOnly=true,retryLimit=10,retryInterval=300s]
```

For handling nextflow code repository, samplesheets and params-file, you have two options: download them or create them directly in the script. Here’s how to do both:

### Downloading Samplesheet and Params File

* Provide download commands for users to obtain samplesheet and params file for Nextflow, ensuring you replace `<download-link>` with the actual URLs:

```bash
# Download samplesheet
gcloud storage cp gs://nextflow-input/samplesheet.csv .

# Download params file
gcloud storage cp gs://nextflow-input/params.yml .
```

### Creating Samplesheet and Params File Directly in the Script

* Alternatively, users can create these files directly within the script using the `cat` command as shown below.

<details>
<summary>Expand to include instructions on creating samplesheet and params files directly in the script</summary>

```bash
# Create samplesheet
cat > samplesheet.csv << EOF
sample,fastq_1,fastq_2
SAMPLE1_PE,gs://mmc-gcp-poc/viralrecon-test-data/sample1_R1.fastq.gz,gs://mmc-gcp-poc/viralrecon-test-data/sample1_R2.fastq.gz
SAMPLE2_PE,gs://mmc-gcp-poc/viralrecon-test-data/sample2_R1.fastq.gz,gs://mmc-gcp-poc/viralrecon-test-data/sample2_R2.fastq.gz
SAMPLE3_SE,gs://mmc-gcp-poc/viralrecon-test-data/sample1_R1.fastq.gz,
SAMPLE3_SE,gs://mmc-gcp-poc/viralrecon-test-data/sample2_R1.fastq.gz,
EOF

# Create params file
cat > params.yml << EOF
multiqc_title: "rnaseq_multiqc"
fasta: "gs://nextflow-input/reference/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz"
gtf: "gs://nextflow-input/reference/Caenorhabditis_elegans.WBcel235.111.gtf.gz"
save_reference: true
remove_ribo_rna: true
skip_alignment: true
pseudo_aligner: "salmon"
EOF
```

</details>

---

* Download your nextflow code repository using `git clone` or copying from GCS

* Finally, ensure you customize your `nextflow_command` with specific pipeline requirements and save the changes:

```bash
nextflow_command='nextflow run your_pipeline.nf \
-r your_revision \
-c mmc.config \
-params-file params.yml \
--input samplesheet.csv \
--outdir your_output_directory \
-resume'
```

> Remember to replace placeholders with your specific pipeline details. 
> Additionally, the gs path needs to contain at least a sub-directory and the bucket name MUST NOT contain underscores due to an internal GCS limitation.

---

## Float Submit

* Login to your MMCloud opcenter:

```bash!
float login -a <opcenter-ip-address> -u <user>
```

* Ensure you are using the latest version of OpCenter & float CLI:

```bash
float release upgrade --sync
```

* Make sure you have the following variables set as-is for `float secret`'s:

```bash
+---------------------+
|        NAME         |
+---------------------+
| OPCENTER_USERNAME   |
| OPCENTER_IP_ADDRESS |
| OPCENTER_PASSWORD   |
| GCP_BUCKET_JSON     |
+---------------------+
```

* You can set the GCP Credential key JSON as secret as below:

```bash
float secret set GCP_BUCKET_JSON "$(< gcp_key.json)"
```

* Useful `float secret` commands:

```bash
# to list stored secrets
float secret ls
# to set a secret
float secret set OPCENTER_IP_ADDRESS 192.0.1.2
# to unset a secret
float secret unset OPCENTER_IP_ADDRESS
```

### Float Submit Command

* Replace the placeholders `<work-bucket>`,  `<job-name>` with your specific values and execute the float submit command:

```
float submit --hostInit transient_JFS_GCP.sh \
-i docker.io/memverge/juiceflow \
--vmPolicy '[onDemand=true]' \
--migratePolicy '[disable=true]' \
--dataVolume '[size=60]:/mnt/jfs_cache' \
--dirMap /mnt/jfs:/mnt/jfs \
-c 2 -m 4 \
-n <job-name> \
--env BUCKET=gs://<work-bucket> \
-j job_submit_GCP.sh
```

**Please Remember to provide the name of your GCP credential key in-between the parentheses.**

Here's a brief explanation of the parameters used in the `float submit` command:

| Parameter | Brief Description |
|-----------|-------------------|
| `--hostInit transient_JFS_GCP.sh` | Shell script to run on the host before the job starts. |
| `-i docker.io/memverge/juiceflow` | Docker image for the job's software environment. |
| `--vmPolicy '[onDemand=true]'` | Uses on-demand VM instance for head-node execution. |
| `--migratePolicy '[disable=true]'` | Disables head-node migration to different hosts/VMs. |
| `--dataVolume '[size=60]:/mnt/jfs_cache'` | Attaches a 60GB data volume at `/mnt/jfs_cache` in the container. |
| `--dirMap /mnt/jfs:/mnt/jfs` | Maps a host directory to a container directory for data sharing. |
| `-c 2` | Allocates 2 CPU cores to the job. |
| `-m 4` | Allocates 4GB of memory to the job. |
| `-n <job-name>` | Assigns a name to the job for identification. |
| `--env BUCKET=gs://<work-bucket>` | Sets an environment variable for the GCS bucket URL. |
| `-j job_submit_GCP.sh` | Specifies the job script or command to run inside the container. |


---

## Important Considerations

- **JuiceFS Bucket Requirements**: JuiceFS requires formatting at the root level of a storage bucket. It cannot be formatted on a sub-directory within a bucket. Ensure the root directory of the bucket, or the entire bucket, is specified in the command line interface (CLI) command.

- **Nextflow `-resume` Functionality**: JuiceFlow supports the `-resume` option with Nextflow. Each workflow execution generates a `<JOB_ID>.meta.json.gz` file in the work bucket. This file is critical for restoring the work directory for subsequent attempts.

---

## Monitoring on OpCenter

### Workflow Execution Log

Once the head node job starts executing, you can monitor the Nextflow standard output in OpCenter GUI as follows:

* Proceed to the OpCenter `Jobs` dashboard to monitor the progress of the job

![image](https://hackmd.io/_uploads/Hk1fv3MxA.png)
    
* Once the job starts `Executing`, you can monitor the Nextflow stdout by clicking on the job -> `Attachments` -> `stdout.autosave`:

![image](https://hackmd.io/_uploads/BygRFG71C.png)

---

To monitor workflow execution and get a detailed view of each process in the Nextflow workflow:

* Click on the `Workflows` dashboard in the OpCenter to monitor workflow execution and get a detailed view for each process in the Nextflow workflow:

![image](https://hackmd.io/_uploads/B1wg2zQJC.png)

* Click on the workflow name, and you can monitor the jobs running in this workflow in this consolidated view:

![image](https://hackmd.io/_uploads/Sk9-g7mkC.png)

### Individual Job Logs

To view the execution log of any particular job:

* Click on the job-ID.
* Navigate to the `Attachments` tab.
* Click `view` (eye-icon) on the `stdout.autosave` file.

![Individual Job Logs](https://baywatch-api.memverge.com/media/upload/202404/240420002911486113.png)

---

### Resuming Workflows with the Job-Submit Script

The job-submit script also allows the resuming of failed Nextflow workflows. You may simply resubmit your previous command again, using the same Bucket. This time, ensure that your Nextflow command include a `-resume`.
    
---

## Create Job Templates to Launch via MMCLoud GUI

Job Templates allow for the ease and customaization of runs that follow a similar format, without having the need to manually set up a command every time. It requires the submission of one job first.

* From the `Jobs` dashboard, select the jfs-service job previously submitted above and click on `More Actions` -> `Save as Template` (NOTE: this is still a BETA feature):

![image](https://hackmd.io/_uploads/rJugK78pp.png)

* Provide a name and tag for the private template:

![image](https://hackmd.io/_uploads/H1kBCGmkC.png)

* Navigate to the `Job Templates` dashboard and click on `Private` templates:

![image](https://hackmd.io/_uploads/rk-FRGmyA.png)

![](https://baywatch-api.memverge.com/media/upload/202404/240420012828342796.png)

* You can click on any job template, edit/change samplesheet, variables etc., and submit new jobs from the GUI.

![image](https://hackmd.io/_uploads/ry88l6GxA.png)

> Users can also submit jobs from templates via CLI
>```bash!
>float submit --template private::<template-name>:<template-tag> \
>-e BUCKET=gs://<work-bucket>
>```

> Addtionally, please keep in mind the features that need to be updated with every run if they deviate from the default values provided in the private template. This will mainly include:
> * GCS Bucket URL
> * Updating of the nextflow run command in the job script

---