I recently hosted a panel with AWS, Amgen, Ardigen, and Seqera on running omics workflows in the cloud. 


Here were my key takeaways from the talk: 

You can watch the full video here: 

<iframe width="560" height="315" src="https://www.youtube.com/embed/zQyh7LF6_h4?si=4QVkjvxM0E6Kj5Oq" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

## Running omics workflows on the cloud is about understanding what tradeoffs you’re making. 

<iframe width="560" height="315" src="https://www.youtube.com/embed/T_keBXj_iuQ?si=KhA4UWL9n5647YkQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

“For example if you're running a production pipeline where you just need to keep it churning, but you don't need to have the results immediately, you can wait and deal with retries. You might also not necessarily need the latest and greatest hardware. 

However, if you're in an R&D activity, the time of your developers is more expensive than the time of the machines. 

So you need to be able to have your developers progress very rapidly, be able to iterate extremely rapidly, because the cost you're paying the humans who are operating machines is higher, and also the cost of the opportunity cost of needing to define the solution faster than maybe your competitors.

What's more important?” 

Getting the answer fast? Getting to a product fast? Or do you have just something that's running in your background?” 


## When going full cloud, it’s important to understand what kinds of CPUs you need to for your pipeline.

<iframe width="560" height="315" src="https://www.youtube.com/embed/D1o36dL5tGQ?si=gEsVaUN5zSuENptI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

“Depending on the shape and size of the data and the kind of scientific algorithm that you're running, there are going to be different computational requirements.

In some cases you might want a beefy VM with a lot of compute, a lot of memory. Sometimes you just, it's going to be a small operation that's just cheap, but it's more disk bound. 

So there's some very different shapes and sizes of machine that might be appropriate.” 


## Case Study: How the Broad Institute leverages spot instances to reduce costs

<iframe width="560" height="315" src="https://www.youtube.com/embed/-OEZnDIB0fs?si=BJTFvermcyj8DUcf" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

The Broad Institute saved ~90% on costs going from their first implementation to their final pipeline. By leveraging tools like Spot instances, they were able to take their costs from $85 per whole genome to $5. 

“When Broad's original genomic pipeline was migrated from on prem to cloud, the initial,  completely naive implementation was about, I want to say $85 per whole genome. With some basic optimization, they very quickly brought that down to $45 per genome, which is almost half. 

Three to six months of further investment, (today it would take much less time because we have all these tools), we took it down to $5 a genome.

And if you think about the cost reduction there, that was huge. Nowadays you can use those kinds of approaches. You don't have to do six months of R&D because a lot of the tools exist. But you can really make a really big difference from the naive implementation to an optimized implementation.”



## Ready to Learn More? 
Spot instances are a great way to use spare capacity in the cloud at 60 to 80% cheaper. 

If you're curious about seamlessly running your research and other types of computational workloads on spot instances, feel free to [book a chat with me](https://calendly.com/jing-memverge/30-minute-general-discussion?month=2023-12&date=2023-12-05). 


## More about Jing
I'm Jing, I run GTM for our memory snapshot product at MemVerge.

<iframe width="560" height="315" src="https://www.youtube.com/embed/gW2uf5-LEYM?si=-ND4_wi-YSGYxGAb" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

We protect your workloads from being taken away or reclaimed by the cloud provider, even when they're not done running. And we are automatically finding you additional instances where you can continue to run your work to completion.

I love helping teams who may not have the resources to conduct their computationally intensive research locally or may not feel fully comfortable migrating their omics workflows to the cloud. 

I post regularly about running omics workflows on spot instances and in the cloud. Check out my writings on [LinkedIn](https://www.linkedin.com/in/jingxie/).

## Abstract

This blog provides the evolution and the overview of the Memverge JuiceFlow solution, which is meant for highly performant and cost-effective Nextflow pipeline executions on Memory Machine Cloud. 

---

## What is JuiceFlow?

JuiceFlow is a transient, head-node setup that utilizes **JuiceFS** as its primary filesystem to execute Nextflow pipelines on MMCloud. [JuiceFS](https://juicefs.com/docs/community/introduction/), an open-source, high-performance distributed file system designed for cloud environments, brings several unique features to the table, including:

- **Separation of Data and Metadata**: JuiceFS stores file data in chunks within object storage platforms like Amazon S3, while metadata is stored in various databases, such as Redis. This architecture enhances performance and scalability.
- **Performance**: JuiceFS is engineered to achieve millisecond-level latency and nearly unlimited throughput, depending on the object storage's scale. This makes it exceptionally suitable for high-performance computing environments like those needed for Nextflow pipelines.
- **Easy Integration with MMCloud**: MMCloud facilitates the deployment process by providing pre-configured Nextflow head node templates with JuiceFS setup, simplifying the user's experience.
- **Comparison with S3FS**: JuiceFS offers significant performance and scalability advantages over S3FS. For those interested in a detailed comparison, refer to the [JuiceFS vs. S3FS](https://juicefs.com/docs/community/comparison/juicefs_vs_s3fs/) documentation.

Once a Nextflow run concludes, whether due to failure or completion, the node will self-terminate, thus the term "transient." Furthermore, JuiceFlow can reload previous runs from your S3 bucket, provided that one metadata file exists per bucket. This feature enables you to `-resume` runs or access data created by other runs within the same JuiceFS file system, streamlining data management and pipeline execution.

---

## Development of JuiceFlow

* The development of the JuiceFS + Nextflow solution has undergone many iterations, leading to the creation of JuiceFlow.

* Initially, the setup integrated the JuiceFS filesystem and Nextflow within the same instance. However, upon deployment via CLI, users were required to `ssh` into the instance to initiate their run and also manually dispose of the VM after execution.

* In the second iteration, we emphasized the "transient-ness" of the Nextflow head-node while aiming for minimal user intervention. The JuiceFS and Nextflow components were allocated to two separate instances. In this setup, JuiceFS remained persistent, whereas the Nextflow component mounted the filesystem and stayed active until the Nextflow run concluded, underscoring its "transient" aspect. This arrangement eliminated the need for users to `ssh` into the head-node instance, allowing them to interact primarily through two CLI command submissions.

* The ultimate configuration, **JuiceFlow**, merges the JuiceFS and Nextflow components into a single node, akin to the original setup, yet it inherits the "transient" hands-off characteristic of the second iteration. This was achieved through a singular CLI command.

---

## Overall JuiceFlow Setup

* The prerequisites for setting up include a functional MMC OpCenter, an AWS S3 Bucket, and the corresponding S3 access and secret keys.

* The user is required to submit a single command that encapsulates the aforementioned details along with two primary scripts: 
    - `transient_JFS.sh`: Formats the work directory S3 bucket to JuiceFS format.
    - `job-submit.sh`: Contains Nextflow input parameters and MMC config.

* Upon submission, a VM will be initiated. This VM is responsible for formatting a directory within the designated S3 bucket for JuiceFS. If the bucket contains a metadata file from a previous job, it will load the corresponding JuiceFS partition, granting the job access to that specific directory. Within the job, a Nextflow configuration file specific for MMCloud using the `nf-float` plugin is generated, and the designated Nextflow command is executed.

* The process integrates the use of the [nf-float](https://github.com/MemVerge/nf-float) plugin, which facilitates the spawning of additional instances for each process within the pipeline on MMCloud. These VMs are configured to terminate upon the completion of their respective processes.

* Following the execution of the pipeline, regardless of the outcome (failure or success), JuiceFlow will archive the metadata of the JuiceFS filesystem to the user's work directory S3 bucket. Should JuiceFlow be executed again using the same bucket, it will recognize the metadata file and reload the identical JuiceFS Filesystem. This feature is particularly beneficial for instances where the `-resume` command is utilized in Nextflow executions.

* For a more detailed setup, please refer to the [JuiceFlow AWS QuickStart Guide](https://www.mmcloud.io/resources/docs/juiceflow).

---

## Important Considerations

- **JuiceFS Bucket Requirements**: JuiceFS cannot be formatted on a sub-directory of a bucket. The root directory of the bucket, or the bucket itself, must be specified in the CLI command.

- **Google Cloud Platform Access Requirements**: For GCP users, the service account must have permissions to get and put objects into your `gs://` bucket.

- **Handling `gcp_key.json`**: This key serves as the credential for a service account with access to your `gs://` bucket. To incorporate the contents into the job, it is read into an environment variable that the container can access.

---

## Monitoring Execution Logs

### Workflow Execution Log

Once the head node job starts executing, you can monitor the Nextflow standard output in OpCenter GUI as follows:
- Click on the head-node job-ID.
- Navigate to the `Attachments` tab.
- Click "view" (eye-icon) on the `stdout.autosave` file.

![Workflow Execution Log](https://hackmd.io/_uploads/SJFcJlIlC.png)

### Individual Job Logs

To monitor workflow execution and get a detailed view of each process in the Nextflow workflow:
- Click on the Workflows dashboard in the OpCenter.

![Individual Job Logs - Workflow Dashboard](https://hackmd.io/_uploads/SyumxlIx0.png)

- Click on the workflow name. This allows you to monitor the jobs running in this workflow in a consolidated view.

![Consolidated Workflow View](https://hackmd.io/_uploads/rkpHeg8gR.png)

- To view the execution log of any particular job:
    - Click on the job-ID.
    - Navigate to the `Attachments` tab.
    - Click "view" (eye-icon) on the `stdout.autosave` file.

![View Execution Log](https://hackmd.io/_uploads/SkvG-lIx0.png)

---

<details>
<summary>Expand here for more details on work directory inspection instructions</summary>

To inspect the files in the work directory, follow these steps:

- Submit a job similar to the commands above with a script designed to "sleep forever," as shown in the script below:

```bash
#!/bin/bash
while true
do
  sleep 1s
done
```

- You can now `ssh` into the instance and inspect all files in the work directory by navigating to `/mnt/jfs`. There's no need to enter the container since JFS is mounted to the instance itself. It's also possible to create a template for easier execution, but ensure to specify the correct bucket in the ENV variables.

**NOTE**: This method assumes that no files in the JFS mount are being changed, as we are not updating the metadata in this setup. This is intended only for reading/copying logs. Should you alter anything in the file system, be aware that changes will NOT be saved unless the metadata is updated.

</details>

---

## Performance Metrics

* Below are some key performance & cost metrics of JuiceFlow using the `test_full` profiles of popular nf-core pipelines

| Pipeline        | Execution Time | Execution Cost ($) |
| --------------- | -------------- | ------------------ |
| nf-core/sarek   | 9hr 32min      | 6.3288             |
| nf-core/atacseq | 4hr 22min      | 6.6143             |
| nf-core/rnaseq  | 8hr 00min      | 21.8443            |

NOTE: All pipeline jobs were run on AWS Spot instances, utilizing MMCloud's spot-protection feature. This feature automatically saves job progress upon spot instance preemption and restores it on a new spot instance, ensuring continuity.

* We previously published detailed benchmark results on performance of JuiceFS for Nextflow Workflows in a blogpost that can be viewed [here](https://www.mmcloud.io/blog/unveiling-the-superiority-of-mmcloud-over-aws-batc)

---

## JuiceFlow Advantages

- **Transient node**: Once the Nextflow pipeline has finished running, the entire node is terminated, eliminating further costs for the user.

- **One-command submission**: JuiceFlow allows the pipeline to be executed with a single command, negating the need for the user to `ssh` into the instance and thus providing a hands-off experience.

---

## Summary

JuiceFlow revolutionizes Nextflow pipeline executions on MMCloud by leveraging JuiceFS for high-performance, cost-effective computing. It introduces a seamless integration that drastically reduces execution times and costs, exemplified by nf-core pipelines running on AWS Spot instances with impressive savings and efficiency. The system's transient nature ensures that nodes terminate post-execution, eliminating unnecessary expenses. Furthermore, JuiceFlow's one-command setup significantly simplifies user interaction, reinforcing its position as a superior choice for executing complex computational workflows with optimal resource utilization and budget control.

# Run Nextflow Workflows on MMCloud via Seqera Platform

### Nextflow and Seqera Platform: A Powerful Duo

* [Seqera Platform](https://seqera.io/platform/) empowers users to launch Nextflow workflows across a variety of [executors](https://www.nextflow.io/docs/latest/executor.html). With Nextflow's executor abstraction, your pipeline processes can be effortlessly transitioned between different environments - from local machines to clusters and cloud systems.

* Seqera Platform has established itself as the go-to solution for deploying, monitoring, and troubleshooting Nextflow pipelines in hybrid environments that blend on-premise resources with cloud-based systems. The platform's versatility makes it a vital tool for today's diverse computational requirements.

### Memory Machine Cloud (MMCloud): Optimizing Resource Management

**[MMCloud](https://www.mmcloud.io/)** stands out with its advanced approach to resource optimization, particularly in CPU and memory management. This capability makes MMCloud an excellent choice for handling complex, memory-heavy tasks, typical of demanding computational workloads and data-intensive genomic applications. MMCloud's efficiency [surpasses traditional cloud solutions](https://www.mmcloud.io/blog/unveiling-the-superiority-of-mmcloud-over-aws-batc), offering a more effective platform for resource-intensive operations.

### Seamless Integration for Hybrid Environments

We are thrilled to present the community with the opportunity to deploy jobs on MMCloud using the **`float`** executor via the Seqera Platform. MMCloud presents a cost-effective and dependable cloud solution for running Nextflow pipelines, marking a significant advancement in cloud-based genomic data processing and analysis.

**Pre-requisites:**
* AWS Batch compute environment setup on Seqera Platform
* MMCloud license & OpCenter setup

## Steps:

>**NOTE: The following steps will use AWS Batch to provision an on-demand instance for the Nextflow head node and subsequently apply the `mmcloud float configuration` to the Nextflow run command. This effectively changes the executor to `float`, thereby submitting the jobs to MMCloud for execution.**

1. Login to your Seqera Platform account.
2. Create or use an existing AWS Batch compute environment that is configured to launch `on-demand` instances.

![Seqera Platform AWS Batch Configuration](https://hackmd.io/_uploads/ryJmNjsIa.png)

3. From the `Launchpad`, add a pipeline to run or edit an existing one.
4. Under `Advanced Options`, provide the additional MMCloud configuration as shown below to use MMCloud's `float` executor.

![MMCloud Float Configuration](https://hackmd.io/_uploads/B1NFSjjIa.png)

* Change the configuration below to provide your `workDir`, `float` & `aws` credentials

> **MMCloud float configuration**
```bash
plugins {
     id 'nf-float'
}

process {
    executor      = 'float'
    errorStrategy = 'retry'
}

float {
   address  = '10.102.34.687'
   username = 'user'
   password = 'pass'
}

aws {
   accessKey = 'access'
   secretKey = 'secret'
}
```

5. Include the following bash command in the **Pre-run script** section. (This is to disable the AWS secrets provider which runs AWS Batch by default) 

```bash
unset AWS_BATCH_JOB_ID
```

![Pre-run Script Configuration](https://hackmd.io/_uploads/S1EsrjsI6.png)

6. Launch the workflow.

---

## Enhanced Benefits of Deploying Nextflow Workflows on MMCloud

MMCloud, through its **[nf-float](https://github.com/MemVerge/nf-float)** plugin, bridges Nextflow with the cloud, bringing a suite of advanced features and services that enhance the efficiency and flexibility of cloud-based workflow deployment.

### WaveWatcher: Your Window to Real-Time Resource Management

WaveWatcher, a key component of MMCloud, provides real-time visibility into application resource usage. This service is essential for optimizing performance and reducing cloud costs by pinpointing and eliminating inefficiencies.

#### Highlights:

- **Live Resource Monitoring**: Offers immediate insights into resource utilization, enabling timely adjustments.
- **Resource Optimization**: Tailors resource allocation to specific task needs, maximizing efficiency.
- **Cost-Saving Insights**: Identifies and helps eliminate wasteful resource usage.
- **Accessible Interface**: Simplifies complex data, making it user-friendly for all skill levels.

WaveWatcher is more than a monitoring tool; it's a vital component for efficient resource management, ensuring that cloud resources are used optimally.

![Wave-Watcher Graphs](https://hackmd.io/_uploads/HkO-1noIa.png)

![Wave-Watcher Graphs](https://hackmd.io/_uploads/H1yNy3j8p.png)

![Wave-Watcher Graphs](https://hackmd.io/_uploads/HktHJnsUp.png)

### SpotSurfer: Ensuring Continuity on Spot Instances

SpotSurfer introduces a robust checkpointing and recovery service for stateful applications running on AWS Spot EC2 instances, significantly enhancing their stability and reducing cloud costs by up to 90%.

#### Key Advantages:

- **Reliable Stateful Application Operation**: Mitigates risks associated with Spot instance interruptions.
- **Significant Cost Reduction**: Offers the economic benefits of Spot instances without compromising application stability.
- **Checkpointing for Safety**: Periodically saves application states for quick recovery.
- **Optimized for Spot EC2**: Tailored to exploit Spot instance benefits while safeguarding data integrity.

* By default, the VM selection policy attached to all float jobs is `SpotFirst`.

![Spot-Surfer Policy](https://hackmd.io/_uploads/r1PmKosLa.png)

* To change the default VM policy for the entire workflow, add it to the `float {}` scope as below:

> Launch all jobs on on-demand instances
```groovy
float {
    address     = '10.102.34.687'
    username    = 'user'
    password    = 'pass'
    commonExtra = '--vmPolicy [onDemand=true]' 
}
```

> Launch all jobs on spot instances only
```groovy
float {
    address     = '10.102.34.687'
    username    = 'user'
    password    = 'pass'
    commonExtra = '--vmPolicy [spotOnly=true,retryLimit=3,retryInterval=200s]' 
}
```

* To change the VM policy for a specific process within the workflow, add it to the `process {}` scope using the `withName:` selector as shown below:

> Launch a specific job on on-demand instance
```groovy
process {
    executor      = 'float'
    errorStrategy = 'retry'
    withName: "STARALIGN" {
          extra   ='  --vmPolicy [onDemand=true]'
        }
}
```

SpotSurfer redefines cloud efficiency, combining cost-effectiveness with operational reliability.

### WaveRider: Intelligent Cloud Resource Right-Sizing
WaveRider's continuous right-sizing service dynamically balances cost and performance of cloud resources, ensuring that they align perfectly with application needs throughout runtime.

#### Core Features:
- **Auto-RightSizing**: Dynamically adjusts resources to match real-time demands.
- **Cost Optimization**: Continuously refines resource allocation to minimize unnecessary expenditure.
- **Performance Focus**: Ensures that applications always have the resources they need to perform optimally.
- **Seamless Cloud Integration**: Integrates effortlessly with existing cloud setups.

#### Default and Customizable `migratePolicy` for Flexible Management:

- MMCloud’s default `migratePolicy` ensures balanced CPU and memory usage, with options to customize these settings for specific workflow or process requirements.

![Default Migrate Policy](https://hackmd.io/_uploads/BkWUpsoL6.png)

* The default `migratePolicy` applied to all float jobs is as follows:

> Default migratePolicy
```groovy
--migratePolicy [cpu.upperBoundRatio=90,
                 cpu.lowerBoundRatio=5,
                 cpu.upperBoundDuration=120s,
                 cpu.lowerBoundDuration=300s,
                 cpu.limit=0,
                 cpu.lowerLimit=0,
                 mem.upperBoundRatio=90,
                 mem.lowerBoundRatio=5,
                 mem.upperBoundDuration=120s,
                 mem.lowerBoundDuration=300s,
                 mem.limit=0,
                 mem.lowerLimit=0,
                 stepAuto=true,
                 evadeOOM=true] 
```

* To override the default `migratePolicy` for all processes within the workflow, apply them under the `float {}` scope:

> Change the default migratePolicy for all jobs
```groovy!
float {
    address     = '10.102.34.687'
    username    = 'user'
    password    = 'pass'
    commonExtra = '--migratePolicy [cpu.upperBoundRatio=95,cpu.lowerBoundRatio=4,cpu.upperBoundDuration=150s,cpu.lowerBoundDuration=350s,cpu.limit=0,cpu.lowerLimit=0,mem.upperBoundRatio=75,mem.lowerBoundRatio=4,mem.upperBoundDuration=150s,mem.lowerBoundDuration=320s,mem.limit=0,mem.lowerLimit=0,stepAuto=true,evadeOOM=true]' 
}
```

* To override the default `migratePolicy` for specific processes within the workflow, apply them under the `process {}` scope using the `withName:` selector as shown below:

> Change the default migratePolicy for a specific job
```groovy!
process {
      executor      = 'float'
      errorStrategy = 'retry'
      withName: "STARALIGN" {
            extra   ='  --migratePolicy [cpu.upperBoundRatio=95,cpu.lowerBoundRatio=4,cpu.upperBoundDuration=150s,cpu.lowerBoundDuration=350s,cpu.limit=0,cpu.lowerLimit=0,mem.upperBoundRatio=75,mem.lowerBoundRatio=4,mem.upperBoundDuration=150s,mem.lowerBoundDuration=320s,mem.limit=0,mem.lowerLimit=0,stepAuto=true,evadeOOM=true]'
    }
}
```

WaveRider is essential for maximizing cloud resource efficiency, striking the right balance between cost and performance.

Together, these services enhance MMCloud's capability to host Nextflow workflows, providing a robust, efficient, and cost-effective cloud environment.

## Current Limitations

* The default file system used is **[S3FS](https://github.com/s3fs-fuse/s3fs-fuse)**, a utility that allows users to mount object storage buckets locally and read and write in a way that the users used to. It targets general use scenarios that are not sensitive to performance and network latency.

* Users can enable use of Fusion FS in their float config as shown below. However, **Checkpoint/restore feature required for Spot-Surfer / Wave-Rider is not available currently when using Fusion file system**

>```bash!
>wave {
>  enabled = true
>}
>
>fusion {
>    enabled                  = true
>    exportStorageCredentials = true
>    exportAwsAccessKeys      = true
>}
>```

* MMCloud's JuiceFS implementation is also not available via this setup currently.

Anyway you look at it, the Chinese mitten crab (also known as the Shanghai hairy crab because of the setae on its claws) has something to offer. For the chef, a seasonal delicacy — full of sweetness and umami flavors. To the aquaculturalist, an important crop and revenue source. This crab is a marvel to biologists — it can live in freshwater and saltwater? Regenerate limbs? Really?

For the geneticist, the challenge is to trace these phylogenetic behaviors (behaviors typical of the species) back to the genome and transcriptome level.

![](https://baywatch-api.memverge.com/media/upload/202311/231127195706586901.jpeg)

## Eriocheir sinensis

The Chinese mitten crab (*Eriocheir sinensis*) is native to China where it is found in lakes and rivers. In Europe and North America, it is considered an invasive species. The mitten crab spends most of its life in freshwater although the reproductive phase occurs in brackish water and in seawater (the species is catadromous). The change in salinity requires a complicated mechanism (osmoregulation) to maintain the ionic balance in its cells.

The first discovery of Chinese mitten crabs in North America occurred in the San Francisco Bay in [1992](https://wildlife.ca.gov/Conservation/Invasives/Species/Mitten-Crab). Since then, specimens have been found in the Chesapeake Bay, Hudson River, and the Great Lakes. They have been found all over Europe — in the Thames River, the Tagus River, the Danube, and even in the Venice lagoon, although finding an isolated specimen does not mean that a permanent population has established itself. Mitten crabs have a high reproductive rate and can withstand physiological stress (such as starvation and dessication) which makes them a formidable invasive species whose burrows undermine river bank stability. It's a mystery why no Chinese mitten crabs have been seen in California since [2010](https://www.smithsonianmag.com/blogs/smithsonian-environmental-research-center/2018/06/05/wanted-chinese-mitten-crabs/).

Beyond its culinary, agricultural, and environmental impacts, the Chinese mitten crab is worthy of study because of its ability to [regenerate](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9481118/) an entire limb (as long as it can molt). Regeneration capability is stronger in inverterbrates than in vertebrates where it seems to have declined. Birds and  mammals both demonstrate some cell and tissue regeneration capability, so it is an interesting topic of medical research to see if regeneration ability can be stimulated.

It is no surprise, then, that *Eriocheir sinensis* has been studied extensively, in particular, to discover the gene expressions that underly its physiological and behavioral traits.  The first genome sequencing, assembly, and annotation of the Chinese mitten crab was reported in [2016](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4730596/) and the current reference assembly was posted in [2022](https://www.ncbi.nlm.nih.gov/datasets/genome/GCF_024679095.1/). The haploid [chromosome number](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4814232/) is 73 (2n=146) and the genome size is about 1.27 Gb, which puts it the high end of chromosome number among [arthropods](https://www.nature.com/articles/s41467-021-22604-3) (inverterbrates with exoskeletons), although there is a large percentage of repetitive elements. For comparison, the haploid chromosome number for humans is 23 and the haploid genome has about 3.1 Gb.

Nextflow and MMCloud
--------------------

So, what has this to do with Memory Machine Cloud (MMCloud)? RNA-seq is a popular computational pipeline for analysing gene expression and transcription activation using data generated by next-generation sequencing (NGS) machines from Illumina, PacBio, Oxford Nanopore Technologies, and others. A complicated pipeline, like RNA-seq, requires a workflow manager, such as Nextflow, to schedule and manage the individual tasks in the pipeline.


In Nextflow terminology, each task is assigned to an "executor," which is a complete environment for running that step in the analysis. By attaching the [nf-float](https://github.com/MemVerge/nf-float/blob/master/README.md) plugin to a workflow, Nextflow can use MMCloud as an "executor." From an MMCloud point of view, the execution task that Nextflow assigns to it is an independent job that it runs just like any other batch job. For the Nextflow user, the user experience remains the same while benefiting from all the MMCloud features, such as SpotSurfer, WaveRider, and WaveWatcher.

MMCloud release 2.3.3 introduced a POSIX-compliant, distributed file system that allows Nextflow pipelines to use a high-capacity storage service (such as AWS S3) as a high-performance file system. For complicated pipelines where individual tasks share a file system for writing and reading intermediate data, the performance of the file system has a dramatic impact. 

Running a Nextflow pipeline on MMCloud is straightforward. To demonstrate using AWS EC2 compute instances, we ran [nf-core/rnaseq](https://nf-co.re/rnaseq/3.13.2) on experimental *Eriocheir sinensis* sequence data available from the [European Nucleotide Archive](https://www.ebi.ac.uk/ena/browser/text-search?query=Eriocheir%20sinensis) and the [National Library of Medicine](https://www.ncbi.nlm.nih.gov/datasets/genome/?taxon=95602).

The nf-float plugin requires a configuration file, in which you specify parameters that determine the execution environment. One parameter is the VM creation policy. For this run, we chose "SpotFirst", which means that MMCloud tries to start a spot instance. If, after three attempts, a spot instance does not start, MMCloud starts an on-demand instance. If AWS reclaims the spot instance that a job is running on, MMCloud moves the job to a new instance and follows the same policy - try for a spot instance first and go to an on-demand instance if not successful.

## RNA-Seq Results

At the end of the successful run, Nextflow displayed the following message, which shows that 170 tasks (jobs) completed with no failures.

`-[nf-core/rnaseq] Pipeline completed successfully -`
`Completed at: 25-Nov-2023 10:42:32`
`Duration    : 6h 14m 7s`
`CPU hours   : 123.3`
`Succeeded   : 170`

Included in the output of nf-core/rnaseq is [MultiQC](https://multiqc.info/), a reporting tool that summarizes (in html format) the statistics and quality metrics obtained from multiple analysis modules and data sets. MultiQC is helpful in checking the results of a pipeline run for errors and anomalies.

Examples of the output from MultiQC for the *Eriocheir sinensis* run are shown in the figures that follow.

![](https://baywatch-api.memverge.com/media/upload/202311/231127200149299985.png)

![](https://baywatch-api.memverge.com/media/upload/202311/231127200212828822.png)


## Conclusion

The Chinese mitten crab is a fascinating bundle of surprises. There is no doubt that the species will continue to receive attention - on menus, in departments of environmental conservation, and as the subject of a range of "Omics" studies (for example, [transcriptomics](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6328174/)). As researchers become aware of the power of MMCloud, many of those Nextflow pipelines will run on MMCloud in AWS, Google Cloud, or AliCloud.

For the residents of Shanghai, however, the annual appearance of the Shanghai hairy crab is always an eagerly anticipated milestone.

### What is application observability and why it is important

Application observability gains insights into the behavior of software applications utilizing the information from logs, metrics and traces. The goals include but not limited to

- identify bottlenecks and optimize performance
- identify issues and troubleshoot problems quickly
- identify resource requirements and determine the most appropriate corresponding infrastructure

### What is Memory Machine Cloud WaveWatcher and how it can help

The Memory Machine Cloud WaveWatcher service builds on its ability to track CPU, memory, network, storage (size, throughput, and IO) used by a ‘job’ (could be an application, a workflow, or a pipeline). Take the below WaveWatcher result of a Sentieon WGS pipeline as an example:

![WaveWatcher for Sentieon WGS Pipeline](https://baywatch-api.memverge.com/media/upload/202311/231120214738149768.png)

We can see how this pipeline can be characterized into different steps, each with corresponding ‘behaviors.’ The resources the steps consume are shown in the table below:
| Step | Function | Observability - Memory Usage | Observability - CPU Usage | Observability - Network | Observability - Storage Usage | Observability - Storage IO | What can / should we do? |
| - | - | - | - | - | - | - | - |
1 | Validate license and download reference and sample data | extremely low | extremely low | extensive | increasing | extensive | Use a less memory and CPU intensive but network and storage optimized instance |
2 | BWA-MEM | continuously growing | extremely high | none | increasing | medium | Use a memory and CPU optimized instance and larger instance
3 | Dedup | dynamically changing | dynamically changing | zero | dynamically changing | dynamically changing | Use a CPU optimized instance and larger instance |
4 | Variant calling, filtering and genotyping | high | extremely high | zero | retain | zero | Use a memory and CPU optimized instance and medium instance

### Go beyond observability - Memory Machine Cloud WaveRider

It is helpful to gain observability but the ultimate question is - what can be improved, and can it be automated? The data from WaveWatcher can then be used by the WaveRider service to automatically right-size resources, keep the application / pipeline / workflow state, and reduce CO2 footprint, hence to

- avoid under-provision which may result application performance bottleneck or even OOM, and
- avoid over-provision which spends unnecessary money and CO2 footprint.

More detailed Sentieon WGS pipeline benchmark can be found [here](https://www.mmcloud.io/blog/sentieon-genomics-tools-boosting-performance-wit-3).

### Three pillars of Memory Machine Cloud - observability (WaveWatcher), right-sizing (WaveRider), and job automation (Float)

Last but not the least, how to automate the whole process with minimum human intervention? Just like the three pillars of observability (logs, metrics and traces), there are three pillars for Memory Machine Cloud - job automation (Float), observability (WaveWatcher), and right-sizing (WaveRider).

- The user submits a job (for example, a bioinformatics analysis pipeline) via Float CLI or GUI.
- Memory Machine Cloud WaveWatcher automatically observes the behavior including the resources needed
- Memory Machine Cloud WaveRider automatically right-sizes / vertical scaling based on WaveWatcher’s observation supporting stateful job

These three pillars make a complete cloud automation solution removing the hassle of integrations among multiple solutions.

### Comparison of Memory Machine Cloud with modern solutions

Below is a summary table of comparison of Memory Machine Cloud with modern solutions, where Memory Machine Cloud provides an all-in-one solution.
| Category | Comparison | MemVerge MMCloud | Grafana | Datadog | Spot by NetApp | YellowDog | AWS CloudWatch | AWS Compute Optimizer | AWS Batch |
|:----------:|:---------------------------------------------------------------:|:------------------:|:----------:|:----------:|:--------------:|:---------------:|:--------------:|:---------------------:|:----------:|
| Feature | Observability | Yes (WaveWatcher) | Yes | Yes | Yes | Yes | Yes | Yes | No |
| Feature | Automatic right-sizing stateless application | Yes (WaveRider) | No | No | Yes | Yes | No | No | No |
| Feature | Automatic right-sizing stateful, non-fault-tolerant application | Yes (WaveRider) | No | No | No | No | No | No | No |
| Feature | Job automation | Yes (Float) | No | No | No | Yes | No | No | Yes |
| Deployment | On-prem version | Yes | Yes | No | No | Yes | No | No | No |
| Pricing | Pricing | All-in-one license | By license | By service | By license | Basic + add-ons | By service | Opt-in | By service |

\*note - more detailed comparisons with Spot by NetApp can be found [here](https://www.mmcloud.io/blog/memory-machine-cloud-vs-spot-by-netapp)

### Call for action - save your spending and time

Do you need to run stateful, non-fault-tolerant workloads or pipelines on the cloud? Would you like to benefit from automatic right-sizing and vertical scaling and reduce both cloud cost and job running hours without worrying about integrations among multiple solutions?  [Please try Memory Machine Cloud for free](https://www.mmcloud.io/customer/register).

Blog - Bioinformatics on the Cloud