## What is AppCapsule and what problems does it solve? 
AppCapsule creates a snapshot of all the data in memory, and all the application states required to restore the application at this moment in time, as well as the data on storage. With AppCapsule, we can transport the application runtime across different machines. 

Three problems that AppCapsule can help to solve - 
1. **Spot** reclaim - If there is a spot eviction event, Memory Machine Cloud creates an AppCapsule, and moves it to another spot instance, where the job will continue. That means, you can save cost running jobs on spot instance, while keeping all your job progress and does not need to run from the beginning all over again. A.k.a. *SpotSurfer*.
2. **OOM** (Out Of Memory) - Memory Machine Cloud detects if OOM is about to happen and if so, creates an AppCapsule, and moves it to another instance to avoid the application crash as well as have the job continue running.
3. **Right-sizing** - vertical scaling down to avoid over-provisioning or up to provide more resources to the application as it is needed for shorter wall-time and possibly lower total cost. You can trigger either manually by using float cli command / OpCenter Web GUI or automatically done by Memory Machine Cloud (a.k.a. *WaveRider*).
   
![](https://baywatch-api.memverge.com/media/upload/202312/231220190505523032.png)

## What is AppCapsule++ and what benefits does it bring? 
Three enhancements and benefits that AppCapsule++ brings to the users - 

1. Only saving **delta** - instead of saving a full snapshot, AppCapsule++ only saves the difference (memory change) since the previous snapshot. This way, the time required for the final snapshot is minimal.
2. **Adaptive** trigger - instead of snapshotting in a fixed, pre-determined interval, AppCapsule++ automatically and intelligently checks the memory change and take a snapshot when it reaches a threshold, which is calculated based on the cloud service provider’s pre-emption window. The benefit is no need to guess the fixed interval for periodic snapshot.
3. **Asynchronous offloading** - instead of synchronous snapshotting which will pause the application, AppCapsule++ does asynchronous data offloading, which decreases the time that the application is frozen. The benefit is lower impact to job completion wall-time. 

Below is an overall comparison table of AppCapsule and AppCapsule++
| Comparison | AppCapsule | AppCapsule++ |
| - | - | - |
| **Snapshot content** | Full snapshot | Delta snapshot |
| **Snapshot trigger** | Can be<br/>- Periodic done by MMCloud, or<br/>- Manual done by user’s script | Adaptive | 
| **Configuration** | Pre-determined interval for periodic snapshot | No special configuration | 
| **Application completion wall-time** | Impact higher<br/>(Synchronous) | Impact lower<br/>(Asynchronous) | 
| **Storage overhead** | Relatively lower<br/>(only need to save final snapshot to restore) | Relatively higher<br/>(need to save all delta snapshots to restore) | 
| **Usage** | `--dumpMode full` | `--dumpMode incremental` |


## Benchmark comparison 
Below is the benchmark experiment result running on AWS comparing AppCapsule and AppCapsule++. The experiment is based on 50GB memory change in terms of 60GB, 120GB, 240GB, and 480GB application memory usage, and how much time in seconds did AppCapsule and AppCapsule++ take to take the final snapshot. 

Three observations from this benchmark - 

1. For AppCapsule, as application memory usage grows, the snapshot time also **increases**. The reason is that AppCapsule takes a complete snapshot every time.
2. For AppCapsule++, as application memory usage grows, the snapshot time **stays more or less the same**. The reason is that AppCapsule++ takes a delta snapshot, which is triggered when the amount of change in memory reaches a certain threshold.
3. Considering using spot instance on AWS, which has 2-minute pre-emption time window, AppCapsule can only support relatively small workload (in this experiment for example, 60GB since the snapshot time is 109 seconds). AppCapsule will not be able to support bigger workload (in this experiment for example, 120GB needs 250 seconds to finish a snapshot which is way over the 2-minute time window). However, since AppCapsule++ only takes delta snapshot, the snapshot time is consistent with memory change, not the application memory usage. 
![AppCapsule vs AppCapsule++](https://baywatch-api.memverge.com/media/upload/202312/231220185252866864.png)


More detailed information in table format below: 
| Experiment No. | Experiment Variable | AppCapsule | AppCapsule++ |
| - | - | - | - |
| 1 | Application memory usage: 60G<br/>Application memory change: 50G | 109s<br/>r5.2xlarge/gp3 | 91s<br/>r5.2xlarge/gp3 |
| 2 | Application memory usage: 120G<br/>Application memory change: 50G | 250s<br/>r5.4xlarge/io1 | 102s<br/>r5.4xlarge/io1 |
| 3 | Application memory usage: 240G<br/>Application memory change: 50G | 533s<br/>r5.8xlarge/io1 | 104s<br/>r5.8xlarge/io1 |
| 4 | Application memory usage: 480G<br/>Application memory change: 50G | 1548s<br/>r5.16xlarge/io1 | 80s<br/>r5.16xlarge/io1 |

## Call to action - save your spending and time 
What is the characteristic of your running job and environment? AppCapsule++ is available now with Memory Machine Cloud 2.4 Goa release. Please reach out or leave comments below and we would love to chat and help!

Anyway you look at it, the Chinese mitten crab (also known as the Shanghai hairy crab because of the setae on its claws) has something to offer. For the chef, a seasonal delicacy — full of sweetness and umami flavors. To the aquaculturalist, an important crop and revenue source. This crab is a marvel to biologists — it can live in freshwater and saltwater? Regenerate limbs? Really?

For the geneticist, the challenge is to trace these phylogenetic behaviors (behaviors typical of the species) back to the genome and transcriptome level.

![](https://baywatch-api.memverge.com/media/upload/202311/231127195706586901.jpeg)

## Eriocheir sinensis

The Chinese mitten crab (*Eriocheir sinensis*) is native to China where it is found in lakes and rivers. In Europe and North America, it is considered an invasive species. The mitten crab spends most of its life in freshwater although the reproductive phase occurs in brackish water and in seawater (the species is catadromous). The change in salinity requires a complicated mechanism (osmoregulation) to maintain the ionic balance in its cells.

The first discovery of Chinese mitten crabs in North America occurred in the San Francisco Bay in [1992](https://wildlife.ca.gov/Conservation/Invasives/Species/Mitten-Crab). Since then, specimens have been found in the Chesapeake Bay, Hudson River, and the Great Lakes. They have been found all over Europe — in the Thames River, the Tagus River, the Danube, and even in the Venice lagoon, although finding an isolated specimen does not mean that a permanent population has established itself. Mitten crabs have a high reproductive rate and can withstand physiological stress (such as starvation and dessication) which makes them a formidable invasive species whose burrows undermine river bank stability. It's a mystery why no Chinese mitten crabs have been seen in California since [2010](https://www.smithsonianmag.com/blogs/smithsonian-environmental-research-center/2018/06/05/wanted-chinese-mitten-crabs/).

Beyond its culinary, agricultural, and environmental impacts, the Chinese mitten crab is worthy of study because of its ability to [regenerate](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9481118/) an entire limb (as long as it can molt). Regeneration capability is stronger in inverterbrates than in vertebrates where it seems to have declined. Birds and  mammals both demonstrate some cell and tissue regeneration capability, so it is an interesting topic of medical research to see if regeneration ability can be stimulated.

It is no surprise, then, that *Eriocheir sinensis* has been studied extensively, in particular, to discover the gene expressions that underly its physiological and behavioral traits.  The first genome sequencing, assembly, and annotation of the Chinese mitten crab was reported in [2016](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4730596/) and the current reference assembly was posted in [2022](https://www.ncbi.nlm.nih.gov/datasets/genome/GCF_024679095.1/). The haploid [chromosome number](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4814232/) is 73 (2n=146) and the genome size is about 1.27 Gb, which puts it the high end of chromosome number among [arthropods](https://www.nature.com/articles/s41467-021-22604-3) (inverterbrates with exoskeletons), although there is a large percentage of repetitive elements. For comparison, the haploid chromosome number for humans is 23 and the haploid genome has about 3.1 Gb.

Nextflow and MMCloud
--------------------

So, what has this to do with Memory Machine Cloud (MMCloud)? RNA-seq is a popular computational pipeline for analysing gene expression and transcription activation using data generated by next-generation sequencing (NGS) machines from Illumina, PacBio, Oxford Nanopore Technologies, and others. A complicated pipeline, like RNA-seq, requires a workflow manager, such as Nextflow, to schedule and manage the individual tasks in the pipeline.


In Nextflow terminology, each task is assigned to an "executor," which is a complete environment for running that step in the analysis. By attaching the [nf-float](https://github.com/MemVerge/nf-float/blob/master/README.md) plugin to a workflow, Nextflow can use MMCloud as an "executor." From an MMCloud point of view, the execution task that Nextflow assigns to it is an independent job that it runs just like any other batch job. For the Nextflow user, the user experience remains the same while benefiting from all the MMCloud features, such as SpotSurfer, WaveRider, and WaveWatcher.

MMCloud release 2.3.3 introduced a POSIX-compliant, distributed file system that allows Nextflow pipelines to use a high-capacity storage service (such as AWS S3) as a high-performance file system. For complicated pipelines where individual tasks share a file system for writing and reading intermediate data, the performance of the file system has a dramatic impact. 

Running a Nextflow pipeline on MMCloud is straightforward. To demonstrate using AWS EC2 compute instances, we ran [nf-core/rnaseq](https://nf-co.re/rnaseq/3.13.2) on experimental *Eriocheir sinensis* sequence data available from the [European Nucleotide Archive](https://www.ebi.ac.uk/ena/browser/text-search?query=Eriocheir%20sinensis) and the [National Library of Medicine](https://www.ncbi.nlm.nih.gov/datasets/genome/?taxon=95602).

The nf-float plugin requires a configuration file, in which you specify parameters that determine the execution environment. One parameter is the VM creation policy. For this run, we chose "SpotFirst", which means that MMCloud tries to start a spot instance. If, after three attempts, a spot instance does not start, MMCloud starts an on-demand instance. If AWS reclaims the spot instance that a job is running on, MMCloud moves the job to a new instance and follows the same policy - try for a spot instance first and go to an on-demand instance if not successful.

## RNA-Seq Results

At the end of the successful run, Nextflow displayed the following message, which shows that 170 tasks (jobs) completed with no failures.

`-[nf-core/rnaseq] Pipeline completed successfully -`
`Completed at: 25-Nov-2023 10:42:32`
`Duration    : 6h 14m 7s`
`CPU hours   : 123.3`
`Succeeded   : 170`

Included in the output of nf-core/rnaseq is [MultiQC](https://multiqc.info/), a reporting tool that summarizes (in html format) the statistics and quality metrics obtained from multiple analysis modules and data sets. MultiQC is helpful in checking the results of a pipeline run for errors and anomalies.

Examples of the output from MultiQC for the *Eriocheir sinensis* run are shown in the figures that follow.

![](https://baywatch-api.memverge.com/media/upload/202311/231127200149299985.png)

![](https://baywatch-api.memverge.com/media/upload/202311/231127200212828822.png)


## Conclusion

The Chinese mitten crab is a fascinating bundle of surprises. There is no doubt that the species will continue to receive attention - on menus, in departments of environmental conservation, and as the subject of a range of "Omics" studies (for example, [transcriptomics](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6328174/)). As researchers become aware of the power of MMCloud, many of those Nextflow pipelines will run on MMCloud in AWS, Google Cloud, or AliCloud.

For the residents of Shanghai, however, the annual appearance of the Shanghai hairy crab is always an eagerly anticipated milestone.

<h2>MemVerge nf-float plugin is now included in the Nextflow plugins index</h2><p>MemVerge is pleased to announce our Nextflow plugin is now included in the Nextflow plugins index repository&nbsp;<a href="https://github.com/MemVerge/nf-float" rel="noopener noreferrer" target="_blank" style="color: inherit;">on GitHub</a>. This means that any process in a Nextflow pipeline can call MemVerge Memory Machine Cloud as an executor. Nextflow will automatically search the Plugins index and download the latest version of nf-float.</p><p>Nextflow currently supports a broad range of executors. Processes can be scheduled to run on the local host, on an HPC cluster, on Kubernetes, or on a cloud platform such as AWS or GCP. Memory Machine Cloud adds several features that provide significant value to Nextflow users in the cloud.</p><p><a href="https://www.mmcloud.io/blog/unveiling-the-mmcloud-nf-surfer-program-a-golden-o" rel="noopener noreferrer" target="_blank"><img src="/img/case_studies/nextflow_04.png" alt="Nextflow"></a></p><h2>nf-float unlocks the added value of Memory Machine Cloud</h2><p>Memory Machine Cloud is software that is deployed on virtual machines in any cloud including AWS, GCP and AliCloud today, with more coming soon.</p><h2>OpCenter</h2><p>The heart of Memory Machine Cloud is the OpCenter that schedules jobs and instantiates the appropriate resources to run containerized applications. The OpCenter continually monitors the resources consumed by jobs</p><h2>WaveRider Continuous Optimization Service</h2><p>Based on user-supplied policy, moves running jobs to optimally-sized virtual machines, all without losing state or interim progress. This WaveRider service enables users to tune cost and wall clock time during job runtime.</p><h2>SpotSurfer Checkpointing and Recovery Service</h2><p>Spot instances are a way for cloud service providers to monetize idle compute capacity. A Spot instance may be offered at a discount of up to 90% off the price of an On-demand instance. There is a catch — a Spot instance may be reclaimed by the cloud service providers with a typical warning of only two minutes. Memory Machine Cloud includes the SpotSurfer checkpoint and restore service that allows a running job to move seamlessly from a Spot instance flagged for reclamation to a new Spot or on-demand instance. This feature, called SpotSurfer, allows users to reduce cloud costs significantly without increasing wall clock time.</p><p><strong><a href="https://www.mmcloud.io/case-studies/tgen" rel="noopener noreferrer" target="_blank">See how TGen used Nextflow &amp; SpotSurfer</a></strong></p><h2>WaveWatcher</h2><p>When the OpCenter launches a worker node, the OpCenter opens a communication channel to a resource monitor inside the container. Real-time metrics are available as a CSV file or can be viewed graphically using the OpCenter web interface. This feature, called WaveWatcher, provides Nextflow users with a detailed view of how executors are consuming compute and storage resources, an insight not available otherwise.</p><p>What does this mean for Nextflow users? Memory Machine Cloud is a feature-rich executor, easily deployed with a few lines in the Nextflow configuration file. The benefits include cost savings, improved time-to-results, and deep insight into resource utilization at the container level.</p><p>Read&nbsp;<a href="https://docs.memverge.com/mmce/2.1/howto/nextflow/pdf/mmce_2_1_howto_nextflow.pdf" rel="noopener noreferrer" target="_blank" style="color: inherit;">Using Nextflow with Memory Machine Cloud&nbsp;</a>for detailed instructions on how to deploy Memory Machine Cloud with its Nextflow plugin.</p>

Blog - mmce.pro