I recently hosted a panel with AWS, Amgen, Ardigen, and Seqera on running omics workflows in the cloud. 


Here were my key takeaways from the talk: 

You can watch the full video here: 

<iframe width="560" height="315" src="https://www.youtube.com/embed/zQyh7LF6_h4?si=4QVkjvxM0E6Kj5Oq" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

## Running omics workflows on the cloud is about understanding what tradeoffs you’re making. 

<iframe width="560" height="315" src="https://www.youtube.com/embed/T_keBXj_iuQ?si=KhA4UWL9n5647YkQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

“For example if you're running a production pipeline where you just need to keep it churning, but you don't need to have the results immediately, you can wait and deal with retries. You might also not necessarily need the latest and greatest hardware. 

However, if you're in an R&D activity, the time of your developers is more expensive than the time of the machines. 

So you need to be able to have your developers progress very rapidly, be able to iterate extremely rapidly, because the cost you're paying the humans who are operating machines is higher, and also the cost of the opportunity cost of needing to define the solution faster than maybe your competitors.

What's more important?” 

Getting the answer fast? Getting to a product fast? Or do you have just something that's running in your background?” 


## When going full cloud, it’s important to understand what kinds of CPUs you need to for your pipeline.

<iframe width="560" height="315" src="https://www.youtube.com/embed/D1o36dL5tGQ?si=gEsVaUN5zSuENptI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

“Depending on the shape and size of the data and the kind of scientific algorithm that you're running, there are going to be different computational requirements.

In some cases you might want a beefy VM with a lot of compute, a lot of memory. Sometimes you just, it's going to be a small operation that's just cheap, but it's more disk bound. 

So there's some very different shapes and sizes of machine that might be appropriate.” 


## Case Study: How the Broad Institute leverages spot instances to reduce costs

<iframe width="560" height="315" src="https://www.youtube.com/embed/-OEZnDIB0fs?si=BJTFvermcyj8DUcf" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

The Broad Institute saved ~90% on costs going from their first implementation to their final pipeline. By leveraging tools like Spot instances, they were able to take their costs from $85 per whole genome to $5. 

“When Broad's original genomic pipeline was migrated from on prem to cloud, the initial,  completely naive implementation was about, I want to say $85 per whole genome. With some basic optimization, they very quickly brought that down to $45 per genome, which is almost half. 

Three to six months of further investment, (today it would take much less time because we have all these tools), we took it down to $5 a genome.

And if you think about the cost reduction there, that was huge. Nowadays you can use those kinds of approaches. You don't have to do six months of R&D because a lot of the tools exist. But you can really make a really big difference from the naive implementation to an optimized implementation.”



## Ready to Learn More? 
Spot instances are a great way to use spare capacity in the cloud at 60 to 80% cheaper. 

If you're curious about seamlessly running your research and other types of computational workloads on spot instances, feel free to [book a chat with me](https://calendly.com/jing-memverge/30-minute-general-discussion?month=2023-12&date=2023-12-05). 


## More about Jing
I'm Jing, I run GTM for our memory snapshot product at MemVerge.

<iframe width="560" height="315" src="https://www.youtube.com/embed/gW2uf5-LEYM?si=-ND4_wi-YSGYxGAb" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

We protect your workloads from being taken away or reclaimed by the cloud provider, even when they're not done running. And we are automatically finding you additional instances where you can continue to run your work to completion.

I love helping teams who may not have the resources to conduct their computationally intensive research locally or may not feel fully comfortable migrating their omics workflows to the cloud. 

I post regularly about running omics workflows on spot instances and in the cloud. Check out my writings on [LinkedIn](https://www.linkedin.com/in/jingxie/).

## What is AppCapsule and what problems does it solve? 
AppCapsule creates a snapshot of all the data in memory, and all the application states required to restore the application at this moment in time, as well as the data on storage. With AppCapsule, we can transport the application runtime across different machines. 

Three problems that AppCapsule can help to solve - 
1. **Spot** reclaim - If there is a spot eviction event, Memory Machine Cloud creates an AppCapsule, and moves it to another spot instance, where the job will continue. That means, you can save cost running jobs on spot instance, while keeping all your job progress and does not need to run from the beginning all over again. A.k.a. *SpotSurfer*.
2. **OOM** (Out Of Memory) - Memory Machine Cloud detects if OOM is about to happen and if so, creates an AppCapsule, and moves it to another instance to avoid the application crash as well as have the job continue running.
3. **Right-sizing** - vertical scaling down to avoid over-provisioning or up to provide more resources to the application as it is needed for shorter wall-time and possibly lower total cost. You can trigger either manually by using float cli command / OpCenter Web GUI or automatically done by Memory Machine Cloud (a.k.a. *WaveRider*).
   
![](https://baywatch-api.memverge.com/media/upload/202312/231220190505523032.png)

## What is AppCapsule++ and what benefits does it bring? 
Three enhancements and benefits that AppCapsule++ brings to the users - 

1. Only saving **delta** - instead of saving a full snapshot, AppCapsule++ only saves the difference (memory change) since the previous snapshot. This way, the time required for the final snapshot is minimal.
2. **Adaptive** trigger - instead of snapshotting in a fixed, pre-determined interval, AppCapsule++ automatically and intelligently checks the memory change and take a snapshot when it reaches a threshold, which is calculated based on the cloud service provider’s pre-emption window. The benefit is no need to guess the fixed interval for periodic snapshot.
3. **Asynchronous offloading** - instead of synchronous snapshotting which will pause the application, AppCapsule++ does asynchronous data offloading, which decreases the time that the application is frozen. The benefit is lower impact to job completion wall-time. 

Below is an overall comparison table of AppCapsule and AppCapsule++
| Comparison | AppCapsule | AppCapsule++ |
| - | - | - |
| **Snapshot content** | Full snapshot | Delta snapshot |
| **Snapshot trigger** | Can be<br/>- Periodic done by MMCloud, or<br/>- Manual done by user’s script | Adaptive | 
| **Configuration** | Pre-determined interval for periodic snapshot | No special configuration | 
| **Application completion wall-time** | Impact higher<br/>(Synchronous) | Impact lower<br/>(Asynchronous) | 
| **Storage overhead** | Relatively lower<br/>(only need to save final snapshot to restore) | Relatively higher<br/>(need to save all delta snapshots to restore) | 
| **Usage** | `--dumpMode full` | `--dumpMode incremental` |


## Benchmark comparison 
Below is the benchmark experiment result running on AWS comparing AppCapsule and AppCapsule++. The experiment is based on 50GB memory change in terms of 60GB, 120GB, 240GB, and 480GB application memory usage, and how much time in seconds did AppCapsule and AppCapsule++ take to take the final snapshot. 

Three observations from this benchmark - 

1. For AppCapsule, as application memory usage grows, the snapshot time also **increases**. The reason is that AppCapsule takes a complete snapshot every time.
2. For AppCapsule++, as application memory usage grows, the snapshot time **stays more or less the same**. The reason is that AppCapsule++ takes a delta snapshot, which is triggered when the amount of change in memory reaches a certain threshold.
3. Considering using spot instance on AWS, which has 2-minute pre-emption time window, AppCapsule can only support relatively small workload (in this experiment for example, 60GB since the snapshot time is 109 seconds). AppCapsule will not be able to support bigger workload (in this experiment for example, 120GB needs 250 seconds to finish a snapshot which is way over the 2-minute time window). However, since AppCapsule++ only takes delta snapshot, the snapshot time is consistent with memory change, not the application memory usage. 
![AppCapsule vs AppCapsule++](https://baywatch-api.memverge.com/media/upload/202312/231220185252866864.png)


More detailed information in table format below: 
| Experiment No. | Experiment Variable | AppCapsule | AppCapsule++ |
| - | - | - | - |
| 1 | Application memory usage: 60G<br/>Application memory change: 50G | 109s<br/>r5.2xlarge/gp3 | 91s<br/>r5.2xlarge/gp3 |
| 2 | Application memory usage: 120G<br/>Application memory change: 50G | 250s<br/>r5.4xlarge/io1 | 102s<br/>r5.4xlarge/io1 |
| 3 | Application memory usage: 240G<br/>Application memory change: 50G | 533s<br/>r5.8xlarge/io1 | 104s<br/>r5.8xlarge/io1 |
| 4 | Application memory usage: 480G<br/>Application memory change: 50G | 1548s<br/>r5.16xlarge/io1 | 80s<br/>r5.16xlarge/io1 |

## Call to action - save your spending and time 
What is the characteristic of your running job and environment? AppCapsule++ is available now with Memory Machine Cloud 2.4 Goa release. Please reach out or leave comments below and we would love to chat and help!

### What is application observability and why it is important

Application observability gains insights into the behavior of software applications utilizing the information from logs, metrics and traces. The goals include but not limited to

- identify bottlenecks and optimize performance
- identify issues and troubleshoot problems quickly
- identify resource requirements and determine the most appropriate corresponding infrastructure

### What is Memory Machine Cloud WaveWatcher and how it can help

The Memory Machine Cloud WaveWatcher service builds on its ability to track CPU, memory, network, storage (size, throughput, and IO) used by a ‘job’ (could be an application, a workflow, or a pipeline). Take the below WaveWatcher result of a Sentieon WGS pipeline as an example:

![WaveWatcher for Sentieon WGS Pipeline](https://baywatch-api.memverge.com/media/upload/202311/231120214738149768.png)

We can see how this pipeline can be characterized into different steps, each with corresponding ‘behaviors.’ The resources the steps consume are shown in the table below:
| Step | Function | Observability - Memory Usage | Observability - CPU Usage | Observability - Network | Observability - Storage Usage | Observability - Storage IO | What can / should we do? |
| - | - | - | - | - | - | - | - |
1 | Validate license and download reference and sample data | extremely low | extremely low | extensive | increasing | extensive | Use a less memory and CPU intensive but network and storage optimized instance |
2 | BWA-MEM | continuously growing | extremely high | none | increasing | medium | Use a memory and CPU optimized instance and larger instance
3 | Dedup | dynamically changing | dynamically changing | zero | dynamically changing | dynamically changing | Use a CPU optimized instance and larger instance |
4 | Variant calling, filtering and genotyping | high | extremely high | zero | retain | zero | Use a memory and CPU optimized instance and medium instance

### Go beyond observability - Memory Machine Cloud WaveRider

It is helpful to gain observability but the ultimate question is - what can be improved, and can it be automated? The data from WaveWatcher can then be used by the WaveRider service to automatically right-size resources, keep the application / pipeline / workflow state, and reduce CO2 footprint, hence to

- avoid under-provision which may result application performance bottleneck or even OOM, and
- avoid over-provision which spends unnecessary money and CO2 footprint.

More detailed Sentieon WGS pipeline benchmark can be found [here](https://www.mmcloud.io/blog/sentieon-genomics-tools-boosting-performance-wit-3).

### Three pillars of Memory Machine Cloud - observability (WaveWatcher), right-sizing (WaveRider), and job automation (Float)

Last but not the least, how to automate the whole process with minimum human intervention? Just like the three pillars of observability (logs, metrics and traces), there are three pillars for Memory Machine Cloud - job automation (Float), observability (WaveWatcher), and right-sizing (WaveRider).

- The user submits a job (for example, a bioinformatics analysis pipeline) via Float CLI or GUI.
- Memory Machine Cloud WaveWatcher automatically observes the behavior including the resources needed
- Memory Machine Cloud WaveRider automatically right-sizes / vertical scaling based on WaveWatcher’s observation supporting stateful job

These three pillars make a complete cloud automation solution removing the hassle of integrations among multiple solutions.

### Comparison of Memory Machine Cloud with modern solutions

Below is a summary table of comparison of Memory Machine Cloud with modern solutions, where Memory Machine Cloud provides an all-in-one solution.
| Category | Comparison | MemVerge MMCloud | Grafana | Datadog | Spot by NetApp | YellowDog | AWS CloudWatch | AWS Compute Optimizer | AWS Batch |
|:----------:|:---------------------------------------------------------------:|:------------------:|:----------:|:----------:|:--------------:|:---------------:|:--------------:|:---------------------:|:----------:|
| Feature | Observability | Yes (WaveWatcher) | Yes | Yes | Yes | Yes | Yes | Yes | No |
| Feature | Automatic right-sizing stateless application | Yes (WaveRider) | No | No | Yes | Yes | No | No | No |
| Feature | Automatic right-sizing stateful, non-fault-tolerant application | Yes (WaveRider) | No | No | No | No | No | No | No |
| Feature | Job automation | Yes (Float) | No | No | No | Yes | No | No | Yes |
| Deployment | On-prem version | Yes | Yes | No | No | Yes | No | No | No |
| Pricing | Pricing | All-in-one license | By license | By service | By license | Basic + add-ons | By service | Opt-in | By service |

\*note - more detailed comparisons with Spot by NetApp can be found [here](https://www.mmcloud.io/blog/memory-machine-cloud-vs-spot-by-netapp)

### Call for action - save your spending and time

Do you need to run stateful, non-fault-tolerant workloads or pipelines on the cloud? Would you like to benefit from automatic right-sizing and vertical scaling and reduce both cloud cost and job running hours without worrying about integrations among multiple solutions?  [Please try Memory Machine Cloud for free](https://www.mmcloud.io/customer/register).

<p><strong class="ql-size-large">Summary</strong></p><p>The <a href="https://www.sentieon.com/" rel="noopener noreferrer" target="_blank">Sentieon®</a>&nbsp;Genomics tools provide a complete rewrite — stressing computational efficiency, accuracy, and consistency — of popular genomics pipelines (such as GATK, Picard, and MuTect) for calling variants from next-generation sequence data. Consistently, Sentieon Genomics tools show a five- to ten-fold improvement in performance over the equivalent open-source pipelines.</p><p>A common characteristic of variant calling pipelines is that the resource demands on CPU cores and memory access vary as the pipeline executes. MemVerge’s&nbsp;<a href="https://www.mmcloud.io/" rel="noopener noreferrer" target="_blank">Memory Machine Cloud</a>&nbsp;includes a checkpoint/restore feature (called WaveRider) that can automatically migrate a running container, without losing execution state, to a virtual machine of different capacity as soon as a change in resource requirements is detected.</p><p>When Sentieon Genomics tools are used in conjunction with WaveRider, performance (measured by wall clock time) improves even further because processes are never starved for resources. The costs associated with pipeline execution also decrease because virtual machines are never overprovisioned. Depending on how EC2 On-demand instance prices scale with the number of vCPUs and memory capacity, it is possible to optimize cost and performance simultaneously by running a larger virtual machine for a shorter period when CPU and/or memory utilization are high, and a smaller virtual machine for a longer period when CPU and/or memory utilization are low.</p><p>The Sentieon-developed whole genome sequencing (WGS) pipeline benchmark simulates a typical bioinformatic analysis pipeline, starting with loading raw data (in the form of fastq files) and ending with output in the form of vcf files. The <a href="https://www.mmcloud.io/blog/sentieon-genomics-tools-boosting-performance-wit-3" rel="noopener noreferrer" target="_blank">complete report</a> compares the performance of the WGS pipeline benchmark executed without using MMCloud to the performance when used with MMCloud.</p><p>The tests show that, compared to a baseline measured by running the WGS pipeline benchmark on a single AWS EC2 instance, the combination of Sentieon Genomics tools and Memory Machine Cloud (with WaveRider) demonstrates a 40% decrease in wall clock time and a 34% reduction in cost.</p>

<h2>Cloud’s promise unfulfilled</h2><p class="ql-align-center"><img src="https://miro.medium.com/v2/resize:fit:1400/1*jCza7dtD08vfltfQw2XhYw.png" height="207" width="700"></p><p>Recent news suggest that the growth of cloud adoption is encountering serious impedance. There are even organizations repatriating compute resources back to on-premises data centers, and cloud cost seems to be the culprit.</p><p>Indeed, a 2 CPU (24 core)/48 GB RAM server sells for ~$10,000. On AWS, a 24 vCPU / 48 GB RAM VM costs ~$1.18/hr. If you continuously ran the VM for 1 year, you would have paid $10,195 and could have bought the server hardware. Since the typical useful life of server hardware is 4–5 years, you would’ve overpaid 4–5 times for the server. Of course in a more rigorous comparison we need to consider operational costs but I think we all agree that continuously running the cloud server the same way as an on-prem server is costly.</p><p>The key word, of course, is “continuously.” To reap the benefit of the cloud you need to use it efficiently, i.e., releasing resources that you are not using, and using resources that match the needs of your applications.</p><h2>Several challenges stand in the way of efficient cloud usage in the enterprise:</h2><h2>App owner access to the cloud</h2><p>The team that manages the cloud resources, i.e., the cloud admin team, is not the team that is running the applications, i.e., the application owners: the developers, the analysts, the scientists, et al. The application users often need to wait hours or days to gain access to cloud resources, and they don’t have the tools or the discipline to release the resources as quickly as possible.</p><h2>App owner flying blind</h2><p>The application owners may not know the most appropriate machine size to use for their applications. Often, they do not have the tools necessary to view the resource utilization of the application. They only know their cloud bill is high, their application seems to take a long time to complete, they suffer out-of-memory errors, and their application crashes and must be restarted.</p><h2>Forced to provision for peak usage</h2><p>Even when the application owner understands the application, it is often difficult to pick one machine size because most applications have peaks and troughs in their resource utilizations. If one picks the machine size that can handle the peak utilization, it often means the machine is under-utilized most of the time. If one sacrifices performance by under provisioning, then the application may crash (OOM) or otherwise run very slowly.</p><h2>The serverless model for complex, stateful apps</h2><p>When we set out to build a product for the cloud, leveraging the fruits of our research in Big Memory Computing, we asked ourselves, “Can we build software that enable all applications, including stateful apps, to be serverless?” That means application owners can focus on their subject areas and never need to worry about management of servers or other cloud resources, nor depend on another team to do so.</p><p>Stateful applications should be able to take advantage of the elasticity of the cloud and consume resources most appropriate for it, even when the resource utilization is bursty. Serverless computing services, such as AWS Lambda, have proven that the serverless model is highly desirable for cloud native microservices. But how do we enable complex applications, even stateful applications, to run serverless?</p><h2>Decoupling complex, stateful applications from servers</h2><p class="ql-align-center"><img src="https://miro.medium.com/v2/resize:fit:1400/0*ETvWyn5e8XFOeJGV.png" height="354" width="700"></p><p class="ql-align-center">Application runtime is “chained” to servers</p><p>Recent news suggest that the growth of cloud adoption is encountering serious impedance. There are even organizations repatriating compute resources back to on-premises data centers, and cloud cost seems to be the culprit.</p><p>Indeed, a 2 CPU (24 core)/48 GB RAM server sells for ~$10,000. On AWS, a 24 vCPU / 48 GB RAM VM costs ~$1.18/hr. If you continuously ran the VM for 1 year, you would have paid $10,195 and could have bought the server hardware. Since the typical useful life of server hardware is 4–5 years, you would’ve overpaid 4–5 times for the server. Of course in a more rigorous comparison we need to consider operational costs but I think we all agree that continuously running the cloud server the same way as an on-prem server is costly.</p><p>The key word, of course, is “continuously.” To reap the benefit of the cloud you need to use it efficiently, i.e., releasing resources that you are not using, and using resources that match the needs of your applications.</p><h2>AppCapsule technology makes it possible</h2><p>We invented a solution for decoupling stateful apps from servers with AppCapsule technology. A stateful application is one whose execution depends on the data stored in memory. Because memory is not persistent, when the server hardware reboots or changes, the data in memory is lost, and the application must start from the beginning again. Our AppCapsule creates a snapshot of all the data in memory, and all the application states required to restore the application to this moment in time, as well as the data on storage. With AppCapsule, we can transport the application runtime across different machines, even different clouds.</p><p class="ql-align-center"><img src="https://miro.medium.com/v2/resize:fit:1400/0*X_gS1OCoelC4ImVA.png" height="303" width="700"></p><p class="ql-align-center">AppCapsule Snapshot</p><p>Armed with this technology, we set out to build control components that natively integrates with the clouds, and the user interface designed to serve the application users, and voilà, we have Memory Machine Cloud, the cloud automation platform that delivers the Serverless experience for most applications.</p><h2>The Floating App</h2><p>The internal engineering codename for Memory Machine Cloud is “Float,” because when you launch your application using Memory Machine, your application is floating on the cloud, untethered to the server it is running on. For that reason, our CLI commands all start with the keyword “float.”</p><p>It’s pretty amazing what you can do when your application is floating. The first use case we tackled was running long, stateful jobs, such as genomic sequencing, on spot instances. Spot instances offer the most economical way to tap into the vast compute resources in the public cloud, but because the CSP can evict your workload on spot with a very short notice period, they are not suitable for these jobs. That is, unless you use Memory Machine Cloud.</p><p>When you launch your genomic sequencing job using Memory Machine, you simply specify the spec of the machine, and that you want to leverage spot instances. Memory Machine takes care of finding the spot instances, deploying your containers, mapping your storage, and launching your job. If there is a spot eviction event, Memory Machine creates an AppCapsule, and moves it to another spot instance, where the job will continue. In fact, the application is unaware of the floating event. This use case alone, we have found, consistently saves our users more than 60% in compute cost.</p><p><strong>Submitting a job with the Float GUI</strong></p><p class="ql-align-center"><img src="https://miro.medium.com/v2/resize:fit:1400/0*yALS-Fy-kHvybR0h.png" height="502" width="700"></p><p class="ql-align-center"><strong>Launching your job in the Cloud takes &lt;1 minute</strong></p><p>Another use case is rightsizing. Often an application’s resource utilization curve has peaks and valleys over time. That is problematic for picking the appropriate machine capacity for the application. However short-lived, the peak memory usage typically becomes the minimum memory capacity for the machine, lest the application will experience an OOM and crash. That also means that during the majority of the execution, the machine is underutilized.</p><p>By making the application runtime float on the cloud VM, Memory Machine Cloud has a great solution for this problem. It constantly monitors the resource utilization of the application, and when it detects under- or over- utilization conditions, it can “float” the application to a more appropriate machine. This becomes especially handy when the user uses a workflow manager, such as NextFlow or Cromwell, to kickoff hundreds and thousands of jobs, and it is virtually impossible to manually pick the most appropriate machine types for each of the jobs.</p><p class="ql-align-center"><strong>Monitoring resource utilization and “floating” the app to a more appropriate VM</strong></p><p class="ql-align-center"><img src="https://miro.medium.com/v2/resize:fit:1400/0*xs9uaSJDYLK_n3m2.png" height="361" width="700"></p><p class="ql-align-center"><strong>WaveWatcher shows how jobs automatically “surf” to the most appropriate instance type</strong></p><h2>The post-server era</h2><p>In the hardware world, we are seeing tremendous progress in the deconstruction of servers, composable CPU, Storage, and most recently, memory. Those innovations will manifest themselves in the public cloud soon. There is a need and an opportunity to build the automation layer that enables application users and IT operators to transition into “post-server” mode TODAY, not only for cloud native applications, but for all applications. That is the mission of Memory Machine Cloud.</p><iframe class="ql-video ql-align-center" frameborder="0" allowfullscreen="true" src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F2juNvP3BbK8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D2juNvP3BbK8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F2juNvP3BbK8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" height="480" width="854"></iframe><p><br></p>

Blog - AppCapsule