Book a Demo
Book a Demo
Observability, Right-sizing, and Automation - Three Pillars of Memory Machine Cloud

Observability, Right-sizing, and Automation - Three Pillars of Memory Machine Cloud

YT Liang 2023-11-20349

What is application observability and why it is important

Application observability gains insights into the behavior of software applications utilizing the information from logs, metrics and traces. The goals include but not limited to

  • identify bottlenecks and optimize performance
  • identify issues and troubleshoot problems quickly
  • identify resource requirements and determine the most appropriate corresponding infrastructure

What is Memory Machine Cloud WaveWatcher and how it can help

The Memory Machine Cloud WaveWatcher service builds on its ability to track CPU, memory, network, storage (size, throughput, and IO) used by a ‘job’ (could be an application, a workflow, or a pipeline). Take the below WaveWatcher result of a Sentieon WGS pipeline as an example:

WaveWatcher for Sentieon WGS Pipeline

We can see how this pipeline can be characterized into different steps, each with corresponding ‘behaviors.’ The resources the steps consume are shown in the table below:

Step Function Observability - Memory Usage Observability - CPU Usage Observability - Network Observability - Storage Usage Observability - Storage IO What can / should we do?
1 Validate license and download reference and sample data extremely low extremely low extensive increasing extensive Use a less memory and CPU intensive but network and storage optimized instance
2 BWA-MEM continuously growing extremely high none increasing medium Use a memory and CPU optimized instance and larger instance
3 Dedup dynamically changing dynamically changing zero dynamically changing dynamically changing Use a CPU optimized instance and larger instance
4 Variant calling, filtering and genotyping high extremely high zero retain zero Use a memory and CPU optimized instance and medium instance

Go beyond observability - Memory Machine Cloud WaveRider

It is helpful to gain observability but the ultimate question is - what can be improved, and can it be automated? The data from WaveWatcher can then be used by the WaveRider service to automatically right-size resources, keep the application / pipeline / workflow state, and reduce CO2 footprint, hence to

  • avoid under-provision which may result application performance bottleneck or even OOM, and
  • avoid over-provision which spends unnecessary money and CO2 footprint.

More detailed Sentieon WGS pipeline benchmark can be found here.

Three pillars of Memory Machine Cloud - observability (WaveWatcher), right-sizing (WaveRider), and job automation (Float)

Last but not the least, how to automate the whole process with minimum human intervention? Just like the three pillars of observability (logs, metrics and traces), there are three pillars for Memory Machine Cloud - job automation (Float), observability (WaveWatcher), and right-sizing (WaveRider).

  • The user submits a job (for example, a bioinformatics analysis pipeline) via Float CLI or GUI.
  • Memory Machine Cloud WaveWatcher automatically observes the behavior including the resources needed
  • Memory Machine Cloud WaveRider automatically right-sizes / vertical scaling based on WaveWatcher’s observation supporting stateful job

These three pillars make a complete cloud automation solution removing the hassle of integrations among multiple solutions.

Comparison of Memory Machine Cloud with modern solutions

Below is a summary table of comparison of Memory Machine Cloud with modern solutions, where Memory Machine Cloud provides an all-in-one solution.

Category Comparison MemVerge MMCloud Grafana Datadog Spot by NetApp YellowDog AWS CloudWatch AWS Compute Optimizer AWS Batch
Feature Observability Yes (WaveWatcher) Yes Yes Yes Yes Yes Yes No
Feature Automatic right-sizing stateless application Yes (WaveRider) No No Yes Yes No No No
Feature Automatic right-sizing stateful, non-fault-tolerant application Yes (WaveRider) No No No No No No No
Feature Job automation Yes (Float) No No No Yes No No Yes
Deployment On-prem version Yes Yes No No Yes No No No
Pricing Pricing All-in-one license By license By service By license Basic + add-ons By service Opt-in By service

*note - more detailed comparisons with Spot by NetApp can be found here

Call for action - save your spending and time

Do you need to run stateful, non-fault-tolerant workloads or pipelines on the cloud? Would you like to benefit from automatic right-sizing and vertical scaling and reduce both cloud cost and job running hours without worrying about integrations among multiple solutions? Please try Memory Machine Cloud for free.