Memory Machine Cloud's Nextflow plugin "nf-float" enables users to easily automate and provision cloud computing resources to supercharge pipeline execution, safely and reliably run on low-cost Spot instances, and optimize batch job execution. Key features like SpotSurfer for job recovery, WaveRider for real-time optimization, and WaveWatcher for resource insight make bioinformatics in the cloud a breeze.
Memory Machine Cloud is a feature-rich cloud batch executor, easily deployed with a few lines in the Nextflow configuration file. The benefits include cost savings, improved time-to-results, and deep insight into resource utilization at the container level.
When Memory Machine launches a worker node, it opens a communication channel to a resource monitor inside the container. Real-time metrics are available as a CSV file or can be viewed graphically. The WaveWatcher Observability Service displays detailed cloud costs for each job in real-time, and provides reports on cloud cost for each app, each user, and each group on the GUI. It also provides fine-grained data on CPU / Memory / Network / Storage I/O utilizations, as well as the carbon footprint. Use this data for optimizing cloud resources for your workloads.
To pinpoint opportunities for optimization, WaveWatcher displays real time app usage of CPU, memory, network, and storage.
The screen on the left shows the WaveRider could lower costs by starting the workload on a smaller instance, them moving to a larger instance only when more resources are needed.
Transparent to NextFlow, WaveRider provides the ability to migrate running jobs to optimally-sized virtual machines based on their real-time resource utilization. The WaveRider service enables users to optimize cost and wall-clock time with a few mouse clicks.
The grey areas show memory and CPU usage on different compute instances as WaveRider continuously right sizes based on the need for resources during runtime.
Spot instances are a way for cloud service providers to monetize idle compute capacity. A Spot instance may be offered at a discount of up to 90% off the price of an On-demand instance. There is a catch – a Spot instance may be reclaimed by the cloud service providers with a typical warning of only two minutes. Memory Machine Cloud includes the SpotSurfer checkpointing and restore service that allows a running job to move seamlessly from a Spot instance flagged for reclamation to a new Spot or on-demand instance. This feature allows users to reduce cloud costs significantly without increasing wall clock time.
The grey areas show memory and CPU usage during runtime. The white areas represent the time period when the workload is automatically migrated to a new compute instance and restarted.