MemVerge nf-float plugin is now included in the Nextflow plugins index
MemVerge nf-float plugin is now included in the Nextflow plugins index
MemVerge is pleased to announce our Nextflow plugin is now included in the Nextflow plugins index repository on GitHub. This means that any process in a Nextflow pipeline can call MemVerge Memory Machine Cloud as an executor. Nextflow will automatically search the Plugins index and download the latest version of nf-float.
Nextflow currently supports a broad range of executors. Processes can be scheduled to run on the local host, on an HPC cluster, on Kubernetes, or on a cloud platform such as AWS or GCP. Memory Machine Cloud adds several features that provide significant value to Nextflow users in the cloud.
nf-float unlocks the added value of Memory Machine Cloud
Memory Machine Cloud is software that is deployed on virtual machines in any cloud including AWS, GCP and AliCloud today, with more coming soon.
OpCenter
The heart of Memory Machine Cloud is the OpCenter that schedules jobs and instantiates the appropriate resources to run containerized applications. The OpCenter continually monitors the resources consumed by jobs
WaveRider Continuous Optimization Service
Based on user-supplied policy, moves running jobs to optimally-sized virtual machines, all without losing state or interim progress. This WaveRider service enables users to tune cost and wall clock time during job runtime.
SpotSurfer Checkpointing and Recovery Service
Spot instances are a way for cloud service providers to monetize idle compute capacity. A Spot instance may be offered at a discount of up to 90% off the price of an On-demand instance. There is a catch — a Spot instance may be reclaimed by the cloud service providers with a typical warning of only two minutes. Memory Machine Cloud includes the SpotSurfer checkpoint and restore service that allows a running job to move seamlessly from a Spot instance flagged for reclamation to a new Spot or on-demand instance. This feature, called SpotSurfer, allows users to reduce cloud costs significantly without increasing wall clock time.
See how TGen used Nextflow & SpotSurfer
WaveWatcher
When the OpCenter launches a worker node, the OpCenter opens a communication channel to a resource monitor inside the container. Real-time metrics are available as a CSV file or can be viewed graphically using the OpCenter web interface. This feature, called WaveWatcher, provides Nextflow users with a detailed view of how executors are consuming compute and storage resources, an insight not available otherwise.
What does this mean for Nextflow users? Memory Machine Cloud is a feature-rich executor, easily deployed with a few lines in the Nextflow configuration file. The benefits include cost savings, improved time-to-results, and deep insight into resource utilization at the container level.
Read Using Nextflow with Memory Machine Cloud for detailed instructions on how to deploy Memory Machine Cloud with its Nextflow plugin.