Memory Machine Cloud 2.2 (El Nido) Release Info

General

Licensing Model

In earlier releases, activating the OpCenter license enabled all features and the same charging mechanism applied to all users. In the El Nido release, there is more granularity: subsets of features can be enabled and different charging mechanisms can be applied.

Subscribers choose from the following license plans.

• MMCloud Essentials: a free, perpetual license that allows licensees to launch jobs using OpCenter but does not enable advanced features such as SpotSurfer and WaveRider. WaveWatcher is enabled.
• MMCloud Pro: a license that enables all features. Accrued charges are usage-based.
• MMCloud Enterprise: a license that enables all features. Charges are not usage-based — charges are based on a contract between MemVerge and the account holder.

License Activation

In earlier releases, users deploying OpCenter in AWS activate the license via the AWS License Manager. In the El Nido release, users (of AWS or any other cloud service provider) activate the license directly from the OpCenter web interface. Prior registration (free) on mmcloud.io is required.

Live Demo Environment

After registering at mmcloud.io, users can try out a live MMCloud environment (called simsurf) with limited functionality. There is no charge and no cloud account is needed.

WaveWatcher Enhancements

WaveWatcher is the feature that allows users to monitor per-job resource utilization in real time. Results are displayed graphically in the OpCenter. Functionality and usability have been enhanced as follows.

• Additional metrics including I/O associated with memory, network, and disk access
• Improved web interface for viewing WaveWatcher metrics, for example, showing times when job migration occurs
• Summary table of usage and performance metrics

Gateway Web Widget

The gateway is a reverse proxy that maintains client-side browser sessions when servers migrate to new VM instances (and new IP addresses). When a browser connects to the gateway, to log in to a Jupyter Lab session, for example, the gateway places a web widget at the top, right-hand side of the window (the user can move the widget). The widget provides an interface to the OpCenter Job Details screen. After logging in, the user can view status and perform actions such as migrate or cancel (for this job only). When the application server (RStudio, for example) is in the "Floating" state, the widget alerts the user that the screen is unresponsive until the state returns to "Executing."

Allow List for VM Instances

A job submitted to the OpCenter specifies the compute environment by name (for example, c5.4xlarge) or capacity (number of virtual CPUs and memory in GB). In the latter case, the requirements may be satisfied by instances powered by several different processor vendors and families. To limit the choices, for example, to stay within a processor family, it is now possible to specify an allow list of instances (specified by name).

Nextflow Plugin Index

To use MMCloud as a Nextflow executor, a plugin called nf-float is required. The nf-float plugin is now included in the Nextflow Plugin Index so that Nextflow automatically downloads the latest version of nf-float whenever a Nextflow configuration file specifies MMCloud as an executor.

Service Templates

Jobs submitted to OpCenter are either batch jobs or interactive sessions. Interactive sessions involve services such as provided by RStudio or Jupyter servers. The float command line to start a service can be complicated. To simplify the process, pre-configured service templates can be used (enter float template -h to see options).

Functional Improvements

Scalability

The OpCenter can now support 1,000 simultaneous jobs.

WaveRider Enhancements

• When autostep is enabled and an upper (or lower) threshold is crossed, a range in the number of vCPUs and a range in memory size are calculated as requirements for the next VM instance. The El Nido release improves the autostep algorithm by separating the calculation for the increase (decrease) in number of vCPUs from the calculation for the increase (decrease) in memory capacity.
• Migration policy can be set so that migration will not occur to a smaller VM instance if a lower limit on the number of vCPUs or a lower limit on memory capacity is reached.

Gateway Enhancements

• Option to allow the OpCenter to automatically select a gateway instance when connecting a job to a gateway.
• Option to create a gateway without a public IP address.
• Ability to connect the gateway to a single server that is listening on multiple TCP ports simultaneously (use the "auto" setting or use the targetPort option multiple times in a single command).

Flexible Job Submit Procedures

• Options associated with a float submit command can be provided in a job definition file. Content usually provided in a job script file (shell commands, for example) can now be included in the job definition file (indicate the script content by using the jobContent: tag in json format).
• Jobs can be submitted without prior loading of the container image into the App Library. As long as the container URI is specified, the container image is automatically pulled.

Additional filter for job reports

To include in a report only those jobs where the job has migrated to a new VM instance, use the --filter "instances>1" filter.

Detailed Job Cost Reporting

Cloud costs associated with a job include compute and storage. Storage costs are incurred by each volume instantiated, for example, the root volume or any data volumes. OpCenter now reports costs for compute and for each type of storage. Users can now compare the incremental costs incurred by MMCloud so that they can determine how much they save by avoiding On-demand Instances.

Using S3 Buckets as Data Volumes

In earlier releases, the user must specify the endpoint when using an S3 bucket (in rw mode) that is in a different region from the OpCenter, for example, --dataVolume [mode=rw,endpoint=s3.us-east-2.amazonaws.com]s3://bucketname:/results. In El Nido, the endpoint is no longer required and multiple S3 buckets can be specified in a single float submit command.

User Experience Improvements

From the OpCenter landing page, a user can launch the MMCloud web interface. New features in the web interface include:

• Ability to view and manage license state, for example, upgrade from MMCloud Essentials to MMCloud Pro.
• WaveWatcher tab in the Job Details screen to display detailed time line of resource utilization including migration events.
• Option to upload job script from an S3 bucket, an http(s) URL, or a local computer.

Bug fixes

This release fixes the following.

• Misconfiguration of swap space on a worker node
• Missing metrics report file for certain jobs
• Missing cost information for UK and France regions
• Failure to initialize host when disk id contains special characters
• Failure to restart OpCenter after VM reboot

For more information, visit our documentation center - New Features in MMCloud El Nido 2.2.1 Release