Book a Demo
Book a Demo

Nextflow - JFS - Multi-User Head Node

Sateesh Peri

Multiple users on the Nextflow head node

  • It is possible to have multiple users using the same nextflow head node

  • It needs the admin of the OpCenter to deploy the head node and create multiple users

Float Login as OpCenter Admin

  • Make sure your local float version is up to-date
sudo float release sync
  • Login to your MMCloud opcenter as ADMIN
float login -a <opcenter-ip-address> -u <admin>

Enter password & make sure Login succeded!

  • Check the current user list on your MMCloud OpCenter
float user list

Example user list

+----------------+------+------+--------+---------------+-------+---------+---------+---------+
|    USERNAME    | UID  | GID  |  ROLE  |     GROUP     | EMAIL |  TYPE   | ENABLED | MANAGES |
+----------------+------+------+--------+---------------+-------+---------+---------+---------+
| admin          |    0 |    0 | admin  | admin         |       | builtin | true    | admin   |
| system         |    1 |    1 | admin  | system        |       | builtin | true    | system  |
| sateesh        | 1002 | 1002 | admin  | admin;default |       | builtin | true    |         |
| ashleyt        | 1003 | 1003 | normal | default       |       | builtin | true    |         |
  • Note the user name and UID of the users you want to share your nextflow runner node with. Those users are to be created on the nextflow head node with the same user name and the same UID.

(For Example, the user sateesh has a UID of 1002)

  • Next, we need to check if the user needs UidBoundary
float config ls | grep UidBoundary

Example output

| security.inlineUidBoundary        | 0                         >                | Y        |             |
  • As shown above the security.inlineUidBoundary for above user is 0 and the UID of the user can be set as is.

  • However, if your security.inlineUidBoundary in your opcenter’s configuration is not zero, the user’s UID on the nextflow runner node has to be UID + UidBoundary.

Deploy head node from template

  • Check if the nextflow JFS template is available by requesting info about the template
float template info -T nextflow:jfs

Example output

name: nextflow
tag: jfs
type: JobTemplate
lastUpdated: 2023-10-12T06:11:25Z
jobParams:
   - image: nextflow
     tag: jfs
     cpu: "2"
     mem: "4"
     dataVolume:
       - '[size=120]:/mnt/jfs_cache'
     publish:
       - 6868:6868
     vmPolicy: '[onDemand=true]'
     migratePolicy: enable=false
     containerInit: https://mmce-data.s3.amazonaws.com>/scripts/prepare_nextflow.sh

S3 bucket as JFS backend storage

  • Create a new S3 bucket for nextflow head node and note its region (you will need to specify this later)

Float Secret

  • Set your AWS access credentials as float secret
float secret set BUCKET_ACCESS_KEY <BUCKET_ACCESS_KEY>
float secret set BUCKET_SECRET_KEY <BUCKET_SECRET_KEY>
  • To list the secrets that have been successfully set
float secret list

Example output

+-------------------+
|          NAME     |
+-------------------+
| BUCKET_ACCESS_KEY |
| BUCKET_SECRET_KEY |
+-------------------+

Deploy Nextflow head node

  • submit nextflow head-node job by providing <head-node-name>, <security-group>, <bucket-name> & <bucket-region>.
float submit -n <head-node-name> \
--template nextflow:jfs \
--securityGroup <sg-00000000000a> \
-e BUCKET=https://<bucket-name>.s3.<bucket-region>.amazonaws.com \
--env BUCKET_ACCESS_KEY={secret:BUCKET_ACCESS_KEY} \
--env BUCKET_SECRET_KEY={secret:BUCKET_SECRET_KEY}
  • Check the status of deployment
float list

Example

+-----------------------+----------------+------------------------------------+---------+-----------+----------------------+------------+-------------+
|          ID           |      NAME      |            WORKING HOST            |  USER   |  STATUS   |     SUBMIT TIME      |  DURATION  |    COST     |
+-----------------------+----------------+------------------------------------+---------+-----------+----------------------+------------+-------------+
| NlygkM1dA0qIucncPwjgD | jfs-head-1     | 54.211.194.151 (2Core4GB/OnDemand) | sateesh   | Executing | 2023-11-02T17:40:06Z | 6m54s      | 0.0049 USD  |
+-----------------------+----------------+------------------------------------+---------+-----------+----------------------+------------+-------------+
  • Note the IP address of the head node in order to ssh into it.

  • Get the SSH key for the nextflow head node

float secret get <head-node-job-id>_SSHKEY > jfs_admin.key
  • Change the permissions of the ssh key file
chmod 600 jfs_admin.key
  • Next, as admin user root ssh login to the head node
ssh -i jfs_admin.key root@<head-node-ip-address>
  • Next create a user and $HOME directory for the user specifying the UID of the user from before (note: can use default GID value of -g 1000)
/usr/sbin/useradd -m -d /mnt/jfs/nextflow/<user> <user> -u <UID> -g 1000

Example

/usr/sbin/useradd -m -d /mnt/jfs/nextflow/sateesh sateesh -u 1002 -g 1000
  • Set password for the user
passwd <user>

You will be prompted to enter a new password for the user

Changing password for user
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
  • To enable the password authentication, we need to modify sshd_config
vi /etc/ssh/sshd_config
  • UNCOMMENT the line PasswordAuthentication yes and then save the file

Example

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication yes
#PermitEmptyPasswords no
  • Next, Restart SSH service on the nextflow runner node
systemctl restart sshd
  • Validate if the user can ssh to the nextflow runner node
ssh <user>@localhost

Example

user@localhost's password:
Last login: Thu Oct 26 04:10:40 2023 from 172.59.128.0
  • The $HOME directory will be set /mnt/jfs/nextflow/<user>. verify it with pwd
pwd

Example

/mnt/jfs/nextflow/user
  • Copy the mmcloud.config.template file to the user’s home directory
cp /mnt/jfs/nextflow/mmcloud.config.template .
  • To exit from the localhost login
exit
  • Finally, provide the user with their password and head node IP address for them to be able to login

In case of any issues, reach out to MMCloud support team on MemVerge-Support Slack or support@memverge.com


User login to nextflow head node

  • User's for whom an account has been created by the admin can login to nextflow head node as shown below with the password provided to them
ssh <user>@<head-node-ip-address>
  • User's can also change their password once after they login to the head node as follows
passwd <user>

Float config

  • Create a copy of the cp mmcloud.config.template mmc-jfs.config and open the config using vi text editor for further changes. (NOTE: if you are new to vi see this link for basic usage know-how)
vi mmc-jfs.config
  • modify the mmc-jfs.config to add OpCenter IP address, credentials & PRIVATE IP address of the nextflow head node <head-node-private-ip>
plugins {
  id 'nf-float'
}

workDir = '/mnt/jfs/nextflow/<user>'

process {
    executor      = 'float'
    errorStrategy = 'retry'
}

float {
    address = '<opcenter-ip-address>'
    username = '<user>'
    password = '<password>'
    commonExtra ='  --dataVolume [opts=" --cache-dir /mnt/jfs_cache "]jfs://<head-node-private-ip>:6868/1:/mnt/jfs --dataVolume [size=120]:/mnt/jfs_cache'
}


aws {
  client {
    maxConnections = 20
    connectionTimeout = 300000
  }
  accessKey = '<bucket_access_key>'
  secretKey = '<bucket_secret_key>'
}
  • Start a tmux session with name nextflow. If you are new to using tmux, here is a useful link
tmux new -s nextflow
  • To attach to an already existing tmux session
tmux attach -t nextflow
  • Check the nextflow version to see if it is available
nextflow -v
nextflow version 23.10.0.5889
  • Export Tower token (optional)
export TOWER_ACCESS_TOKEN=<token>

Launch Nextflow

  • Launch nf-core/<pipeline>
nextflow run nf-core/<pipeline> \
-profile test_full \
-c mmc-jfs.config \
--outdir s3://nextflow-work-dir-public/<pipeline> \
-with-tower