Topics on HPC

Connection setup

Remote HPC connection setup

For users who do not wish to run Providentia locally, there is the option to run it on a remote HPC.

For BSC users, Providentia has been designed to be run optimally on the MareNostrum5 and Nord4 HPC machines. It is recommended that users run Providentia on MareNostrum5 as it is guaranteed that there will be no lack of computational resources.

As users connect to these remote machines by ssh, the graphical refresh speed of Providentia is limited by x11 forwarding, thus in order to exploit the full potential of the Providentia, the connections to these machines should be setup in an optimal fashion.

Note that not all remote BSC HPC machines have access to the same data locations. In case you want to load observations that are located on the /esarchive, then you should run Providentia on Nord4, as MareNostrum5 does not have access to the /esarchive.

.ssh/config

To optimise the ssh connection to the remote HPC machine, you will modify the .ssh/config file in the $HOME directory of the local machine used to connect to the remote machines following the examples for the connection to the MareNostrum5 and Nord4 BSC machines:

Host mn5
    HostName glogin4.bsc.es
    User bscXXXXXX
    IdentityFile ~/.ssh/id_rsa
    ForwardX11 yes
    ForwardX11Trusted yes
    Compression yes
    Ciphers aes128-gcm@openssh.com
    ForwardX11Timeout 7d

Host nord4
    HostName n4login0.bsc.es
    User bscXXXXXX
    IdentityFile ~/.ssh/id_rsa
    ForwardX11 yes
    ForwardX11Trusted yes
    Compression yes
    ForwardX11Timeout 7d
    Ciphers aes128-gcm@openssh.com

ForwardX11 yes ensures x11 forwarding is active.
ForwardX11Trusted yes ensures X11 forwarding is trusted.
Compression yes compresses the connection (i.e. speeds up).
Ciphers aes128-gcm@openssh.com sets a speed-optimised cipher for the connection to ensure less time is spent on encryption/decryption.
ForwardX11Timeout 7d ensures the x11 forwarding does not time out (it times out by default after 15 minutes on some machines).

.bashrc for BSC HPC machines

For BSC users, you should also have these lines in your .bashrc file in the $HOME directory of the remote HPC machine:

if [ "$BSC_MACHINE" == "mn5" ]; then
    module use /gpfs/projects/bsc32/software/rhel/9.2/modules/all
elif [ $BSC_MACHINE == "nord4" ]; then
   module load nord3-singu
   module load bsc/current
   module use /gpfs/projects/bsc32/software/suselinux/11/modules/all
   unset PYTHONSTARTUP
fi

External connections (VPN)

If you are connecting to a HPC machine externally (i.e. via VPN), then the ssh connection configuration on the local machine to the gateway machine you are connecting to should be setup in the exact same way.

Running the tool on debug

If you are on Nord4 or Marenostrum 5, this will request an interactive session:

./bin/providentia

After some seconds you will have entered onto an allocated node and the dashboard will be launched. If you want to launch it multiple times and avoid waiting in the queue you can use the debug mode. To do this, you will need to run the tool using the --debug flag and then rerun it:

./bin/providentia --debug
./bin/providentia

Mounting HPC filesystems

It is possible to mount filesystems, such as esarchive from MN5, on your personal computer. This can then be used to load observations/models that are stored remotely on these filesystems while using Providentia on your local machine. The Providentia data paths should be updated to point to the local mounted directory in settings/data_paths.yaml.

How this is done varies by operating system. In this section you will find information on how to mount esarchive, you can use this as inspiration for other filesystems.

Mounting esarchive

Linux

On the Linux the esarchive can be setup permanently.

Firstly, create a local directory where you want the esarchive to be mounted, e.g. /localdir/tomount/esarchive.

Next, add the following line to the /etc/fstab file (changing your BSC username and local directory):

sshfs#bscXXXXXX@dt01.bsc.es:/gpfs/archive/bsc32/esarchive /localdir/tomount/esarchive fuse defaults,allow_other,reconnect,ro 0 0

Install sshfs if it is not in your computer yet:

sudo apt-get install sshfs

The esarchive can finally be mounted by:

sudo mount -av

To unmount the esarchive, the following command can be used:

sudo umount /localdir/tomount/esarchive

Mac

On Mac, the mounting cannot be setup permanently, but will stay open while the terminal session is active. Before trying to setup the connection you must install FUSE and SSHFS, please follow the instructions here: https://earth.bsc.es/wiki/doku.php?id=computing:addesarchive&s[]=sshfs.

After that is done, create a local directory where you want the esarchive to be mounted, e.g. /localdir/tomount/esarchive.

Then run the following command (changing your BSC username and local directory):

sshfs bscXXXXXX@dt01.bsc.es:/gpfs/archive/bsc32/esarchive /localdir/tomount/esarchive -o volname=esarchive,defer_permissions,allow_other,IdentityFile=$HOME/.ssh/id_rsa,reconnect,ro

To unmount the esarchive, the following command can be used:

diskutil umount force /localdir/tomount/esarchive

Windows

As present, the esarchive cannot be mounted on Windows machines.