Logging in to HPCI using Ubuntu on WSL

0. Change Log

0.1 Major changes

2024.09.09

  • Updated Docker image to gsi-openssh-20240902.tar.xz.

2024.04.05

  • The wording of the login procedure has been partially changed to match the user’s environment.

2024.03.29

  • Updated Docker image to gsi-openssh-20240209.tar.xz.

2023.12.28

  • Updated Docker image to gsi-openssh-20231128.tar.xz.

2023.07.31

  • The wording and structure of some items have been arranged.

  • The text version of this document has been discontinued.

2023.06.16

  • Updated Docker image to gsi-openssh-20230322.tar.xz.

2023.04.28

  • Corrected the URL of the referenced document.

2022.12.19

  • Changed title

2022.07.15

  • Created an English version of the document.

  • Updated Docker image to gsi-openssh-20220325.tar.xz.

2022.01.31

  • Initial release

1. Overview

This document is for those using Windows 10 and explains how to log in to the HPCI environment using Ubuntu on WSL.

  • This document explains how to run Docker in Ubuntu, on top of WSL 2 — this method does not use Docker Desktop for Windows.

1.1 Terminology

Image

The foundation for a container.

Container

An entity that executes an image.

Repository

A collection of Docker images.

Tag

An identifying label that is used to distinguish images in a repository.

1.2 Operating environment

Windows machines must satisfy the following requirements.

System requirements for installing WSL.

  • Windows 10 64bit version 2004 (build 19041) or higher

1.3 Installing WSL

Install WSL on a Windows machine, and install Ubuntu as the Linux distribution.

  • For more information on the installation procedure, see Microsoft’s documentation Install Linux on Windows with WSL .

  • You do not need to perform this step on environments where you are already using Docker Desktop for Windows with the WSL 2 backend.

Installation procedure

  1. Launch PowerShell as administrator.

  • Display the [Quicklink] menu.

    • Right-click the [ Start ] button, or press the Windows logo key + X .

  • Select [Windows PowerShell (Admin)].

  • When asked to confirm, “Do you want to allow this app to make canges to your device?” click [ Yes ].

  1. Execute the WSL command with the –install option.

  • Type the following into PowerShell.

wsl --install

  • This will install WSL and the default Ubuntu distribution of Linux.

PS C:\WINDOWS\system32> wsl --install
Installing: Virtual Machine Platform
Virtual Machine Platform has been installed.
Installing: Windows Subsystem for Linux
Windows Subsystem for Linux has been installed.
Downloading: WSL Kernel
Installing: WSL Kernel
WSL Kernel has been installed.
Downloading: Ubuntu
The requested operation is successful. Changes will not be effective until the system is rebooted.
PS C:\WINDOWS\system32>

  1. Follow the messages. Once WSL installation is complete, restart the system.

  2. Set up a default Unix user account.

Directly after logging in to Windows, you will need to create a default Unix user account for running Ubuntu.

For more information on default UNIX user accounts, see Microsoft’s documentation Set up your Linux username and password .

Ubuntu will automatically start after restarting Windows. If Ubuntu does not start, try the following to start it:

  • Click the [ Start ] button and choose [ Ubuntu ] from the list of application

This completes the process of installing WSL and Ubuntu.

Updating and upgrading packages

We recommend that you use the package manager to periodically update and upgrade packages. Do this by typing in the following command.

$ sudo apt update && sudo apt upgrade

2. Install Docker Engine

Attention

Those who wish to stop using Docker Desktop for Windows as Docker Desktop has transitioned to a paid model can avoid problems associated with accidentally issuing commands for Docker Desktop by uninstalling Docker Desktop for Windows before installing Docker Engine. Follow 2.3 Uninstallation procedure in Logging in to HPCI using Docker (Windows 10)

Install Docker Engine in Ubuntu on top of WSL.

For more information on the installation procedure, see the Docker document Install Docker Engine on Ubuntu . Note that it was not written in the expectation that you will be running Ubuntu on top of WSL 2, so it may be at variance with parts of the below procedures.

2.1 Installation procedure

Launch Ubuntu

Launch Ubuntu following the instructions in 1.3 Installing WSL .

  • Click the [ Start ] button and choose [ Ubuntu ] from the list of application

Uninstall previous versions

If any existing Docker packages (docker, docker.io, docker-engine) are installed, uninstall them as follows.

$ sudo apt-get remove docker docker-engine docker.io containerd runc

Set up a Docker repository

Record the Docker repository information Docker distributes in apt.

  1. In order to use apt over HTTPS to access the repository and install the package, install the following packages.

$ sudo apt-get update
$ sudo apt-get install ca-certificates curl gnupg lsb-release

  1. Add Docker’s public GPG key.

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

  1. Set up a stable repository with the following command.

$ echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

参考

You can download and run the following script, which will execute Steps 2 and 3 above for you.

Install Docker Engine

  1. Install the current versions of Docker and containerd.

$ sudo apt-get update
$ sudo apt-get install docker-ce docker-ce-cli containerd.io

  1. Confirm that you are able to run the docker command.

$ sudo docker -v
Docker version 20.10.17, build 100c701
$

  1. Launch the Docker daemon. (The Docker daemon will not launch automatically in Ubuntu running on WSL, so you need to do this manually.)

$ sudo service docker start
 * Starting Docker: docker                                  [ OK ]
$

  1. To confirm that Docker Engine is running properly, run the hello-world image.

$ sudo docker run hello-world

You should see the following sample output.

$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete
Digest: sha256:975f4b14f326b05db86e16de00144f9c12257553bba9484fed41f9b6f2257800
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

$

2.2 Post-installation setup

Permit the docker command to be run without special privileges

Normally, running the docker command requires sudo. To make it so that it can be run without special privileges, you can add the user to the “docker” group.

For details on this, see the Docker document Manage Docker as a non-root user .

  1. Add the user to the “docker” group

$ sudo usermod -aG docker $USER

  1. In order for Ubuntu to recognize the new group membership, exit Ubuntu using the exit command, then restart Ubuntu immediately. [Added for clarity]

  2. Confirm that you can run the docker command without sudo.

$ docker run hello-world

Set up automatic launching

Following the above Install Docker Engine procedure, the Docker daemon will not automatically run in Ubuntu on WSL: you will need to launch it manually after starting Ubuntu.

The following procedure will make it so that the Docker daemon starts automatically when you launch Ubuntu.

  1. Run visudo. This will launch the default nano editor.

$ sudo visudo

  1. Enter the following.

# Allow members of group docker to
%docker ALL=(ALL) NOPASSWD: /usr/sbin/service docker start, /usr/sbin/service docker stop, /usr/sbin/service docker restart

  1. Save your changes and quit the editor.

  2. Add a script that launches docker services to $HOME/.bashrc.

# docker daemon
service docker status > /dev/null 2>&1
if [ $? = 1 ]; then
    sudo service docker start
fi

The next time you launch Ubuntu, the Docker daemon will start automatically.

Creating a WSL 2-mounted directory

Create a directory to WSL 2-mount before running the Docker image for gs-openssh. In this example, we are creating a directory named “work” under the default user’s home directory.

  1. Create a directory named “work” in the home directory.

$ cd
$ mkdir work

  1. Change the owner and group of the “work” directory to the uid and gid of the container user “hpciuser”.

$ sudo chown 2000:2000 work

Information on the uid and gid of the container user “hpciuser” is in the Bash environment .

3. Log in to the HPCI environment

This section describes the procedure for logging in to the HPCI environment using Docker Engine from Ubuntu running on WSL of Windows 10.

The operation flow is as follows.

  1. Launch Ubuntu

  2. Launch the Docker daemon

  3. Load the Docker image (only when installing)

  4. Launch the Docker container

  5. Run bash

  6. Log in to the login server

  7. Shut down the Docker container

3.1 Launching Ubuntu

  • Click the [ Start ] button and choose [ Ubuntu ] from the list of application

3.2 Launching the Docker daemon

If you did not set up the Docker daemon to run automatically following the procedure Set up automatic launching , you will need to launch it manually.

$ sudo service docker start
 * Starting Docker: docker                                  [ OK ]
$

3.3 Loading the Docker image (only when installing)

You will need to store the Docker image in the local-storage domain on the Docker host. [Is this addition OK?]

Download

  1. Download the latest Docker image from this URL:

    https://www.hpci.nii.ac.jp/gt6/docker/

The docker image will have the file name gsi-openssh-20240209.tar.xz – substitute the name of the downloaded file.

Load the Docker image

  1. Execute the docker load command to load the Docker image.

username@WINDOWSPC:~$ docker load -i gsi-openssh-20231128.tar.xz
51af9e44b4ea: Loading layer    196MB/196MB
88a6e38e08cb: Loading layer  285.3MB/285.3MB
730b7f317e6e: Loading layer   7.68kB/7.68kB
30e384109cb7: Loading layer  4.096kB/4.096kB
58d49e02e5bd: Loading layer  607.7kB/607.7kB
5f70bf18a086: Loading layer  1.024kB/1.024kB
Loaded image: hpci/gsi-openssh:20231128
username@WINDOWSPC:~$

  1. To confirm that it is stored in local storage, execute the docker images command. Check that the REPOSITORY is hpci/gsi-openssh, and that the value for TAG matches the date in the docker image file name.

username@WINDOWSPC:~$ docker images hpci/gsi-openssh
REPOSITORY         TAG        IMAGE ID       CREATED       SIZE
hpci/gsi-openssh   20231128   12a54c4e3ec3   4 weeks ago   466MB
username@WINDOWSPC:~$

3.4 Launching the Docker container

  1. Execute the docker run command to run the Docker image you loaded in Step 3.3.

username@WINDOWSPC:~$ docker run -d --rm --name gsi-openssh -v ~/work:/home/hpciuser/work hpci/gsi-openssh:20231128
10c2073ab4d311aa51cdb114545540f27aa6668d39d002efd5798985ebc0857a
username@WINDOWSPC:~$

The long string is the container ID (UUID). When running or stopping this container, you will use this ID (or its short form).

The options used in this command are:

-d

Run the container in the background

--rm

Automatically delete the container after shutdown.

--name NAME

Sets the name of the container. In the above example, it is named gsi-openssh. To identify this container when running it in the future, you can use the container name gsi-openssh instead of the container ID.

-v LIST

Mounts the host OS directory HOST-DIR in the container directory CONTAINER-DIR. Here, LIST is expressed as HOST-DIR:CONTAINER-DIR. In the above example, the work directory that was created to be bind-mounted using “Creating a WSL 2-mounted directory” (~/work) is mapped to the container directory /home/hpciuser/work.

Without the -d option, Docker will not return you to the Ubuntu prompt. If this happens, you will need to open Ubuntu again and execute the command docker stop gsi-openssh, which will shut down the container and return you to the Ubuntu prompt from which docker run was executed. From that point, you can execute docker run again.

—The name set using the –name option must not conflict with any existing containers. If there is already a container with the same name, either use a different name or delete the existing container (if it is no longer needed).

If you fail to give a name when using the –name option, the container’s name will be set to a random string of characters. You can check the name using the docker ps command, explained later.

You need to use the -v option to set a directory that the container user can access from CONTAINER-DIR. The Docker image gsi-openssh-20240209.tar.xz has the container user fixed to “hpciuser,” so specify a directory under /home/hpciuser.

  1. To confirm that docker is running correctly, execute the docker ps command. Check that the parameters below are correct.

CONTAINER ID

The container ID (first 12 characters) displayed when executing docker run

IMAGE

The Docker repository name:tag name specified in the option

NAMES

The name specified by the –name option.

username@WINDOWSPC:~$ docker ps
CONTAINER ID   IMAGE                       COMMAND                  CREATED         STATUS         PORTS     NAMES
10c2073ab4d3   hpci/gsi-openssh:20231128   "sudo /usr/sbin/cron…"   2 minutes ago   Up 2 minutes             gsi-openssh
username@WINDOWSPC:~$

If the output from docker ps does not look like the above, the container may have failed to launch.

The types of information displayed and their order can be changed using the –format option. To show only NAMES, CONTAINER ID, and IMAGE, you will need to execute the following command.

username@WINDOWSPC:~$ docker ps --format "{{.Names}} {{.ID}} {{.Image}}"
gsi-openssh 10c2073ab4d3 hpci/gsi-openssh:20231128
username@WINDOWSPC:~$

3.5 Running bash

Bash environment

The bash environment’s settings are as follows. These are all fixed values in the gsi-openssh-20240902.tar.xz Docker image.

  • Username (user ID)

    hpciuser (2000)

  • Group name (group ID)

    hpciuser (2000)

  • Home directory

    /home/hpciuser

  • Time zone (“TZ” environment variable setting)

    JST-9

Run bash

  1. Use the docker exec command to run /bin/bash in the container launched in Step 3.4. Here, this is gsi-openssh.

username@WINDOWSPC:~$ docker exec -i -t gsi-openssh /bin/bash
[hpciuser@XXXXXXXXXXXX ~]$

The options -i and -t are needed to run bash interactively; you can combine these as “-it”.

-i

Keep standard input (STDIN) open.

-t

Allocate a pseudo-terminal (pseudo TTY) to the container process.

After the bash process starts, you will see the following.

  • The prompt will not be an Ubuntu prompt but a bash prompt.

  • The “XXXXXXXXXXXX” part is the short form of the container ID.

3.6 Logging into the login server

This section provides an overview of the login procedure. See User’s Guide, HPCI Login Manual for details.

Issue a proxy certificate

First, a proxy certificate must be issued using HPCI Certificate Issuing System and stored in the proxy certificate repository.

Attention

  • Proxy certificates are valid for up to 168 hours.

  • It should be issued (stored) each time it expires.

Download a proxy certificate

Download the proxy certificate to the Docker container .

  1. Download the proxy certificate using the myproxy-logon command.

[hpciuser@XXXXXXXXXXXX ~]$ myproxy-logon -s portal.hpci.nii.ac.jp -l [HPCI-ID]
Enter MyProxy pass phrase: ********
A credential has been received for user <HPCI-ID> in /tmp/x509up_up2000.
[hpciuser@XXXXXXXXXXXX ~]$

  1. Verify the information in the resulting proxy certificate using the grid-proxy-info command.

[hpciuser@XXXXXXXXXXXX ~]$ grid-proxy-info
subject  : /C=JP/O=NII/OU=HPCI/CN=user/CN=proxy/CN=proxy/CN=proxy/CN=proxy
issuer   : /C=JP/O=NII/OU=HPCI/CN=user/CN=proxy/CN=proxy/CN=proxy
identity : /C=JP/O=NII/OU=HPCI/CN=user
type     : RFC 3820 compliant impersonation proxy
strength : 2048 bits
path     : /tmp/x509up_u2000
timeleft : 12:01:15
[hpciuser@XXXXXXXXXXXX ~]$

Log in to the login server

  1. Log in to the login server. In the following example, we are logging in to the login server at the Tokyo Institute of Technology, login.t3.gsic.titech.ac.jp.

[hpciuser@XXXXXXXXXXXX ~]$ gsissh -p 2222 login.t3.gsic.titech.ac.jp
Last login: Tue Jan 24 11:09:13 2023 from xxx.xxx.xxx.xxx
--------------------------------------------------------------------
Last modified: 2023-04-06 17:00:00 JST

 ** Do not run heavy programs like ISVs on login nodes login[01]. **

    (The current TSUBAME 3.0 operational status)
    https://www.t3.gsic.titech.ac.jp/      Twitter:@Titech_TSUBAME
--------------------------------------------------------------------
<HPCI-ID>@login1:~>

  1. When you exit the login server, you will return to the container’s bash shell .

<HPCI-ID>@login1:~> exit
logout
Connection to login.t3.gsic.titech.ac.jp closed.
[hpciuser@XXXXXXXXXXXX ~]$

3.7 Exiting bash

  1. To return to Ubuntu from the container, execute exit in bash.

[hpciuser@XXXXXXXXXXXX ~]$ exit
exit
username@WINDOWSPC:~$ exit

The exit command shuts down the bash process, but the container has not stopped. You can run bash as in Step 3.5 to use the gsissh command again. If you do this while the proxy certificate is valid, you will not need to re-download the proxy certificate.

3.8 Shutting down the Docker container

  1. To shut down the container, execute the docker stop command. The container you shut down will be the one you launched in Step 3.4, in this case, gsi-openssh

username@WINDOWSPC:~$ docker stop gsi-openssh
gsi-openssh
username@WINDOWSPC:~$

Attention

When you shut down a container, it is deleted. Any files created and saved in that container will also be deleted. Before shutting down, you should transfer any important files to Ubuntu.

3.9 About saving files to the volum

If you WSL 2-mount to an Ubuntu directory using the -v LISToption to the docker run command, you will be able to share files between the container and Ubuntu. See Creating a WSL 2-mounted directory for instructions on creating a WSL 2-mounted directory.

4. Updating Docker images

This section will explain how to update an old version of a Docker image to a new version.

We will be using the following Docker image names in these instructions. Substitute the version you are using as appropriate.

Docker image names

New version

hpci/gsi-openssh:20240902

Old version

hpci/gsi-openssh:20240209

4.1 About storing different versions of Docker images

You can use tag identifiers to differentiate multiple versions of Docker images, making it possible to store multiple versions of hpci/gsi-openssh in local storage. To save both versions, load the new version (hpci/gsi-openssh:20240902) into an environment where the old version (hpci/gsi-openssh:20240209) is already loaded.[This sentence has been rephrased to improve the flow. Please confirm.]

username@WINDOWSPC:~$ docker image ls
REPOSITORY         TAG        IMAGE ID       CREATED         SIZE
hpci/gsi-openssh   20240902   b3c96bb6283d   10 days ago    310MB
hpci/gsi-openssh   20240209   dd8e64a66353   7 months ago   320MB
username@WINDOWSPC:~$

In other words, even loading the new version does not automatically erase the previously loaded old version. If the old version of the image is no longer needed, you will need to explicitly erase it.

4.2 Installing the new version of the Docker image

To install the new version of the Docker image, follow the same instructions you used for the initial deployment in 3.3 Loading the Docker image (only when installing)

4.3 Deleting an old version of a Docker image

If the old version of the Docker image is no longer needed, follow these instructions to delete it from local storage.

Shut down the Docker container

If the Docker image you want to delete is running as a container, you will need to first shut down that container.

  1. Check what Docker container is running using the docker ps command.

username@WINDOWSPC:~$ docker ps -a
CONTAINER ID   IMAGE                       COMMAND                  CREATED         STATUS         PORTS     NAMES
30cbfe896943   hpci/gsi-openssh:20240209   "sudo /usr/sbin/cron…"   2 minutes ago   Up 2 minutes             gsi-openssh
username@WINDOWSPC:~$

You can see that the container gsi-openssh generated by the old version of the Docker image (hpci/gsi-openssh:20240209) is running.

  1. Shut down the running container using the docker stop command.

    To shut down the container named gsi-openssh, execute the following command.

username@WINDOWSPC:~$ docker stop gsi-openssh
gsi-openssh
username@WINDOWSPC:~$

If you used the –rm option when you executed the docker run command, the container will be deleted when it is shut down.

Delete the Docker container

If you did not use the –rm option when you executed the docker run command, the container will not be deleted when it is just shut down. Perform the following steps to delete the container.

  1. Check all the containers, including stopped ones, using the docker ps -a command.

username@WINDOWSPC:~$ docker ps -a
CONTAINER ID   IMAGE                       COMMAND                  CREATED         STATUS                     PORTS     NAMES
30cbfe896943   hpci/gsi-openssh:20240209   "sudo /usr/sbin/cron…"   4 minutes ago   Exited (0) 2 minutes ago             gsi-openssh
username@WINDOWSPC:~$

You can see that the container gsi-openssh generated by the old version of the Docker image (hpci/gsi-openssh:20240209) exists and is in an “Exited” state.

  1. To delete the Docker container, use the docker rm command.

    Here, we are deleting the container named gsi-openssh.

username@WINDOWSPC:~$ docker rm gsi-openssh
gsi-openssh
username@WINDOWSPC:~$

Delete the Docker image

  1. To delete the Docker image, execute the docker rmi command.

    Here, we are deleting the image named hpci/gsi-openssh:20240209.

username@WINDOWSPC:~$ docker rmi hpci/gsi-openssh:20240209
Untagged: hpci/gsi-openssh:20240209
Deleted: sha256:dd8e64a66353987d617cf242913b2145a7d19cbc80c1fec01329d29997fe6fe0
Deleted: sha256:02d45363cf7630d18dc82f0561ad606a5371b31d21401bcee8d66b5d3cfe5475
Deleted: sha256:53a8a85090e505fae2ed361790bfde3e3fe35e25d5e513827be6dbe76e0aadc0
Deleted: sha256:a0d6a1a8cc9943689acd24e6402aaf89bdf3bb0c32a0af6d8761e316cc999c8c
Deleted: sha256:ebfcbccf216ba5ac337f2d2a04909a7e6fa8ce69ab4adc19b39203efd04183d5
Deleted: sha256:8b72ea9dbe57bf486c9a45f5be486cb0d6a2172838e117894af48c34ee79d505
Deleted: sha256:bb077eef2bb1df3d5cd9dee745744763984d88397026509020f9fa7ff9697619
username@WINDOWSPC:~$

Attention

If you attempt to delete a container generated from an image while it still exists, you will get the following error message:

username@WINDOWSPC:~$ docker rmi hpci/gsi-openssh:20240209
Error response from daemon: conflict: unable to remove repository reference “hpci/gsi-openssh:20240209” (must force) - container 30cbfe896943 is using its referenced image dd8e64a66353

username@WINDOWSPC:~$