Do tests before swapping the floating IP

It's always better to make sure there's no issue with the newly built image. Running a set of test on an instance that starts with the newly built image, only delete the old instance when it passes all of the tests.

Since we are having an intermittent problem with shibboleth, I will start with the shibd status checking as our first test.

Frameworks like Selenium might be used in the future for website tests.

added Backlog label

changed the description

assigned to @louistw

changed milestone to %ood nightly builds

marked this issue as related to #23

changed title from Do tests before swap the floating IP to Do tests before swapping the floating IP

Success:

$ systemctl status shibd
● shibd.service - Shibboleth Service Provider Daemon
   Loaded: loaded (/usr/lib/systemd/system/shibd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-10-20 17:02:16 CDT; 1min 47s ago
     Docs: https://wiki.shibboleth.net/confluence/display/SP3/Home
 Main PID: 1169 (shibd)
    Tasks: 6
   Memory: 33.1M
   CGroup: /system.slice/shibd.service
           └─1169 /usr/sbin/shibd -f -F

Failed:

$ systemctl status shibd
● shibd.service - Shibboleth Service Provider Daemon
   Loaded: loaded (/usr/lib/systemd/system/shibd.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2022-10-21 10:14:22 CDT; 16s ago
     Docs: https://wiki.shibboleth.net/confluence/display/SP3/Home
  Process: 1054 ExecStart=/usr/sbin/shibd -f -F (code=exited, status=253)
 Main PID: 1054 (code=exited, status=253)

Use wget to verify shibd:

Success(0):

--2022-10-24 10:57:14--  https://rc.uab.edu/Shibboleth.sso/Metadata
Resolving rc.uab.edu (rc.uab.edu)... 138.26.48.47
Connecting to rc.uab.edu (rc.uab.edu)|138.26.48.47|:443... connected.
WARNING: cannot verify rc.uab.edu's certificate, issued by ‘emailAddress=root@ood-b69be5b.openstack.internal,CN=ood-b69be5b.openstack.internal,OU=SomeOrganizationalUnit,O=SomeOrganization,L=SomeCity,ST=SomeState,C=--’:
  Self-signed certificate encountered.
    WARNING: certificate common name ‘ood-b69be5b.openstack.internal’ doesn't match requested host name ‘rc.uab.edu’.
HTTP request sent, awaiting response... 200 OK
Length: 7505 (7.3K) [application/samlmetadata+xml]
Saving to: ‘Metadata’

Metadata                  100%[==================================>]   7.33K  --.-KB/s    in 0s

2022-10-24 10:57:14 (107 MB/s) - ‘Metadata’ saved [7505/7505]

Fail(8):

--2022-10-24 10:57:23--  https://rc.uab.edu/Shibboleth.sso/Metadata
Resolving rc.uab.edu (rc.uab.edu)... 138.26.48.47
Connecting to rc.uab.edu (rc.uab.edu)|138.26.48.47|:443... connected.
WARNING: cannot verify rc.uab.edu's certificate, issued by ‘emailAddress=root@ood-b69be5b.openstack.internal,CN=ood-b69be5b.openstack.internal,OU=SomeOrganizationalUnit,O=SomeOrganization,L=SomeCity,ST=SomeState,C=--’:
  Self-signed certificate encountered.
    WARNING: certificate common name ‘ood-b69be5b.openstack.internal’ doesn't match requested host name ‘rc.uab.edu’.
HTTP request sent, awaiting response... 500 Internal Server Error
2022-10-24 10:57:29 ERROR 500: Internal Server Error.

added Sprint 22-21 label and removed Backlog label

From the pipeline, you can see the test, download Metadata from the VM, was successful, but when I ssh to the newly deployed Knightly, the shibd is down.

Need further investigation.

Switch from wget to curl to test downloading shibboleth Metadata. (276f4448)

Had to update docker image to include curl. (f6968992)

closed