Fix Job Composer erroring out on virtual nodes

Job Composer requires SQLite database, and due to the sqlite db sitting on NFS mounted drive, we are running into locking issue.

added Backlog label

changed the description

https://discourse.openondemand.org/t/exception-occurred-disk-i-o-error/678/8

the problem you are experiencing could also be due to a locking issue because of using sqlite3 on NFS.

At OSC we are currently using NFSv4 and the NFS options we pass when mounting include rw,vers=4.0,rsize=65536,wsize=65536 and local_lock=none appears to also be getting set.

I’m not sure what this means in terms of solutions for you though.

https://github.com/OSC/ondemand/blob/master/apps/myjobs/config/configuration_singleton.rb#L101

Found this environment variable, DATABASE_PATH, that we could probably use to solve this issue but I couldn't find anything regarding this environment variable on the OOD documentation page.

I have created a topic on OOD discourse regarding this topic.

SQLite error with job composer - NFS mounted drive https://discourse.openondemand.org/t/sqlite-error-with-job-composer-nfs-mounted-drive/2244

changed title from Job Composer erroring out on virtual nodes to Fix Job Composer erroring out on virtual nodes

added Sprint 22-17 label and removed Backlog label

assigned to @atlurie

I'm still seeing this issue after making the change in the /etc/fstab with local_lock=none option as suggested in the above link.

Per user setup failed for script at /var/www/ood/apps/sys/myjobs/./bin/setup-production for user atlurie with output: /var/www/ood/apps/sys/myjobs/bin/setup-production:37: warning: rb_check_safe_obj will be removed in Ruby 3.0
Exception occurred: database is locked

added 3h of time spent

I feel they meant to say local_lock=posix instead of local_lock=none cause none is the default according to nfs(5):

If this option is not specified, or if none is specified, the client assumes that the locks are not local.

After setting local_lock=posix, I did not see it solving the issue. Also, I see that it only applied mount option local_lock=posix to /data/user but not /data/user/home

$ cat /proc/mounts
gpfs.rc.uab.edu:/data/user/home /home nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=xx.xx.xx.xx,mountvers=3,mountport=39157,mountproto=udp,local_lock=none,addr=xx.xx.xx.xx 0 0
...
gpfs.rc.uab.edu:/data/user /data/user nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=xx.xx.xx.xx,mountvers=3,mountport=55420,mountproto=udp,local_lock=posix,addr=xx.xx.xx.xx 0 0

NOTE: The issue is resolved after unmounting and remounting the /data/user

since /data/user is already mounted we need not have an explicit mount point specified for /data/user/home as this dir subtree can be bind mounted.

So in ansible we set up a bind mount from /data/user/home directory to /home instead of an explicit mount.

added 4h of time spent

mentioned in merge request !6 (merged)

After setting local_lock=posix in the mount options for the /data/user nfs mount, the job composer app started working

PR !6 (merged)

added Sprint 22-18 label and removed Sprint 22-17 label

mentioned in commit 8bc72d29

closed with merge request !6 (merged)