something overwrote vagrant authorized_keys file on ohpc node

I was unable complete vagrant up ohpc because ssh auth stopped working. The machine did complete booting and was accessible on the console.

Inspecting the authorized_keys file for vagrant showed the provisioned keys were no longer present. Only the cluster key (generated by ohpc) was there. I had to manually store the authorized_keys for the public key used for the provision in order to make vagrant up complete successfully.

Have the same issue. Even with ood node.

[vagrant@ood ~]$ ls -lr .ssh/
total 8
-rw-r--r--. 1 vagrant vagrant 718 Mar 15 16:05 known_hosts
-rw-------. 1 vagrant vagrant   0 Feb 12 00:30 config
-rw-r--r--. 1 vagrant vagrant   0 Feb 12 00:30 cluster.pub
-rw-------. 1 vagrant vagrant   0 Feb 12 00:30 cluster
-rw-------. 1 vagrant vagrant 421 Mar 14 19:30 authorized_keys

The work around for this issue is:

• to open the ohpc console in virtualbox
• log in as vagrant/vagrant
• edit the /etc/ssh/sshd_config and set PasswordAuthentication yes
• restart sshd systemctl restart sshd
• from the host ssh to ohpc via vagrant ssh ohpc using password authn
• copy your ssh id_rsa.pub (or equiv) key to the ohpc vagrant users ~/.ssh/authorized_keys
• you should now be able to vagrant ssh ohpc without error
• optional: undo the password authentication change above

If you cluster keys are destroyed, simply rerun the cluster-env script that gets run on first login.

I was led to this issue by looking through the issues after encountering the following problem after using vagrant up ohpc for the first time.

    ohpc: 
    ohpc: PLAY [ohpc] ********************************************************************
    ohpc: 
    ohpc: TASK [Gathering Facts] *********************************************************
    ohpc: ok: [ohpc]
    ohpc: 
    ohpc: TASK [pre_ohpc : fail] *********************************************************
    ohpc: fatal: [ohpc]: FAILED! => {"changed": false, "msg": "Kernel version mismatch between loaded kernel and group-vars/all! Please fix and run again."}
    ohpc: 	to retry, use: --limit @/vagrant/CRI_XCBC/site.retry
    ohpc: 
    ohpc: PLAY RECAP *********************************************************************
    ohpc: ohpc                       : ok=1    changed=0    unreachable=0    failed=1   
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

The problem I encountered may be different, though. I found that my public key had already been picked up and was available within the VM, so that vagrant ssh ohpc worked right away without the above steps.

@alansill thanks for the additional data point. The error you are having relates to a variable setting in the CRI_XCBC/groups_var/all file. It needs to be changed to match the current version of the kernel on the ohpc node as determined by uname -r.

I've opened issue #14 to fix this.

Thanks. A related problem: the cluster keys are not created and there is no cluster-env script on the ohpc node.

[vagrant@ohpc ~]$ ls -al .ssh
total 8
drwx------. 2 vagrant vagrant  48 Mar 18 17:33 .
drwx------. 3 vagrant vagrant  95 Mar 18 17:35 ..
-rw-------. 1 vagrant vagrant 822 Mar 18 17:11 authorized_keys
-rw-r--r--. 1 vagrant vagrant 340 Mar 18 17:36 known_hosts

[vagrant@ohpc ~]$ ls -l /etc/profile.d/
total 68
-rw-r--r--. 1 root root  771 Oct 30 22:57 256term.csh
-rw-r--r--. 1 root root  841 Oct 30 22:57 256term.sh
-rw-r--r--. 1 root root  660 Jun 10  2014 bash_completion.sh
-rw-r--r--. 1 root root  196 Mar 24  2017 colorgrep.csh
-rw-r--r--. 1 root root  201 Mar 24  2017 colorgrep.sh
-rw-r--r--. 1 root root 1741 Oct 30 14:37 colorls.csh
-rw-r--r--. 1 root root 1606 Oct 30 14:37 colorls.sh
-rw-r--r--. 1 root root   80 Oct 30 19:48 csh.local
-rw-r--r--. 1 root root 1706 Oct 30 22:57 lang.csh
-rw-r--r--. 1 root root 2703 Oct 30 22:57 lang.sh
-rw-r--r--. 1 root root  123 Jul 30  2015 less.csh
-rw-r--r--. 1 root root  121 Jul 30  2015 less.sh
-rw-r--r--. 1 root root   81 Oct 30 19:48 sh.local
-rw-r--r--. 1 root root  105 Oct 30 19:57 vim.csh
-rw-r--r--. 1 root root  269 Oct 30 19:57 vim.sh
-rw-r--r--. 1 root root  164 Jan 27  2014 which2.csh
-rw-r--r--. 1 root root  169 Jan 27  2014 which2.sh

@alansill the provisioning of the ohpc node failed and should be restarted. See the work around in #14 for steps to fix this.