Slurm job submit plugin to submit jobs to different filesystem

Using slurm job submit plugin and node feature to submit the job to node with correct filesystem on nodes.

The slurm version on CoD is 19.05.5 which set the lua script dir to sysconfig at compile time, see slurm bug 11411. It's fixed later in version slurm-21-08-0-0rc1, which means the job_submit.lua has to be in /etc/slurm folder before that version.

Steps to enable lua interface for job submit plugin.

1. Add JobSubmitPlugins=lua line to Slurm.conf.
2. Place job_submit.lua file from the repo and place it in /etc/slurm or whereever your slurm.conf dir is present.
3. Execute scontrol reconfigure from Head node.

assigned to @atlurie

added Sprint 24-16 label

changed the description

Our current proof of concept kind of code is as follow:

--[[

 Example lua script demonstrating the Slurm job_submit/lua interface.
 This is only an example, not meant for use in its current form.

 For use, this script should be copied into a file name "job_submit.lua"
 in the same directory as the Slurm configuration file, slurm.conf.

--]]
function dump(o)
   if type(o) == 'table' then
      local s = '{ '
      for k,v in pairs(o) do
         if type(k) ~= 'number' then k = '"'..k..'"' end
         s = s .. '['..k..'] = ' .. dump(v) .. ','
      end
      return s .. '} '
   else
      return tostring(o)
   end
end
-- Load necessary library
local pwd = require("posix").pwd
local grp = require("posix").grp

function slurm_job_submit(job_desc, part_list, submit_uid)

	local target_group = "gpfs5"
        local username = pwd.getpwuid(submit_uid).pw_name
	local group_member = grp.getgrnam(target_group).gr_mem
	local found = false

	slurm.log_info("username:" .. username)
        slurm.log_info("job_desc:" .. dump(job_desc) .. type(job_desc))
        slurm.log_info("job_desc.features:" .. dump(job_desc.features))
        slurm.log_info("part_list:" .. dump(part_list) .. type(part_list))
        slurm.log_info("submit_uid:" .. dump(submit_uid) .. type(submit_uid))

        -- Checking if the user is in the target_group
        -- By compare group member and the username
	for i,member in ipairs(group_member) do
        	if member == username then
			found = true
			job_desc.features = "gpfs5"
			break
		end
	end

	slurm.log_info("found:" .. dump(found))
        slurm.log_info("job_desc.features:" .. dump(job_desc.features))

	return slurm.SUCCESS
end

function slurm_job_modify(job_desc, job_rec, part_list, modify_uid)
	if job_desc.comment == nil then
		local comment = "***TEST_COMMENT***"
		slurm.log_info("slurm_job_modify: for job %u from uid %u, setting default comment value: %s",
				job_rec.job_id, modify_uid, comment)
		job_desc.comment = comment
	end

	return slurm.SUCCESS
end

slurm.log_info("initialized")
return slurm.SUCCESS

changed the description

assigned to @atlurie

changed milestone to %GPFS5 migration

changed the description

changed the description

changed the description

The slurm_job_modify function in job_submit.lua script is executed when a submitted job is modified with scontrol like scontrol update.

The slurm_job_submit function is executed whenever a job is submitted via sbatch

mentioned in issue devops#544 (closed)

Add default group/feature to the rest of jobs if not in gpfs5 group

added Sprint 24-17 label and removed Sprint 24-16 label

changed the description

changed the description

Merge request for lua plugin code- !1 (merged)

Follow this issue for further info https://gitlab.rc.uab.edu/rc/gpfs5-migration/-/issues/53

closed