Lustre extractor

[blastmaster]

As already said we need a specialized extractor for the lustre parallel file system.
Please start with the points from our last meeting.
Don't mention the lctl command at the moment, just focus on the lustre API.
The lctl stuff must be executed on the lustre server nodes, so that this might be another special case for the future but not in this task.

[blastmaster]

Hint to use the lustre API, include #include <lustre/lustreapi.h> and link against liblustreapi -llustreapi afterward. Lustre is a system installation on taurus, under /usr/lib64..., therefore loading additional modules is not necessary.

[k4lipso]

Overview of lfs commands. If they have a tick they are working and will be implemented.

file required

• lfs getstripe
• lfs data_version
• lfs hsm_state

no file required

• lfs quota - works partially "Some errors happened when getting quota info."
• lfs check <osts|mds|server> - help i dont get it to work, what can i provide as "server"?
• lfs getdirstripe - looks like it is allways returning the same
• lfs df
• lfs getname
• lfs path2fid
• lfs pool_list - no errors, but also no pools listed
• lfs osts
• lfs mdts

[k4lipso]

@blastmaster
For gathering striping information we have to create a dummy file on a lustre mountpoint. Functions for dummy file creation and getting file striping information allready exists, but we still need to find a place to create that file.

Right now, to find a place, we gather every lustre mountpoint using the lustreapi and then pass the mountpoint path to the following function:

sysmap/lib/src/linux/lustre_extractor.cc

Lines 298 to 322 in 3bf46ac

	bool Lustre_Extractor::handle_getstripe(const std::string& mount_path, data& result)
	{
	//get username of current user
	const std::string username = getlogin();
	//create path from mountpoint of lustre filesystem and username
	const std::string userpath = mount_path + '/' + username;
	//check if path exists
	if(boost::filesystem::exists(userpath)){
	//create path to yet non existing dummy file using userpath
	std::string dummy_file = userpath + "/lustre_dummy_test_file";
	//try to create dummy file
	if(create_dummy_file(dummy_file)){
	//gather striping information of the dummy file
	file_getstripe(dummy_file, result);
	//try to remove dummy file
	if( std::remove(dummy_file.c_str()) ){
	utils::log::logging::error()
	<< "[Lustre_Extractor] could not delete dummy file:";
	}
	return true;
	}
	}
	return false;
	}

Basicly this function gets the username of the current user, appends it to the mountpoint path, checks if that path exists and if that is the case it creates the dummy file.
The problem is that the dummy file creation only works if an folder named like the current user exists on that mountpoint. This approach could fail on other HPCs.
Another approach would be to check every folder on the mountpoint for access rights which could be a very time intensive task.
Do you have any recommendation on handling this problem?