Hadoop integration in the Open Cluster Scheduler

[nmartin0]

I've read through some materials from the period when Sun Microsystems was developing their Sun Grid Engine. It appears that, nearing the end of their development on SGE, there were integrations for Hadoop being released. I haven't found a clear indication as to whether this Hadoop integration has been retained to HPC's modern continuation of the SGE. Is there Hadoop integration in the HPC cluster schedulers?

[ernst-bablick]

The repository contains the Hadoop directory. I don't know what state these scripts and code are in. Hadoop is not on our roadmap, but we'll support you if you want to try it...

[jgabler-hpc]

The Hadoop integration is in clusterscheduler/source/dist/hadoop.
It comes with a README.txt describing step by step what to do,
but it hasn't been updated since it was introduced by Sun Microsystems somewhen in 2009, so it might be outdated.

I did fixes to the setup.pl script which can be used to configure required complex variables, a parallel environment and the load sensor. This basic set-up should work now, but I don't have a Hadoop installation at hand, so can't do further testing for now.

[ernst-bablick]

@nmartin0 Did you make some progress? Need any help?

[nmartin0]

@ernst-bablick Thanks for reaching out. I had intended to do more testing with the clusterscheduler sooner. I'm trying to bootstrap an illumos variant and intend to add the clusterscheduler to its build system once it's in place. Once the resources are available, I would like to continue testing with the clusterscheduler and its Hadoop integration

[ernst-bablick]

Sounds great! Ernst