Runs independent parallel R sessions by executing a remote script in a high performance cluster via SSH. The script is currently configured for the Ukko cluster.
- Task (session) id is appended as the last argument for R.
- Task ids can be specified flexibly.
- If the number-of-tasks > number-of-available-remote-nodes, tasks are queued and nodes are recycled.
- A pool of available nodes is maintained and unresponsive/troublesome nodes are removed.
- Number of nodes in the pool can be limited or extra nodes can be reserved for later use.
- Nodes can be filtered by maximum allowed load and a blacklist of hosts.
- Output (stdout, stderr) is redirected to log files in remote nodes.
Runs test.R in the remote hosts with task ids 1,2,3 and 5 on two nodes with maximum load 10.0:
parallel_r.py -t 1:3,5 -n 2 -l 10.0 -b blacklist.txt -v test.RThe file blacklist.txt contains excluded hosts separated by new lines.
See parallel_r.py --help for more details.
Kills all your R processes in all remote hosts:
remote_command.py "killall -s SIGKILL R"Since the file system may not be up-to-date due to caching mechanism of NFS, there is a script that checks if a local git repository matches remote. The script can be run inside a shell script like this:
if ! "$exec_path"/git_uptodate.sh; then
exit
fi
```
in the repository directory. Variable `$exec_path` points to the parallel tasks directory.
Extending
---------
The Python class in `independent_parallel_tasks.py` can be extended for custom HP clusters,
see `ukko_cluster.py` for an example.
TODO
----
* Allow specifying remote username, port, nice parameter, etc.
* Allow running other programs than R.
FIXME
-----
* Replace popen with thread pooling.
Feedback
--------
Jussi Jousimo, jvj@iki.fi