R

R Parallel

Purpose

Here, we briefly explain different ways to use R in parallel on the Harvard University FASRC Cannon cluster.

• For R basics, refer to R and R Studio
• For R Parallel, refer to R-Parallel

Processing large datasets

• Working with large data that does not fit into memory
• Processing Single instruction multiple data problem on shared and distributed memory systems

Single-node, multi-core (shared memory)

• Package parallel

  • FAS RC embarrassingly parallel documentation
  • FAS RC embarrassingly parallel Cannon example (using parLapply)
  • FAS RC Embarrassingly parallel VDI example (using parLapply)
  • parallel documentation

• Package future

  • Install future on Cannon
  • Example of multisession (not shared memory) and multicore (shared memory) and its submit script
  • future documentation

Multi-node, distributed memory

• Package Rmpi

  • Install Rmpi on Cannon
  • Example and its submit script
  • Rmpi documentation

• Package pbdMPI (programming big data MPI)

  • Install pbdMPI on Cannon
  • Examples based on the pbdMPI demos – after installing pbdMPI package, all demos can be found in your R library folder $HOME/apps/R/4.0.5/pbdMPI/demo
  • pbdMPI documentation and GitHub
  • pbdR website

Hybrid: Multi-node + shared-memory

Using nested futures and package future.batchtools, we can perform a multi-node and multi-core job.

• Package future and future.batchtools
  • Install future and future.batchtools on Cannon
  • Example and its submit script
  • future documentation and GitHub
  • future.batchtools documentation and GitHub