spark-MemoryManager-properties.adoc

MemoryManager Configuration Properties

The page describes the memory-related configuration properties of Apache Spark (mostly MemoryManager and "relatives").

Note	It may be that the page should be somewhere else. Any suggestions?

Use the following idiom to set the configuration properties.

import org.apache.spark.internal.config._
import org.apache.spark.SparkConf
// all properties are private[spark], sorry
// the following won't work unless the code is under org.apache.spark package
val sparkConf = new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false")

import org.apache.spark.SparkConf
val sparkConf = new SparkConf()
scala> println(sparkConf.getOption("spark.memory.offHeap.enabled"))
None

// Using ConfigEntry-based SparkConf.set that is also private[spark]
// Use :pa -raw in spark-shell to paste the following snippet
// BEGIN
package org.apache.spark.japila

object Config {
  import org.apache.spark.SparkConf
  import org.apache.spark.internal.config._
  def with_MEMORY_OFFHEAP_ENABLED(sparkConf: SparkConf = new SparkConf()): SparkConf = {
    sparkConf.set(MEMORY_OFFHEAP_ENABLED, true)
  }
}
// END

import org.apache.spark.japila.Config
val sparkConf_OffHeap = Config.with_MEMORY_OFFHEAP_ENABLED(sparkConf)

scala> println(sparkConf_OffHeap.get("spark.memory.offHeap.enabled"))
true

Table 1. Memory-Related Configuration Properties

Name	Default Value	Description
spark.memory.offHeap.enabled	false	Controls whether Spark will attempt to use off-heap memory for certain operations (true) or not (false). Tracks whether Tungsten memory will be allocated on the JVM heap or off-heap (using sun.misc.Unsafe). If enabled, spark.memory.offHeap.size has to be greater than 0. Used exclusively when MemoryManager is requested for tungstenMemoryMode. Note Before Apache Spark 1.6, spark.memory.offHeap.enabled was known as spark.unsafe.offHeap.
spark.memory.offHeap.size