Package org.apache.spark

Core Spark classes in Scala.

See: Description

Interface Summary

Interface	Description
AccumulableParam<R,T>	Helper object defining how to accumulate values of a particular type.
AccumulatorParam<T>	A simpler version of AccumulableParam where the only data type you can add in is the same type as the accumulated value.
FutureAction<T>	:: Experimental :: A future for the result of an action to support cancellation.
Logging	:: DeveloperApi :: Utility trait for classes that want to log data.
Partition	A partition of an RDD.
TaskEndReason	:: DeveloperApi :: Various possible reasons why a task ended.
TaskFailedReason	:: DeveloperApi :: Various possible reasons why a task failed.

Class Summary

Class	Description
Accumulable<R,T>	A data type that can be accumulated, ie has an commutative and associative "add" operation, but where the result type, R, may be different from the element type being added, T.
Accumulator<T>	A simpler value of Accumulable where the result type being accumulated is the same as the types of elements being merged, i.e.
Aggregator<K,V,C>	:: DeveloperApi :: A set of functions used to aggregate data.
ComplexFutureAction<T>	:: Experimental :: A FutureAction for actions that could trigger multiple Spark jobs.
Dependency<T>	:: DeveloperApi :: Base class for dependencies.
ExceptionFailure	:: DeveloperApi :: Task failed due to a runtime exception.
ExecutorLostFailure	:: DeveloperApi :: The task failed because the executor that it was running on was lost.
FetchFailed	:: DeveloperApi :: Task failed to fetch shuffle data from a remote node.
HashPartitioner	A Partitioner that implements hash-based partitioning using Java's Object.hashCode.
InterruptibleIterator<T>	:: DeveloperApi :: An iterator that wraps around an existing iterator to provide task killing functionality.
NarrowDependency<T>	:: DeveloperApi :: Base class for dependencies where each partition of the parent RDD is used by at most one partition of the child RDD.
OneToOneDependency<T>	:: DeveloperApi :: Represents a one-to-one dependency between partitions of the parent and child RDDs.
Partitioner	An object that defines how the elements in a key-value pair RDD are partitioned by key.
RangeDependency<T>	:: DeveloperApi :: Represents a one-to-one dependency between ranges of partitions in the parent and child RDDs.
RangePartitioner<K,V>	A Partitioner that partitions sortable records by range into roughly equal ranges.
Resubmitted	:: DeveloperApi :: A ShuffleMapTask that completed successfully earlier, but we lost the executor before the stage completed.
SerializableWritable<T extends org.apache.hadoop.io.Writable>
ShuffleDependency<K,V>	:: DeveloperApi :: Represents a dependency on the output of a shuffle stage.
SimpleFutureAction<T>	:: Experimental :: A FutureAction holding the result of an action that triggers a single job.
SparkConf	Configuration for a Spark application.
SparkContext	Main entry point for Spark functionality.
SparkContext.DoubleAccumulatorParam$
SparkContext.FloatAccumulatorParam$
SparkContext.IntAccumulatorParam$
SparkContext.LongAccumulatorParam$
SparkEnv	:: DeveloperApi :: Holds all the runtime environment objects for a running Spark instance (either master or worker), including the serializer, Akka actor system, block manager, map output tracker, etc.
SparkFiles	Resolves paths to files added through SparkContext.addFile().
Success	:: DeveloperApi :: Task succeeded.
TaskContext	:: DeveloperApi :: Contextual information about a task which can be read or mutated during execution.
TaskKilled	:: DeveloperApi :: Task was killed intentionally and needs to be rescheduled.
TaskResultLost	:: DeveloperApi :: The task finished successfully, but the result was lost from the executor's block manager before it was fetched.
UnknownReason	:: DeveloperApi :: We don't know why the task ended -- for example, because of a ClassNotFound exception when deserializing the task result.

Exception Summary

Exception	Description
SparkException
TaskKilledException	:: DeveloperApi :: Exception thrown when a task is explicitly killed (i.e., task failure is expected).

Package org.apache.spark Description

Core Spark classes in Scala. A few classes here, such as Accumulator and StorageLevel, are also used in Java, but the org.apache.spark.api.java package contains the main Java API.