Configuration of Apache Spark, Scala and IntelliJ IDEA

Configuration of Apache Spark, Scala and IntelliJ IDEA

Hello! I assume that you have already installed IntelliJ IDEA software (in otherwise please go to official IntelliJ website and download the community edition).

1. Install SBT and Scala plugins to IntelliJ

In IntelliJ please go to Plugins–>Browse repositories and install SBT and Scala plugins. After that please restart your IntelliJ.



2. Create Scala project

Let’s create new Scala project. Click “Create new project” and select “SBT”.


In the next window set the project name and choose correct Scala version. For Spark 2.3.1 version the Scala must be in 2.11.x minor version. I selected 2.11.8.


3. Add Apache Spark libraries

In build.sbt file please add Spark libraries. Please make sure that new libraries were downloaded. You can select auto-import option and in the future if you will add new libraries thay will be downloaded automatically.

name := "FirstSparkScalaProject"
version := "0.1"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.3.1",
  "org.apache.spark" %% "spark-sql" % "2.3.1"
)

4. Create Spark Application

Now we are ready to create Spark application. Let’s create new Scala object and set as name “FirstSparkApplication”. 

package com.bigdataetl

import org.apache.spark.sql.SparkSession

object FirstSparkApplication extends App {
  val spark = SparkSession.builder
    .master("local[*]")
    .appName("Word count")
    .getOrCreate()
  val data = spark.sparkContext.parallelize(
    Seq("I like Spark", "Spark is awesome", "My first Spark job is working now and is counting these words"))
  val wordCounts = data
    .flatMap(row =& gt;
  row.split(" ")
  )
  .map(word =& gt;
  (word, 1)
  )
  .reduceByKey(_ + _)
  wordCounts.foreach(println)
}

5. Run the application

Spark will run locally on your computer. You should see the results like this one:

(is,3)
(Spark,3)
(like,1)
(first,1)
(awesome,1)
(job,1)
(now,1)
(I,1)
(words,1)
(working,1)
(and,1)
(counting,1)
(these,1)
(My,1)

If you enjoyed this post please leave the comment below or share this post on your Facebook, Twitter, LinkedIn or another social media webpage.

Thanks in advanced!

Please follow and like us:

This Post Has One Comment

  1. Great content useful for all the candidates of Spark Training who want to kick start these career in Spark Training field.

Leave a Reply to Kelly Technologies Cancel reply

Close Menu
Social media & sharing icons powered by UltimatelySocial

Enjoy this blog? Please spread the word :)