You are currently viewing [SOLVED] Configuration of Apache Spark Scala and IntelliJ IDEA – short and easy 5 steps!
Could You Please Share This Post? I Appreciate It And Thank YOU! :) Have A Nice Day!
4.8
(931)

Let’s start with configuration of Apache Spark Scala and IntelliJ! I assume that you have already installed IntelliJ IDEA software (in otherwise please go to official IntelliJ website and download the community edition).

Configuration of Apache Spark Scala and IntelliJ – 5 steps!

1. Install SBT and Scala Plugins To IntelliJ

In IntelliJ please go to Plugins–>Browse repositories and install SBT and Scala plugins. After that please restart your IntelliJ.

Configuration of Apache Spark Scala and IntelliJ IDEA - short and easy 5 steps!
Configuration of Apache Spark Scala and IntelliJ IDEA - short and easy 5 steps!

2. Create Scala Project

Let’s create new Scala project. Click “Create new project” and select “SBT”.

Configuration of Apache Spark Scala and IntelliJ IDEA - short and easy 5 steps!

In the next window set the project name and choose correct Scala version. For Spark 2.3.1 version the Scala must be in 2.11.x minor version. I selected 2.11.8.

Configuration of Apache Spark Scala and IntelliJ IDEA - short and easy 5 steps!

3. Add Apache Spark Libraries

In build.sbt file please add Spark libraries. Please make sure that new libraries were downloaded. You can select auto-import option and in the future if you will add new libraries thay will be downloaded automatically. (Configuration of Apache Spark Scala and IntelliJ)

name := "FirstSparkScalaProject"
version := "0.1"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.3.1",
  "org.apache.spark" %% "spark-sql" % "2.3.1"
)
Configuration of Apache Spark Scala and IntelliJ IDEA - short and easy 5 steps!

4. Create Spark Application

Now we are ready to create Spark application. Let’s create new Scala object and set as name “FirstSparkApplication”. 

package com.bigdataetl

import org.apache.spark.sql.SparkSession

object AnalyzerWords extends App {

  val spark = SparkSession.builder
    .master("local[*]")
    .appName("Word count")
    .getOrCreate()
  val data = spark.sparkContext.parallelize(
    Seq("I like Spark", "Spark is awesome", "My first Spark job is working now and is counting these words"))
  val wordCounts = data
    .flatMap(row => row.split(" "))
    .map(word => (word, 1))
    .reduceByKey(_ + _)
  wordCounts.foreach(println)
}

5. Run The Application

Spark will run locally on your computer. You should see the results like this one:

(is,3)
(Spark,3)
(like,1)
(first,1)
(awesome,1)
(job,1)
(now,1)
(I,1)
(words,1)
(working,1)
(and,1)
(counting,1)
(these,1)
(My,1)
Could You Please Share This Post? 
I appreciate It And Thank YOU! :)
Have A Nice Day!

BigData-ETL: image 7YOU MIGHT ALSO LIKE

How useful was this post?

Click on a star to rate it!

Average rating 4.8 / 5. Vote count: 931

No votes so far! Be the first to rate this post.

As you found this post useful...

Follow us on social media!

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?