Apache Spark: Convert DataFrame to DataSet – Scala
Many times you might want to have strong typing on your data in Spark. The best to get it is to DataSet insted of DataFrame. In this post I give…
Apache Spark: ReduceByKey vs GroupByKey – differences and comparison
In this post I will try to introduce you to the main differences between ReduceByKey and GroupByKey methods and why you should avoid the latter. But why? The answer is…
Apache Spark: how to rename or delete a file from HDFS?
In this short post I will show you how you can change the name of the file / files created by Apache Spark to HDFS or simply rename or delete any file.
Scala: How to run a shell command from the code level?
In this post I will show you how to run the shell command by programming in Scala and how you can use it in Apache Spark.
Apache Spark: How to save DataFrame as a single file on HDFS?
If you want to save DataFrame as a file on HDFS, there may be a problem that it will be saved as many files. This is the most correct behavior and it results from the parallel work in Apache Spark.
Apache Spark: How to check if the file exists on HDFS?
We will use the FileSystem and Path classes from the org.apache.hadoop.fs library to achieve it.
Apache Spark: Machine Learning – predicting diabetes in patients
Today I will show you how you can use Machine Learning libraries (ML), which are available in Spark as a library under the name Spark MLib.
How to install Apache Spark Standalone in CentOs?
In this tutorial I will show you how you can easily install Apache Spark in CentOs
How to check if table exists in Apache Hive using Apache Spark?
Simple short tip how to check if table exists int Hive using Spark
