You are currently viewing [SOLVED] Spark Check If Table Exists In Hive Using Apache Spark Or PySpark? – Great And Simple 2 Snippets Of Code!
Could You Please Share This Post? I Appreciate It And Thank YOU! :) Have A Nice Day!
4.9
(2703)

Spark check if table exists in Hive

When you are looking for hive table please provide table name in lowercase, due to fact that spark.sqlContext.tableNames returns the array of table names only in lowercase (PySpark check if table exists in Hive or Scala).

Information about tables in Hive are stored in Hive Metastore.

Spark 2.0 or higher

// Create SparkSession object with enabled Hive support
val spark = SparkSession
.builder()
.appName("Check table")
.enableHiveSupport()
.getOrCreate()
// Select database where you will search for table - lowercase
spark.sqlContext.sql("use bigdata_etl")
spark.sqlContext.tableNames.contains("schemas")
res4: Boolean = true

// With Uppercase
spark.sqlContext.tableNames.contains("Schemas")
res4: Boolean = false

Since Spark 1.6 to 2.0

// Get HiveContext from SparkContext
val sparkConf = new SparkConf().setAppName("Check table")
val sc = new SparkContext(sparkConf)
val hiveContext = new HiveContext(sc)
hiveContext.sql("use bigdata_etl")
hiveContext.tableNames.contains("schemas")

// With Uppercase
hiveContext.tableNames.contains("Schemas")
res4: Boolean = false

If table will exist you will give the “true”, otherwise “false”

PySpark Check if table exists

The above examples are presented using Scala language, but we are using a pure Spark API here, so in PySpark it will look the same. In that easy way you can check if table exists PySpark.

Could You Please Share This Post? 
I appreciate It And Thank YOU! :)
Have A Nice Day!

BigData-ETL: image 7YOU MIGHT ALSO LIKE

How useful was this post?

Click on a star to rate it!

Average rating 4.9 / 5. Vote count: 2703

No votes so far! Be the first to rate this post.

As you found this post useful...

Follow us on social media!

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?