[SOLVED] Spark Check If Table Exists In Hive Using Apache Spark Or PySpark? – Great And Simple 2 Snippets Of Code!

Spark check if table exists in Hive

When you are looking for hive table please provide table name in lowercase, due to fact that spark.sqlContext.tableNames returns the array of table names only in lowercase (PySpark check if table exists in Hive or Scala).

Information about tables in Hive are stored in Hive Metastore.

Spark 2.0 or higher

// Create SparkSession object with enabled Hive support
val spark = SparkSession
.builder()
.appName("Check table")
.enableHiveSupport()
.getOrCreate()
// Select database where you will search for table - lowercase
spark.sqlContext.sql("use bigdata_etl")
spark.sqlContext.tableNames.contains("schemas")
res4: Boolean = true

// With Uppercase
spark.sqlContext.tableNames.contains("Schemas")
res4: Boolean = false

Since Spark 1.6 to 2.0

// Get HiveContext from SparkContext
val sparkConf = new SparkConf().setAppName("Check table")
val sc = new SparkContext(sparkConf)
val hiveContext = new HiveContext(sc)
hiveContext.sql("use bigdata_etl")
hiveContext.tableNames.contains("schemas")

// With Uppercase
hiveContext.tableNames.contains("Schemas")
res4: Boolean = false

If table will exist you will give the “true”, otherwise “false”

PySpark Check if table exists

The above examples are presented using Scala language, but we are using a pure Spark API here, so in PySpark it will look the same. In that easy way you can check if table exists PySpark.

Could You Please Share This Post? 
I appreciate It And Thank YOU! :)
Have A Nice Day!

YOU MIGHT ALSO LIKE