In this tutorial, you will learn "How to Change column names from CamelCase to Snake_Case by using Scala" in Databricks.
💡Imagine we have an input Dataframe (as in the image). Our goal is to achieve the desired output Dataframe (also in the image).
Age -> Age ,
FirstName -> First_Name,
CityName -> City_Name,
CountryName -> Country_Name
To create a Dataframe in Scala, you can use Apache Spark's Dataframe API.
// import libararies
import org.apache.spark.sql.{SparkSession, Row}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
//Create Spark session
val spark=SparkSession.builder().appName("CamelToSnakeCase").getOrCreate()
//define the schema of dataframe
val schema = StructType(Array(StructField("Age", IntegerType, nullable=true),
StructField("FirstName", StringType, nullable=true),
StructField("CityName", StringType, nullable=true),
StructField("CountryName", StringType, nullable=true)
))
// Define the data as a Seq of rows
val data=Seq(Row(30,"John Ramsay","New York", "USA"),
Row(32,"Alice Wang","New York", "USA"),
Row(41,"Bob Builder","Los Angeles", "USA"),
Row(25,"Ryan Arjun","New York", "USA"))
//bind the data into dataframe
val df= spark.createDataFrame(spark.sparkContext.parallelize(data), schema)
// diaplay df
df.show()
// show schame of the dataframe
df.printSchema()
// Define Pattern
val regexPattern="(?=[A-Z])".r
// Change the column names
val newColumns:Array[String]=for {col <- df.columns}
yield {
// define variable to split column name
val splitName = regexPattern.split(col)
if (splitName.length>1)
splitName.mkString("_")
else
splitName(0)
}
//print new columns
print(newColumns)
//create new dataframe
val newDF=df.toDF(newColumns:_*)
// display records from new dataframe
newDF.show()
Show output
Please watch our demo video at Youtube-
To learn more, please follow us -
🔊 http://www.sql-datatools.com
To Learn more, please visit our YouTube channel at —
🔊 http://www.youtube.com/c/Sql-datatools
To Learn more, please visit our Instagram account at -
🔊 https://www.instagram.com/asp.mukesh/
To Learn more, please visit our twitter account at -
No comments:
Post a Comment