Scala — Transpose or Pivot | Rows to Columns in Dataframe | Databricks

In this tutorial, you will learn "How to Transpose or Pivot | Rows to Columns in Dataframe by using Scala" in Databricks.

Data integrity refers to the quality, consistency, and reliability of data throughout its life cycle. Data engineering pipelines are methods and structures that collect, transform, store, and analyse data from many sources.

Scala is a computer language that combines the object-oriented and functional programming paradigms. Martin Odersky invented it, and it was initially made available in 2003. "Scala" is an abbreviation for "scalable language," signifying the language's capacity to grow from simple scripts to complex systems.

Scala is a language designed to be productive, expressive, and compact that can be used for a variety of tasks, from large-scale corporate applications to scripting. It has become more well-liked in sectors like banking, where its robust type system and expressive syntax are very helpful.

If you want Transpose or Pivot | Rows to Columns in Dataframe by using Scala in Databricks, then you have to follow the following steps - 💎 Import necessary Spark classes for DataFrame operations.

//import libraries

import org.apache.spark.sql.{SparkSession, Row}

import org.apache.spark.sql.functions._

import org.apache.spark.sql.types._

💎 Create a SparkSession.

// Create Spark Session

val spark = SparkSession.builder().appName("TransposeRows").getOrCreate()

💎Create DataFrames from a sample CSV file which is already stored in DBFS.

// file path

val FilePath="dbfs:/FileStore/tables/StoreSales.csv"

// read data into dataframe from file

val df=spark.read.option("header","true").csv(FilePath)

// show dataframe schema

df.printSchema()

💎Show the data from the Dataframe as given below -

// display records from dataframe

df.show()

💎 Create new Dataframe by applying the transpose or pivot logics as given below -

// transpose data into new dataframe

val transposeDF=df.groupBy("Ship_Mode").pivot("Region").agg(sum("Sales"))

// display records

transposeDF.show()

Please watch our demo video at YouTube-

To learn more, please follow us - 🔊 http://www.sql-datatools.com To Learn more, please visit our YouTube channel at — 🔊 http://www.youtube.com/c/Sql-datatools To Learn more, please visit our Instagram account at - 🔊 https://www.instagram.com/asp.mukesh/ To Learn more, please visit our twitter account at -

🔊 https://twitter.com/macxima