Sunday, February 25, 2024

Scala — Retrieve matched rows from two Dataframes in Databricks

In this tutorial, you will learn "How to Retrieve matched rows from two Dataframes by using Scala" in Databricks.

Data integrity refers to the quality, consistency, and reliability of data throughout its life cycle. Data engineering pipelines are methods and structures that collect, transform, store, and analyse data from many sources.

Scala is a computer language that combines the object-oriented and functional programming paradigms. Martin Odersky invented it, and it was initially made available in 2003. "Scala" is an abbreviation for "scalable language," signifying the language's capacity to grow from simple scripts to complex systems.

Scala is a language designed to be productive, expressive, and compact that can be used for a variety of tasks, from large-scale corporate applications to scripting. It has become more well-liked in sectors like banking, where its robust type system and expressive syntax are very helpful.

If you want to retrieve matched rows from two DataFrames based on two or more columns, you can still use the join method in Spark DataFrame API but you'll need to specify multiple columns in the join condition. πŸ’Ž Import necessary Spark classes for DataFrame operations.
//import libraries
import org.apache.spark.sql.{SparkSession, Row}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
πŸ’Ž Create a SparkSession.
// Create Spark Session
val spark= SparkSession.builder().appName("RetrieveMatchedRows").getOrCreate()
πŸ’ŽCreate two sample DataFrames, df1 and df2, with some common columns called "EmpId" from sample CSV files.
// File1 - Employee Info
val FileEmpInfo="dbfs:/FileStore/EmployeeInfo.csv"

// File2 - Employee Distribution
val FileEmpDist="dbfs:/FileStore/EmployeeDistribution-1.csv"

// Read data into dataframe 1 from File1

// show the data from df1

// Read data into dataframe 2 from File2

// show the data from df2

πŸ’ŽPerform an inner join using join method, specifying column "id" ) in the join condition and the join type ("inner").
// join df1 and df2 on cloumn EmpId with inner join
val joinDF=df1.join(df2,Seq("EmpId"),"inner")
πŸ’Ž Finally, display the matched rows using show() method on the joined DataFrame.
//display the data

// print schema of the data

Please watch our demo video at Youtube-

To learn more, please follow us - πŸ”Š To Learn more, please visit our YouTube channel at — πŸ”Š To Learn more, please visit our Instagram account at - πŸ”Š To Learn more, please visit our twitter account at -

No comments:

Post a Comment