In this tutorial, you will learn "How to remove NULL values from PySpark arrays?" in DataBricks.
In PySpark, the array_compact function is used to remove null elements from an array. It returns a new array with null elements removed. This function is useful when dealing with arrays in DataFrame columns, especially when you want to clean up or filter out null values from array-type columns.
πYou will frequently want to throw out the NULL values in a PySpark array rather than write logic to deal with these values. array_compact makes getting rid of NULL values quite easy.
Steps to be followed -
π Import necessary classes and functions from Apache Spark.
// import libraries
from pyspark.sql import SparkSession
from pyspark.sql.functions import array, lit, array_compact
π Create SparkSession
# Create a Spark session
spark = SparkSession.builder.appName("ArrayCompact").getOrCreate()
π Create Sample data with arrays
# Sample data with arrays containing null elements
data = [(1, ["Blue", None, "Green", None]),
(2, [None, "Yellow", "Orange", None]),
(3, ["Black", "Gray", None, None]),
(3, ["White", "Red", None, "Purple"])]
π We create a sample DataFrame df with an "id" column and a "colors" column containing arrays with null elements as given below
# Create a DataFrame from the sample data
df = spark.createDataFrame(data, ["id", "colors"])
π Show the data from dataframe
# Show the original DataFrame
print("Original Data")
df.show(truncate=False)
πuse the array_compact function to remove null elements from the "colors" arrays and create a new column "colors_compact" in the DataFrame df_compact.
# Apply array_compact to remove null elements from the arrays
df_comfact= df.withColumn("colors_compact", array_compact("colors"))
The withColumn function is used to apply the array_compact transformation to the "colors" column and create a new DataFrame df_compact.
πshow the DataFrame after compacting arrays to observe the changes.
# Show the DataFrame after compacting arrays
print("After array compact")
df_comfact.show(truncate=False)
Please watch our demo video at YouTube-
As shown in the output, the
array_compact
function removes null elements from the arrays in the "colors" column, resulting in a cleaner array without null values in the "colors_compact" column. Adjust the column names and data as needed for your specific use case.To learn more, please follow us -
To Learn more, please visit our YouTube channel at —
To Learn more, please visit our Instagram account at -
To Learn more, please visit our twitter account at -
PySpark provides a convenient approach to cleaning array-type columns, and the array_compact function is especially useful when NULL elements need to be removed without writing additional filtering logic. This kind of transformation is valuable in practical data-processing workflows where maintaining clean and consistent datasets is important for downstream analysis.
ReplyDeleteThe use of PySpark and Databricks makes Big Data Projects a relevant area for this tutorial. Working with array columns and handling missing values demonstrates how distributed data-processing platforms can simplify common data-cleaning operations, particularly when datasets become large and require scalable processing techniques.
The array-cleaning example also connects naturally with a Data Analytics Course. Understanding how to identify and remove NULL values is an important part of preparing data for analysis, since inconsistent or missing elements can affect subsequent transformations and analytical operations.
ReplyDeleteOnce the data has been cleaned and prepared, presenting the resulting information effectively becomes an important next step. Data Visualization Training in India can help learners develop the skills needed to communicate analytical results clearly through appropriate charts, dashboards, and visual representations.
ReplyDeleteThis is a clear and practical PySpark tutorial. The array_compact() function is especially useful when working with array-type columns that contain NULL elements, as it simplifies data-cleaning operations and makes downstream processing more efficient. Including sample data and step-by-step instructions helps readers understand how to apply this function in real-world data engineering workflows.
ReplyDeleteFor developers who want to strengthen their Apache Spark skills and gain hands-on experience with distributed data processing, PySpark Online Training in India is relevant for learning how to process large datasets and build practical data pipelines.
PySpark is also an important component of modern data engineering workflows, where data ingestion, transformation, and processing need to work together efficiently. Data Engineering Online Training in India can help learners explore broader data engineering concepts and understand how tools such as Apache Spark fit into end-to-end data pipelines.
ReplyDelete