In this tutorial, you will learn "How to remove NULL values from PySpark arrays?" in DataBricks.
In PySpark, the array_compact function is used to remove null elements from an array. It returns a new array with null elements removed. This function is useful when dealing with arrays in DataFrame columns, especially when you want to clean up or filter out null values from array-type columns.
💎You will frequently want to throw out the NULL values in a PySpark array rather than write logic to deal with these values. array_compact makes getting rid of NULL values quite easy.
Steps to be followed -
💎 Import necessary classes and functions from Apache Spark.
// import libraries
from pyspark.sql import SparkSession
from pyspark.sql.functions import array, lit, array_compact
💎 Create SparkSession
# Create a Spark session
spark = SparkSession.builder.appName("ArrayCompact").getOrCreate()
💎 Create Sample data with arrays
# Sample data with arrays containing null elements
data = [(1, ["Blue", None, "Green", None]),
(2, [None, "Yellow", "Orange", None]),
(3, ["Black", "Gray", None, None]),
(3, ["White", "Red", None, "Purple"])]
💎 We create a sample DataFrame df with an "id" column and a "colors" column containing arrays with null elements as given below
# Create a DataFrame from the sample data
df = spark.createDataFrame(data, ["id", "colors"])
💎 Show the data from dataframe
# Show the original DataFrame
print("Original Data")
df.show(truncate=False)
💎use the array_compact function to remove null elements from the "colors" arrays and create a new column "colors_compact" in the DataFrame df_compact.
# Apply array_compact to remove null elements from the arrays
df_comfact= df.withColumn("colors_compact", array_compact("colors"))
The withColumn function is used to apply the array_compact transformation to the "colors" column and create a new DataFrame df_compact.
💎show the DataFrame after compacting arrays to observe the changes.
# Show the DataFrame after compacting arrays
print("After array compact")
df_comfact.show(truncate=False)
Please watch our demo video at YouTube-
As shown in the output, the
array_compact
function removes null elements from the arrays in the "colors" column, resulting in a cleaner array without null values in the "colors_compact" column. Adjust the column names and data as needed for your specific use case.To learn more, please follow us -
To Learn more, please visit our YouTube channel at —
To Learn more, please visit our Instagram account at -
To Learn more, please visit our twitter account at -
PySpark provides a convenient approach to cleaning array-type columns, and the array_compact function is especially useful when NULL elements need to be removed without writing additional filtering logic. This kind of transformation is valuable in practical data-processing workflows where maintaining clean and consistent datasets is important for downstream analysis.
ReplyDeleteThe use of PySpark and Databricks makes Big Data Projects a relevant area for this tutorial. Working with array columns and handling missing values demonstrates how distributed data-processing platforms can simplify common data-cleaning operations, particularly when datasets become large and require scalable processing techniques.
The array-cleaning example also connects naturally with a Data Analytics Course. Understanding how to identify and remove NULL values is an important part of preparing data for analysis, since inconsistent or missing elements can affect subsequent transformations and analytical operations.
ReplyDeleteOnce the data has been cleaned and prepared, presenting the resulting information effectively becomes an important next step. Data Visualization Training in India can help learners develop the skills needed to communicate analytical results clearly through appropriate charts, dashboards, and visual representations.
ReplyDeleteThe array_compact example is a practical way to handle NULL elements inside PySpark arrays without having to write additional filtering logic. I especially like how the tutorial focuses on a common DataFrame cleaning task and shows how the function produces a cleaner array for downstream processing.
ReplyDeleteThis type of array and DataFrame transformation is an important part of PySpark Training, where understanding how Spark functions manipulate structured data is essential. Removing NULL elements at the transformation stage can make later operations much simpler and more predictable.