Databricks in Early 2026: Advancements and AI-Driven Innovations

 We will the key advancements and AI-driven innovations within the Databricks Data Intelligence Platform as of late January 2026. The updates focus on unified management, AI agent extensibility, enhanced governance, and the seamless integration of transactional and analytical workloads. These releases emphasize usability improvements, security enhancements, and deeper AI capabilities, building upon the foundation laid in late 2025.

Unified Lakebase Management: A Streamlined Experience

The introduction of the Lakebase App, accessible through the apps switcher in the Databricks UI, marks a significant improvement in usability. This unified interface consolidates the management of both Lakebase Provisioned and Lakebase Autoscaling instances, streamlining the previously cumbersome workflow that required navigating through the Compute tab in the Lakehouse UI.

Lakebase, Databricks’ fully managed PostgreSQL-compatible OLTP database, now offers a more intuitive project-based structure. This enhancement simplifies the process for developers and data teams to handle transactional data alongside analytics, eliminating the need for complex ETL pipelines. Billing for Lakebase Autoscaling commenced in January 2026, following a free exploration period in late 2025. Key features include autoscaling compute, scale-to-zero functionality, database branching, and instant restore capabilities.

Databricks Runtime 18.0 Goes Generally Available

Databricks Runtime 18.0, including its Machine Learning variant, achieved general availability in January 2026. This runtime, powered by Apache Spark 4.1.0 and utilizing JDK 21 as the default (an LTS release), incorporates a range of bug fixes, security patches, performance optimizations, and library upgrades.

Notable behavioral extensions include the application of time travel and VACUUM changes to serverless compute, Databricks SQL, and Unity Catalog managed tables across supported runtimes. These updates ensure consistent data management across diverse environments.

AI/BI and Genie Enhancements for Better Collaboration

The AI/BI suite, which includes Genie (the conversational analytics tool), has received several usability enhancements designed to improve collaboration and efficiency:

• JSON handling in tables: Objects now remain expanded during copy-paste operations, simplifying data manipulation.
• Parameterized defaults for catalogs and schemas: This feature is available in dashboard deployments via Databricks Asset Bundles, enabling greater customization and control.
• Multi-select filter improvements: Pasting comma- or newline-separated values is now easier, streamlining the filtering process.
• Enhanced question categorization in Genie: The accuracy of Genie has been improved for common query patterns such as top-K queries, percentages, and distributions.
• Default Consumer access for new users: Administrators can now set this via group cloning, granting view-only access to dashboards, Genie spaces, and apps. This is particularly useful for business users who require access to insights without the ability to create objects.

Furthermore, Databricks Assistant now supports custom agent skills for domain-specific tasks in agent mode, adhering to the open Agent Skills standard. These skills load automatically when relevant, significantly extending the Assistant’s capabilities and making it more adaptable to specific business needs.

Knowledge Assistant became generally available in select US regions (for non-enhanced security workspaces), offering connectors for popular platforms like Google Analytics, Salesforce, and ServiceNow. Row filtering has been added to managed connectors, allowing users to ingest only the necessary data via SQL-like WHERE conditions, optimizing data transfer and storage.

Security, Governance, and Model Serving Updates

Several updates have been implemented to enhance security, governance, and model serving capabilities:

• Automatic email notifications for expiring personal access tokens are now generally available, providing proactive security measures.
• Mosaic AI Model Serving has added support for hosted OpenAI GPT-5.1 Codex Max and Codex Mini models, which are specifically optimized for code-related tasks.
• Delta Sharing continues to evolve with the introduction of recipient-specific URL formats and one-year token expirations, enhancing security and control over shared data.
• AI/BI Genie and Databricks Assistant are now generally available on AWS GovCloud (including DoD environments), ensuring secure, government-compliant AI-powered workflows.

Looking Ahead in 2026

The Data + AI Summit 2026 is scheduled for June 15–18 in San Francisco, and the call for proposals is already open. Major announcements are anticipated regarding AI agents, multi-model governance, and lakehouse expansions.

Meanwhile, legacy features such as Community Edition were retired on January 1, 2026, with users being directed to the perpetual Free Edition. Legacy dashboards have been fully phased out, encouraging teams to transition to modern AI/BI tools.

These updates collectively reflect Databricks’ ongoing commitment to building a more integrated, AI-native platform. The platform aims to empower data teams to build production-grade agents, unify OLTP and OLAP workloads, and provide business users with trustworthy insights, all while maintaining centralized governance through Unity Catalog.

For the most up-to-date details, please refer to the official Databricks release notes, as rollouts are staged across AWS, Azure, and GCP.

PySpark— Update multiple columns in a dataframe

Working as a PySpark developer, data engineer, data analyst, or data scientist for any organisation requires you to be familiar with dataframes because data manipulation is the act of transforming, cleansing, and organising raw data into a format that can be used for analysis and decision making.

 

Note: We are using Databricks environment to articulate this example.

We understand, we can add a column to a dataframe and update its values to the values returned from a function or other dataframe column’s values as given below -

## importing sparksession from  

## pyspark.sql module 

from pyspark.sql import SparkSession 

from pyspark.sql.functions import col



# Create a Spark session and giving an app name 

spark = SparkSession.builder.appName("UpdateMutliColumns").getOrCreate()

When you see data in a list in PySpark, it signifies you have a collection of data in a PySpark driver. This collection will be parallelized when you construct a DataFrame. Here, we have 5 elements in a list and let’s convert this to a DataFrame as given below —

### Create a list of data

MyData=[("Finance",2), ("Marketing",4), ("Sales",6), ("IT",8),("Admin",9) ]



#### convert above list to a DataFrame

sdf =  spark.createDataFrame(data=MyData, schema=['Dept','Code'])



### show the schema of the dataframe

sdf.printSchema()



## Show the DataFrame

sdf.show(10, False)

The most important aspect of Spark SQL & DataFrame is PySpark UDF (User Defined Function), which is used to enhance the PySpark built-in capabilities.

 

Note — UDFs are the costliest procedures, therefore use them only when you have no other option and when absolutely necessary. In the next part, I will explain in detail why utilising UDFs is a costly activity.

 

User-defined scalar functions — Python : This page covers examples of Python user-defined functions (UDFs). It demonstrates how to register and invoke UDFs.

 

#create square() function to return single value 

#passing variable is x

#return single value

def square(x):

  return x*x

  

#create Cube function to return single value 

#passing variable is x

#return cubes

def Cube(x):

  return x*x*x

 

Register a function as a UDF — In PySpark, you can add custom UDFs in PySpark spark context as given below-

 

## Register the function as a UDF (User-Defined Function)

spark.udf.register("square_udf", square)

spark.udf.register("Cube_udf", Cube)

 

Add and Update multiple columns in a dataframe — If you want to update multiple columns in dataframe then you should make sure that these columns must be present in your dataframe. In case, updated columns are not in your dataframe, you must create them as given below — 

 

### Lets add a new column in the dataframe as SalesAmount

sdf2=sdf.withColumn("Square", expr("square_udf(Code)"))

sdf2=sdf2.withColumn("Cube", expr("Cube_udf(Code)"))



## Show the DataFrame

sdf2.show(10, False)

Based on the official documentation, withColumn returns a new DataFrame by adding a column or replacing the existing column that has the same name.

 

Now, the above example shows you how to update multiple columns inside your dataframe in PySpark. By using withColumn, you can only create or modify one column at each time.

 

To learn more, please follow us -
http://www.sql-datatools.com

To Learn more, please visit our YouTube channel at —
http://www.youtube.com/c/Sql-datatools

To Learn more, please visit our Instagram account at -
https://www.instagram.com/asp.mukesh/

To Learn more, please visit our twitter account at -
https://twitter.com/macxima

To Learn more, please visit our medium account at -
https://macxima.medium.com/

User Experience — Databricks Vs Snowflake

 Cloud is the fuel that drives today’s digital organisations, where businesses pay only for those selective services or resources that they use over a period of time.

If you are trying to evaluating Databricks (on AWS/Azure) and Snowflake for your company, then some users’ experience and thoughts can be very helpful to answer your questions.

You can consider an example scenario where there are no cluster/SQL warehouses running on both platforms. A data analyst comes in and starts hitting a table with some queries, and this is what happens on both platforms.

Databricks: It takes 3–4 minutes to spin up the compute cluster/SQL warehouse, and the first query that hits it takes a long time to return some results. Databricks is more open and the types of features they are releasing cover most of the things such as data governance, security , change data capture , AI/ML etc.

Snowflake: The SQL warehouse startup time is within seconds, and even the first query returns results quickly.

We can create cluster pools and keep some clusters idle (warm) in Databricks, which can reduce startup time, but are we not paying more money to keep servers inactive 24/7?

The question is how teams are managing Databricks to be up all the time and at the same time maintain the costs. Doesn’t Snowflake have an edge over this as you are not always keeping the cluster/warehouse active.

Big Data Tools: When to Use Snowflake vs Databricks

Cloud is the fuel that drives today’s digital organisations, where businesses pay only for those selective services or…

macxima.medium.com

One of our main evaluation criteria is user experience (at the same time maintaining low costs). We don’t want people using these platforms to wait for a considerable amount of time to run their queries.

Are there any guidelines for choosing the instance families for the cluster, as there are quite a few of them, and it would be great if you guys could provide some tips on choosing them, as we feel the jobs are running slowly because we are not choosing the right instance family for the cluster? Also, do you guys recommend SQL warehouses, as they spin up quite large machines (which can cost more) even for a smaller warehouse.

• Snowflake clusters run within the snowflake plane, that’s the reason it can repurpose VMs instantaneously for its customers whereas in Databricks, clusters run in the customer plane (customer VPC or VNet), so acquiring a VM and starting the cluster takes time.
• There’s a serverless option in Databricks also, which runs within no time. It’s a new offering where the VMs run in the Databricks plane. Databricks SQL warehouse has simplified cluster sizing similar to snowflake(t-shirt sizing).

Databricks SQL has set a new world record in 100TB TPC-DS, the gold standard performance benchmark for data warehousing. Databricks SQL outperformed the previous record by 2.2x. Unlike most other benchmark results, this one has been formally audited and reviewed by the TPC council. 
(These results were corroborated by research from the Barcelona Supercomputing Center, which frequently runs benchmarks that are derivatives of TPC-DS on popular data warehouses.)
Their latest research benchmarked Databricks and Snowflake, and found that Databricks was 2.7x faster and 12x better in terms of price performance. This result validated the thesis that data warehouses such as Snowflake become prohibitively expensive as data size increases in production.Note: Snowflake often won't do benchmarks on these things anymore-they say that their focus is not on performance in a benchmarking context. The benchmarks that are referenced are standardized. They're not relevant to every use case, but they're not just made up by Databricks.

As far as Snowflake vs Databricks, the biggest difference is that Snowflake stores their data in a proprietary format inside their own servers and uses their own servers for compute costs, so there isn’t that provisioning stage that takes 5 minutes.

Databricks uses mostly open source software and utilizes cloud companies’ compute and storage costs. For instance, Databricks just deploys a root folder onto S3, connects permissions via instance profiles, and requests their EC2 instances for the nodes. You won’t pay AWS for the EC2, but Databricks charges you via DBU (Databricks units) and their business model is that their platform will save you money compared to going to the cloud directly.

With a bunch of cool features that others lack, Snowflake actually helps you manage the costs. Auto-suspend warehouse can be as little as 1 minute, and it can wake up almost instantly, and you can do cool things like having differently sized virtual warehouses for different workloads. Caching technology is awesome and works really well. Multi-clustering on demand works great too. Databricks is ages behind in this regard. They are only now and still testing serverless mode. It is not available in Azure, only in AWS.

What kind of user experience do teams who have onboarded Databricks give to their users querying it?

Users can use Databricks and have SQL clusters running with a 30–60 minute timeout, and they run essentially during all business hours, and then they have streaming jobs running 24/7.

What instance family do you use for streaming job clusters?

A user can use a combination of X1000X and X1002T for streaming jobs and run them on non-premium instances to save money.

What size SQL warehouse did you use for SQL endpoints?

For SQL warehouse, you have xl for loading data to cubes and DataScience and xs for low-load development work.

What was the query response time for the SQL endpoints? Did any of your users complain about any slowness or wait time?

Since most users consume data via cubes and PowerBI, it hasn’t been an issue so far, and they perform similarly to the previous SQL server. But if instant response times are crucial, you would consider a proper SQL Server for those.

What’s the cost comparison of this new Databricks offering compared to Snowflake?

The most expensive Databricks is $0.55. Clusters range lower depending on what you choose for resources. Expect cores to be more expensive and memory to be not too bad.

The separation of compute and storage in Snowflake is way more advanced than in Databricks.

• Databricks compute is customer-managed and takes a long time to start-up unless you have EC2 nodes waiting in hot mode, which costs money. Snowflake compute is pretty much serverless and will start in most cases in less than 1 second.
• Databricks compute will not auto-start, which means you have to leave the clusters running to be able to allow users to query DB data. Snowflake compute is fully automated and will auto start in less than a second when a query comes in without any manual effort.

Snowflake can access and write the same data(parquet, CSV, JSON, Orc, Avro ) that is sitting on external blob stores that you manage just like Databricks.

Databricks is a proprietary software layer based on open-source code. This means neither DeltaLake nor any of the DB-specific features will work with anything other than Databricks. So you are locked in to the Databricks software stack for all the workloads. Just because you store the data files on S3 or an Azure blob does not mean you can use them with any other platform at your leisure.

Definitely look at serverless. You can get very similar startup times, and Databricks has made a lot of improvements to Databricks SQL just over the last 12 months. With Databricks, you can also schedule all your production data pipelines, and now you can also call SQL queries from workflows.

I think the Lakehouse concept is going to be the future. Performance is very close on both platforms, and even when something is slower, you are not talking about hours vs minutes.

You get a lot more with Databricks at a lower cost. If you have more predictable workloads, you can also take advantage of reserved compute instances and lower your costs even more. Also, your data is stored in your own account in an open format.

In Snowflake, you want to keep the warehouse open for reporting tools for fast serving and cache reuse, but that costs. Snowflake handled small ad hoc queries from self-service points quite nicely.

It really depends how much control you want over your costs and how your contracts are negotiated. Databricks has a lot more customizability and they have some internal libraries that are useful for data engineers. For instance, Mosaic is a really useful geospatial library that was created inside Databricks.

Snowflake is much more intuitive and similar to an SQL client. Snowflake has their own variant of Lakehouse buildout called “SnowPark”. Lakehouse isn’t really particular to Databricks, they just follow their own variant called “Medallion Architecture”

Databricks should have the most optimized libraries for dealing with delta, but I have not encountered a scenario in operational workloads where those optimizations have mattered. That’s because, unlike in the old days when we used highly customized third party drivers to squeeze every bit out of the processing, Delta allows consumers to bring their own compute.

In Azure, start-up for Databricks clusters is insane — 15–30 minutes unless you reserve a pool of VMs — this is an instance because AWS Glue can start workers in 5–10 seconds and this has been a feature now for a few years.

Databricks is generally cheaper (cost for X performance), so it’s easier to keep a shared autoscaling cluster running in Databricks than in Snowflake. Same for warm-start pools. It’s not a 1:1 comparison with regard to cost over time for the same performance.
Last but not least, you can use any platform you feel is best for the job, but be aware of the maintenance, cost, and performance factors for anything you implement. Snowflake is especially essential for applications involving advanced analytics and data science. Data scientists primarily utilize R and Python to handle large datasets. Databricks provides a platform for integrated data science and advanced analysis, as well as secure connectivity for these domains.

To learn more, please follow us -
http://www.sql-datatools.com
To Learn more, please visit our YouTube channel at —
http://www.youtube.com/c/Sql-datatools
To Learn more, please visit our Instagram account at -
https://www.instagram.com/asp.mukesh/
To Learn more, please visit our twitter account at -
https://twitter.com/macxima