Databricks launches AI-based data intelligence platform
Introducing the Databricks Data Intelligence Platform
Databricks recently introduced the Data Intelligence Platform, an advanced platform that leverages artificial intelligence to help companies process their data. The company presented this platform on Wednesday. It is based on Databricks' leading data lakehouse, which combines the capabilities of data lakes and data warehouses in a single product. Lakehouses enable the analysis of structured, unstructured and semi-structured data sets in a centralized environment and provide access to error prevention features and SQL support.
Data Lakehouse: Core of the new platform
The latest data intelligence platform combines Databricks' data lakehouse with technologies the company acquired through its acquisition of MosaicML Inc. in June for $1.3 billion. MosaicML, co-founded by Naveen Rao, former vice president of Intel Corp.'s AI product group, offered a platform for developing neural networks as well as a set of pre-trained large-scale language models.
Integration of MosaicML technology into DatabricksIQ
Databricks used MosaicML's technology to develop a software engine called DatabricksIQ. This engine powers the new Data Intelligence Platform. Databricks has detailed several components of DatabricksIQ in the months leading up to this week's official launch.
Databricks Assistant: Natural language data queries
One of the components, the Databricks Assistant, allows users to query data using natural language queries. The software automatically converts these queries into SQL queries or Python code. Additionally, the Databricks Assistant facilitates tasks such as optimizing an existing Python script that a company's data scientists use for analysis projects.
Unity Catalog: AI-based data management and search
“Through the use of AI models, DI platforms enable working with data in natural language, adapted to the specific jargon and abbreviations of each organization,” Databricks CEO Ali Ghodsi, the company’s other co-founders and Rao explained in a blog post. “The platform learns from how data is used in existing workflows to capture organizational terms and provides a tailored natural language interface to all users, from laypeople to data engineers.”
Optimization of backend components through AI
Databricks reports that its existing Unity Catalog tool now also includes AI capabilities. This tool creates a central directory of a company's internal data assets, allowing employees to more easily find specific records. According to Databricks, Unity Catalog uses AI to automatically attach tags and descriptions to datasets, making search easier.
In addition to user-centric features, the company uses AI to optimize certain backend components of its Lakehouse platform. Databricks explains that a neural network powers the autoscaling mechanism of its Delta Live Tables ETL (Extract, Transform, Load) tool and Serverless Jobs feature. The company also uses AI to rewrite user queries to increase their speed.
Wide acceptance and application in various industries
The CEO added that Databricks is also seeing strong demand for its platform's AI development tools. The company has leased 15,000 cloud-based graphics cards to support its customers' AI projects. These chips are already “at capacity” due to high usage, and some customers are having to wait to gain access.
High demand for AI development tools and impressive sales growth
This demand is reflected in strong revenue growth for Databricks. After a $500 million funding round in September, the company announced that its annual revenue is now more than $1.5 billion. Databricks reached this milestone after posting annual revenue growth of over 50% in the second quarter.
Do you want to tackle these topics?
Ailio GmbH is a service provider from Bielefeld that specializes in data science and artificial intelligence. We advise in both areas and unleash the potential of data that is currently lying idle in German medium-sized businesses. We proceed in a cost-optimizing and risk-minimizing manner. If you are interested, please contact us directly!