Databricks recently introduced the Data Intelligence Platform, an advanced platform that uses artificial intelligence to help companies process their data.
The company presented this platform on Wednesday. It is based on Databricks’ leading Data Lakehouse, which combines the functions of data lakes and data warehouses in a single product. Lakehouses enable the analysis of structured, unstructured and semi-structured data sets in a centralized environment and provide access to error prevention functions and SQL support.
Data Lakehouse: centerpiece of the new platform
The latest Data Intelligence Platform combines Databricks’ Data Lakehouse with technologies the company acquired through its $1.3 billion purchase of MosaicML Inc. in June. MosaicML, co-founded by Naveen Rao, former vice president of Intel Corp.’s AI product group, offered a platform for developing neural networks as well as a set of pre-trained large-scale language models.
Integration of MosaicML technology in DatabricksIQ
Databricks used MosaicML’s technology to develop a software engine called DatabricksIQ. This engine drives the new Data Intelligence Platform. Databricks has detailed several components of DatabricksIQ in the months leading up to this week’s official launch.
Databricks Assistant: Natural language data queries
One of the components, the Databricks Assistant, enables users to query data using natural language queries. The software automatically converts these queries into SQL queries or Python code. In addition, the Databricks Assistant facilitates tasks such as optimizing an existing Python script that a company’s data scientists use for analysis projects.
Unity Catalog: AI-based data management and search
“By using AI models, DI platforms enable working with data in natural language, adapted to the specific jargon and acronyms of each organization,” explained Databricks CEO Ali Ghodsi, the company’s other co-founders and Rao in a blog post. “The platform learns from the use of the data in existing workflows to capture the organization’s terminology and provides a tailored natural language interface for all users, from non-experts to data engineers.”
Optimization of backend components through AI
Databricks reports that its existing Unity Catalog tool is now also equipped with AI functions. This tool creates a central directory of a company’s internal databases, making it easier for employees to find specific data records. According to Databricks, Unity Catalog uses AI to automatically attach tags and descriptions to databases, which simplifies the search.
In addition to the user-oriented functions, the company uses AI to optimize certain backend components of its Lakehouse platform. Databricks explains that a neural network supports the autoscaling mechanism of its Delta Live Tables ETL (Extract, Transform, Load) tool and the Serverless Jobs feature. The company also uses AI to rewrite user queries to increase their speed.
Broad acceptance and application in various industries
The CEO added that Databricks is also seeing strong demand for its platform’s AI development tools. The company has rented 15,000 cloud-based graphics cards to support its customers’ AI projects. These chips are already “at capacity” due to high usage, and some customers have to wait to gain access.
High demand for AI development tools and impressive sales growth
This demand is reflected in strong sales growth for Databricks. Following a 500 million dollar financing round in September, the company announced that its annual turnover now exceeds 1.5 billion dollars. Databricks reached this milestone in the second quarter following annual sales growth of over 50%.
Do you want to address these issues?
Ailio GmbH is a Bielefeld-based service provider specializing in data science and artificial intelligence. We advise in both areas and unleash the potential of data that is currently lying fallow in German SMEs. In doing so, we take a cost-optimizing and risk-minimizing approach. If you are interested, please contact us directly!
In the chemical industry, the development of new products is at the heart of technological progress. A key method in this area is Chemical Product
In the world of data analysis and big data management, Azure Synapse Analytics and Databricks are two prominent names. Both offer powerful tools for processing
The implementation of Databricks, a leading platform for big data analytics and artificial intelligence, is a crucial step for companies looking to improve their data
