New Starburst Galaxy capabilities offer metadata, schema capabilities, reducing time to discovery
Boston, United States – Starburst, the analytics anywhere company is launching foundational Data Products capabilities for its managed service, Starburst Galaxy. New data and schema discovery and data privileges capabilities simplify and streamline the traditional extract, transform, load (ETL) process for the curation of Data Products, accelerating data querying, access and analytics across the organisation. The foundational Data Products capabilities are now available in public preview.
Data volumes and use are exploding across industries. Connected internet of things (IoT) devices are expected to generate almost 80B zettabytes (ZB) of data by 2025 and organisations are inheriting data through M&A and newly developed applications at a fast pace. Regardless of intention, nearly every modern enterprise is or will be driven to the cloud as a result.
While data lakes and data warehouses have been effective in solving many data management challenges, the cloud is still becoming everyone’s reality and data use is continuing to explode. In turn, these data lakes and warehouses can quickly become data swamps – murky or cluttered with disorganised data that presents significant challenges around accessibility and the ability to leverage the data for actionable insights. Delivered as a managed service through Starburst Galaxy, these new discoverability features are addressing these challenges, reducing time to discovery from hours to seconds, and laying the foundation for self-service Data Product curation, regardless of technical expertise. New capabilities include:
- Data discoveryenables data users to easily search and understand what data they have, where it lives, and where it came from. Metadata is automatically populated with query history and context, providing key insights into how data is being used.
- Schema discovery takes this a step further by enabling the discovery of not just existing datasets across sources and clouds, but also net new datasets, no matter what form they’re in. This takes the “Transform” out of “ELT,” supporting a more simplified process where data engineers loading data don’t need to consider the schemas beforehand.
- Granular access control enables data administrators to clearly see and understand who has access to what data, and how it’s being used, in the context of the data itself. This means data administrators can monitor and change permissions through policy as code to ensure security and risk reduction within a continuous integration / continuous delivery / continuous deployment (CI/CD) pipeline.
“Data volumes are growing exponentially and simultaneously becoming more distributed, which makes finding, managing, and curating datasets a highly time-consuming, resource-intensive process,” says Justin Borgman, chairman & CEO, Starburst. “New discoverability capabilities in Starburst Galaxy empower organisations to find and understand data before querying, laying the foundation for Data Product curation by streamlining data discovery and accelerating ELT processes. Enabling organisations to more efficiently discover the right datasets, Starburst Galaxy is helping reduce costs while getting more value out of their data.”
This Data Products update to Starburst Galaxy comes shortly after Starburst announced enhanced Data Products functionality for its flagship product, Starburst Enterprise. It also comes on the heels of a big year for Starburst’s software-as-a-service (SaaS) product where it enhanced its distributed data analytics capabilities and, less than a year after its introduction, closed its first-ever seven-figure customer deal. While enabling data producers and consumers to create, publish, discover, and manage curated Data Products is core to the emerging Data Mesh paradigm focused on decentralisation and self-service, Starburst remains steadfast in its commitment to realising the Data Mesh vision.
To learn more about Starburst, Data Mesh, and Data Products, please visit Starburst.
Comment on this article below or via Twitter @IoTGN