Weride chooses Weka to manage Its AI data pipeline
WekaIO (Weka), a specialist in high-performance and scalable file storage, announced today that WeRide, a smart mobility company with autonomous driving technologies, has selected the Weka File System (WekaFS), the world’s fastest shared parallel file system from WekaIO, to manage its artificial intelligence (AI) data pipeline from the edge to the core to the cloud.
WeRide implemented WekaFS using a hybrid model to manage compute and storage resources both on-premises using commodity Intel x86-based servers and in the Amazon Web Services (AWS) Cloud. WeRide chose WekaFS because it presented a hardware-agnostic solution that was also the most cost-efficient, delivering high-bandwidth I/O to the company’s GPU farm for high performance with mixed workloads.
WeRide is a multi-faceted AI startup that works on advanced research and development (R&D) cycles for Level 4 (L4) autonomous driving vehicles and on partnerships with transportation platform providers that support robotaxi services for commuters. The company processes data at the petabyte (PB) level with a daily mix of large video and image files generated from mapping the operational design domain for the robotaxi service. The images are collected from more than 2 million kilometers of driving distance. WeRide produces millions of high-quality labeling data that is annotated at the core, trained by the AI model on the cloud-based cluster, and fed back to the on-premises AI engine.
WeRide needed a cost-effective solution that would provide high I/O bandwidth to keep the GPU farm saturated with data. It also had to deliver high performance for mixed workloads with significant volumes of metadata, enable a hybrid implementation for the organisation to maximise its investment, and be hardware-agnostic to allow for flexible and scalable expansion. WeRide selected Weka as it met every one of these requirements and provided technical support expertise and positive customer references.
“We had built a GPU farm, and we needed a high-speed data pipe to feed it without significantly impacting our bottom line. After an extensive cost analysis and evaluation of several alternative solutions, including open-source and HDFS, Weka stood out as the clear leader,” says Paul Liu, engineering operations lead at WeRide. “Weka was the best choice for our hardware procurement model and was ideal for fulfilling our objective to make our storage a utility for our users—completely hardware-agnostic and transparent to the end-user. Weka’s customer references demonstrated product maturity and the technical support team proved invaluable by getting us launched in AWS.”
Beyond delivering high I/O bandwidth to data-hungry GPUs to keep them fully utilised, WekaFS is the world’s fastest and most scalable file system, perfect for data-intensive applications, whether hosted on-premises or in the public cloud. It is a POSIX file system that scales performance linearly as the GPU server farm grows, so WeRide will not have to compromise performance with future expansions. And since WeRide is running WekaFS on GPU servers in converged mode, creating a single namespace from all the locally attached NVMe drives, they will not have to invest in expensive hardware for their on-premises cluster.
“The Weka software has allowed WeRide to take maximum advantage of its investment in GPUs while achieving the required infrastructure cost efficiencies. In addition, with our file system they get the benefit of workload uniformity and mobility between their on-premises cluster and the public cloud,” said Liran Zvibel, co-founder and chief executive officer at WekaIO. “Weka is solving big problems for customers who require the flexibility of a software-only storage solution that provides all the traditional enterprise features while delivering superior performance at scale. Innovators such as WeRide share my vision to make storage a utility, it’s there and it works, and data available to anyone in the organisation who needs it in a predictable time, no matter where they are.”
“Datacentres are evolving, incorporating accelerated computing technologies and cloud strategies to support new workloads such as AI or machine learning. WekaFS is cloud-optimised and architected to provide high bandwidth I/O to GPU-enabled compute clusters playing a big role in enabling digital transformation. We are pleased to have been the solution provider of the Weka software licenses for WeRide to drive their AI workflow,” added Chris Saso, CTO, at Dasher Technologies, a WIN Leader Partner.
Comment on this article below or via Twitter @IoTGN