Monday, December 4, 2023
HomeBig DataObject Storage a ‘Whole Cop Out,’ Hammerspace CEO Says. ‘You All Received...

Object Storage a ‘Whole Cop Out,’ Hammerspace CEO Says. ‘You All Received Duped’

(NAS CREATIVES/Shutterstock)

The mass adoption of object storage techniques like Amazon S3 could seem like a prime achievement of the massive information period, since we obtained basically limitless storage accessible by REST instructions. However to Hammerspace CEO David Flynn, object storage is a “complete cop out” that perpetuates information orchestration complications. “That was the cut price you made with the satan,” Flynn mentioned on the current HPC + AI on Wall Avenue occasion.

The trade standardized on object storage as a result of the market discovered it too troublesome to do the fitting factor: turning normal NFS right into a distributed file system able to dealing with cloud scale, Flynn mentioned September 27 throughout his keynote deal with at Tabor Communications’ 2023 HPC + AI on Wall Avenue convention in New York Metropolis.

“That is all about fixing the evil that file techniques weren’t in a position to do cloud-scale,” Flynn mentioned. “Let’s face it: your complete motive Amazon created S3–Tremendous Easy Storage–was as a result of file techniques are so rattling onerous to make and even tougher to make them scale that they dumbed it down and mentioned ‘Let’s simply use REST interfaces from consumer house and never attempt to get that top bar of getting the OS know truly mount it and eat information from it.’

“Object storage was all the time a cop out, and also you guys all obtained duped into considering you need to rewrite your entire purposes to make use of it simply to get to cloud,” he continued. “And it’s as a result of they didn’t do the onerous work of creating an actual file system that OSes natively know use, and that’s nonetheless thought-about the standard knowledge. ‘Cloud means rewrite every part to make use of object storage and speak to persistence from consumer house.’

Hammerspace CEO David Flynn at HPC + AI on Wall Avenue September 27, 2023

“That may be a complete cop out,” Flynn mentioned. “It’s throwing the newborn out with the bathtub water. However they needed to do it. There was no different strategy to host the a number of tenants and the large quantity of scale that was wanted within the cloud. However that was the cut price you made with the satan to get to the cloud as you rewrite that stuff.”

These had been sturdy phrases, however with many years of expertise within the enterprise IT and HPC markets, Flynn has discovered a factor or two about excessive efficiency storage. Earlier than co-founding flash storage startup Fusion-io, which was acquired by SanDisk for $1.1 billion 2014, Flynn helped construct among the world’s greatest supercomputer at Linux Networx. These travels confirmed Flynn a major hole exists between the enterprise storage market and the HPC neighborhood, which he proposes to assist fill along with his information orchestration startup, Hammerspace.

AI’s Good Storm

The arrival of AI has created a “excellent storm” of wants that can show that the thing storage compromise is not enough, Flynn mentioned.

Everyone now wants HPC capabilities to coach AI fashions, however distributed file techniques are the one strategy to effectively handle I/O and preserve GPU nodes saturated. The present established order with object storage, which requires customers to rewrite their purposes from native NFS to make use of HTTP instructions (i.e. REST) over the community, doesn’t minimize it.

Knowledge I/O stays a bottleneck to AI

Whereas NFS is the defacto normal within the enterprise NAS enterprise and is natively supported in Linux, NFS has had its share of false begins. The group behind NFS tried to deal with the statefulness situation with NFS 4.0, however ended up butchering the file system, Flynn mentioned.

“NFS 4 took all the sins of NFS 3–the statelessness and all of that–and tried to treatment it by including statefulness so as to cache stuff and get it extra environment friendly,” he mentioned. “But it surely was retrofitted in each the purchasers and the servers. And also you ended up with all the overheads and evils of statelessness mixed with the overheads and evils of statefulness.”

For instance, having a program create a file after which write it so one other program on the identical laptop learn it, took 5 spherical journeys serial to the filer in NFS 3. With NFS, that jumped to fifteen serial spherical journeys, Flynn mentioned.

“They form of tousled once they launched all of the statefulness as a result of it wasn’t effectively tuned to actually be exploited by the purchasers and servers collectively, and it ended up simply being huge overhead,” Flynn mentioned. “We nonetheless pay the value immediately. You need to go to NFS 4 to perhaps get higher safety or different issues, and but you find yourself paying a large worth in a efficiency perspective. Principally, NFS has from inception all the time form of sucked.”

After he left Fusion-io, Flynn vowed to discover a strategy to treatment this example. He teamed up with Trond Myklebust, the maintainer and lead developer of the Linux kernel NFS shopper, and co-founded a brand new enterprise Hammerspace. Led by Myklebust, a Hammerspace co-founder and its CTO, the staff took what was successfully a tutorial undertaking to develop a parallel model of NFS led by Los Alamos Nationwide Lab and turned it into an enterprise-ready product.

“We launched the NFS 4.2 spec. That got here from my staff right here at Hammerspace,” Flynn mentioned. “Due to Trond, we had been in a position to trick it out to really actually work effectively, particularly with our NFS 4.2 parallel NFS server.”

A International Storage Abstraction

Growing a parallel model of NFS was essential to Hammerspace, nevertheless it was however one step on its general journey to the last word purpose: creating a brand new storage abstraction that eliminates the limitation of bodily storage and places information again within the driver’s seat.

As an information orchestration layer, Hammerspace basically connects any storage infrastructure, whether or not it speaks NFS, object storage, and even block storage, to an utility. Whether or not it’s a NAS gadget from NetApp, Dell EMC, Qumulo, or Huge–whether or not it’s an Amazon S3 bucket or a Linux server configured as a storage node–Hammerspace can take all that information and make it seem to an utility to be sitting on native storage, even when the information is sitting on the opposite aspect of the world.

Simply as characters in Japanese cartoons can pull a mallet out of skinny air and whack their opponents within the head, Hammerspace turns into the skinny air (or the only world namespace) out of which you’ll pull any piece of knowledge (simply don’t hit your opponents within the head with it).

The important thing to this seeming magic trick is unified metadata, Flynn mentioned.

“Hammerspace solves the seeming paradox of how will you have information in all places and wherever you want it, with out ever having copied it?” Flynn mentioned. “That’s not as a result of it’s not native and accessed at native entry efficiency. It’s as a result of the metadata is unified. So whereas the information is bodily distributed, the metadata is logically unified.”

Pulling the metadata out of the storage layer into a brand new layer that transcends these different layers addresses a litany of knowledge administration ache factors, Flynn mentioned. Knowledge migration turns into a factor of the previous when you’ll be able to hook up Hammerspace to an current community file system and immediately begin accessing it by an utility beforehand related to a unique storage repository, Flynn mentioned.

Hammerspace implements a worldwide information abstraction layer atop bodily storage

“When was the final time you had to consider having your information if you transfer from one cellphone to the following, or if you go out of your cellphone to your laptop computer or pill?” he mentioned. “Our shopper information already lives in a Hammerspace and the iOS platform, the Android platform have principally orchestrated your expertise and your information for you. That’s what we’re speaking about doing right here, however for the petabytes to exabytes scale unstructured information that’s behind every part.”

The headache of sustaining a excessive availability server configuration will likely be a factor of the previous when Hammerspace is guaranteeing that information from one node will be accessed from one other node. “You may eliminate all of those totally different types of copy, like your information copy sync applications, the information migration duties, if you’re going from one system to a different,” Flynn mentioned.

In the long run, Flynn hopes that Hammerspace’s notion of knowledge orchestration flips the script on trendy concepts of knowledge administration. As a substitute letting information storage outline what information is and what it means to us, the information orchestration layer defines the information as soon as, and makes the place it’s saved a mere implementation element.

“Knowledge orchestration differs from information administration in a quite simple approach: Knowledge administration is what you do from the surface of the information presentation layer,” Flynn mentioned. “The true evil is the truth that the file system is embedded within the storage system or service. The info presentation layer being inside that storage system signifies that the information is admittedly nothing however a mirage that’s being rendered by that storage. And in the event you put it in several storage, it’s by definition totally different information, as a result of its very existence is an artifact of the storage system or service presenting it.”

It appears very pure to say that your information exists in your NAS filer or your S3 bucket, as a result of that’s what possesses the metadata and presents the information to you, Flynn mentioned. However that’s truly getting it backward, as a result of information, by definition, is the next stage abstraction than storage, which is simply infrastructure. And therein lies the important thing factor that Hammerspace permits.

“Knowledge doesn’t exist apart from as rendered by storage,” Flynn mentioned. “That appears very pure to say, however it’s the other way up. It signifies that the infrastructure is in cost and the platform layer relies upon it. However with Hammerspace, that modifications since you pull all the metadata out of the storage layer into one thing that may transcend any of these storage techniques. So now you could have the place information can sit appropriately as the upper stage factor that you just give attention to, and the place the information truly lives, on which storage, can change.”

You may view Flynn’s whole HPC and AI on Wall Avenue presentation by registering at

Associated Objects:

Hammerspace Raises $56M to Reimagine Knowledge Orchestration

Three Methods to Join the Dots in a Decentralized Large Knowledge World

Hammerspace Hits the Market with International Parallel File System





Please enter your comment!
Please enter your name here

Most Popular

Recent Comments