A new report has shed light on the magnitude of the dilemma organizations face as demand for data storage capacity continues to rise.
Jointly published by Fujitsu and Twist Bioscience, both of which are active in the archival storage market, the report (opens in new tab) predicts that the gap between available storage capacity and demand will exceed 7.8 million petabytes by 2030.
In this scenario, companies have no choice but to delete large chunks of old data to make room for new ones, which is enough to send a shiver down the spine of any company with aspirations in areas such as artificial intelligence.
As the amount of data produced by internet activity, digital devices and IoT sensors continues to grow at an aggressive pace, businesses are running out of time to solve a critical problem: where to put everything.
While hard disks (HDDs) and solid state drives (SSDs) do a great job of holding and delivering the amounts of data that servers and client devices must function, but neither is capable of storing information massively and for long periods of time.
When it comes to archive storage, Linear Tape-Open (LTO) magnetic tape reigns supreme, with the lowest cost per capacity of any technology. The current-generation tape, LTO-9, has an original capacity of 18 TB and can be purchased for as little as $150 (or about $8.30/TB).
According to the report, large companies will need to invest heavily in tape and other archival media as the amount of data produced by their operations continues to grow. The alternative would be to discard old data, but that would mean missing out on its potential value as a source of insight; the most advanced AI products are typically informed by the largest, most comprehensive sets of data.
“We Believe Most Of This” [new] corporate data will be unstructured, ‘cold’, rarely accessed and maintained at minimal cost,” explains report author Jon Monroe, who says the spread of storage spend should reflect that fact.
While it is cost-effective, tape also has its weaknesses; data can only be accessed serially, making it difficult to locate specific files, and companies must also migrate to new tape on a semi-regular basis to avoid data loss.
In light of these issues, researchers are looking for new ultra-dense and ultra-durable storage technologies. A few different candidates have emerged, but one concept seems particularly promising: DNA.
DNA, the basic material of living organisms, consists of four molecular building blocks: adenine (A), guanine (G), cytosine (C) and thymine (T). These compounds connect in pairs (AT & GC) to form the rungs of the famous double helix ladder.
This structure can be used as an extremely dense and durable form of data storage, converting binary ones and zeros into the four-letter genetic alphabet. A single gram of DNA has been found suitable for storing 215 PB (220,000 TB) of data.
“DNA promises to offer the magic three in storage: ultra-high density, reasonable cost, and durability,” said Emily Leproust, CEO and co-founder of Twist Bioscience, which is investing heavily to bring the technology to fruition.
“We anticipate that new media will be required to meet the more than $7 billion in unmet storage demand expected in the coming years.”
As it stands, the technology remains largely unusable due to the time it takes to write data to DNA and several other challenges. Of course, the report should also be taken with a grain of salt, as it was produced by two organizations with an interest in increasing spending on records preservation.
However, there is no denying that the increase in capacity of traditional data storage technologies is not keeping pace with the pace of data production, meaning that a reorientation of the storage stack is inevitable.
“The data centers of the future will need everything the SSD, HDD and tape industries can manufacture and supply, as well as new DNA and optical and perhaps other enterprise storage technologies, to store the priceless artifacts of our personal storage in a cost-effective way. and reliable way. , corporate and cultural history,” Monroe added.
“Availability and sustainability challenges, combined with the cost of managing our multimillion-fold-petabyte dataverse over increasingly long periods of time, will create new use cases for legacy storage technologies and the creation of new, more cost-effective and energy-efficient storage technologies.”
Through Blocks and files (opens in new tab)