Big Data and its Need for Storage

As information becomes digital more and more large companies are facing an increasing need to manage their Big Data needs with adaptable storage.  What's taking up all their storage?  Written records, customer service calls, IoT information, and other industry-specific information.  For example, insurance giant IAG managed 80TB last year but now it's managing 2PB and that data store is growing by 14TB a month.  All of this data falls under what the industry calls "volume, variety, and velocity" of data.

Our data needs are growing too - Cisco estimates that global internet traffic will crest 3.3ZB by 2021 a year.  Monthly internet demand is also estimated to reach 35GB per capita.  Though these numbers reflect a fair amount of data held on consumer devices, it does show that our consumption of data is on the rise.

At IAG, storage architecture largely consists of open source storage, and added commodity and converged storage in software-defined networks has enabled IAG to get storage in place more quickly when needed.  This ability to access additional storage comes in handy for situations like natural disasters, where demand for insurance services and room to store associated data is high.  However, this capability also comes with the need to balance cost with growth and capability.  IAG manages this by deploying a shared OpenStack cluster with multiple hosting providers.  It also has five storage tiers, with the highest tier being all flash.  Each tier is designed with a 25% buffer for quick needs and is reviewed quarterly.

More and more companies are turning toward cloud and flash storage, but they all can't switch over just yet.  Many are still running backups on different storage standards, and even daily software that isn't designed for the speed that these methods offer.  To address this, Dell EMC has taken the approach of containerization, which lessens the demand for addtional resources, but that doesn't solve the issue that old software faces.

Cloud storage is no perfect method - it doesn't work for all software and latency is an issue - so for now its major use is in backup and recovery.  However, as technology continues to develop we may see more of a reliance on cloud storage and processing for all things computing.

This article was based on a ComputerWeekly.com article by Beverley Head.  Read it here.

File To
Archived
File To
Current News