When it comes to cloud and hybrid storage there’s a lot to think about. In this extended blog post, Masstech CDO/CTO James Whitebread gives an oversight of what you need to think about, what you can expect, and some of the tools that can help.
Why are more and more companies moving to the cloud?
An increasing number of organisations are moving to cloud for their video and wider file storage, but what are the primary drivers behind this? Naturally there are a several, of differing importance to individual companies – let’s take a look at a few: cost, enterprise readiness, flexibility, scalability, data security, and online collaboration
Cost: As we all know by now, broadly speaking accessing resources in the cloud means you only pay for what you need, when you need it. Cloud saves on human resources (you don’t need large admin teams), requires no self-hosted data centres, and eliminates storage-related power, air-con and resource handling (these are just the ‘big ticket’ items). All of these mean that cloud storage can often be lower cost, and importantly more predictable.
Enterprise Ready: Cloud-based providers offer enterprise grade infrastructure with all of the considerations an enterprise would expect built into each and every instantly available service.
Flexibility: Cloud providers generally offer ‘consume what you need, when you need it’ models. The nature of cloud infrastructure makes it easy to spin up and spin down services on demand, allowing for batch and burst process expansions without specific infrastructure investment. It should be noted, however, that short term usage can come at a higher price.
Scalability: Many companies, from the smallest all the way to large enterprises, often struggle to scale quickly. Imagine needing to purchase hundreds of servers for a large batch job, or petabytes of additional storage for an event-based activity. The cloud makes this level of scaling simple and easy, with predictable cost models around that scaling.
Security: Most high-profile cloud based services such as Microsoft, Google and Amazon have made significant investments in security of data stored within the cloud, even ensuring cross-industry certification for their services. This guarantees their customers comfort levels they would struggle to enjoy with their own on-premise deployments.
Collaboration: As more companies move data and services into the cloud collaboration becomes more rapid. For example when a company wants to deliver terabytes of data to another (for example a content owner to an OTT platform), this can be done by simply providing access to a cloud storage bucket with shared access for both companies. There’s no on-premise to cloud, no premise-to-premise transfers with large amounts of bandwidth, and no human resource administration or supervision required.
Considerations when moving content into the cloud
As with the adoption of any new technology, you need to be aware of the main service features. These dictate just how reliable the service might be, the performance levels to be expected, and speed of access to content. In the same way that you’d include these considerations into planning on premise deployments, you need to plan these for whichever cloud service you might choose. In many cases, cloud providers have gone out of their way to provide clear indications for all of these features.
Reliability: Cloud providers often provide tiering options to customers for differing levels of reliability. For example, it’s common for cloud vendors to offer highly available object storage replicated across regions for those requiring maximum redundancy, with lower cost tiers providing reduced levels of redundancy, which can be useful when fewer copies of the objects are created, where cost is more of a focus, and content maybe more transient.
Performance: Similarly to reliability, performance generally varies with the various cloud storage tiers offered by cloud providers. For example high performance attached storage might be more suitable for content processing, whereas lower performance, long term storage (in which content is typically moved off onto data tape to lower costs) might be a more appropriate location for archive material; in the case of Amazon for example this is called S3 Glacier Deep Archive.
Access times: Cloud tiers don’t only provide differing levels of reliability and performance. Access times is another important consideration that varies quite widely across different storage tiers. Object storage services capable of holding huge amounts of data are actually highly available disk arrays, and therefore access times are instant. However services such as Amazon’s Glacier Deep Archive (whose storage media systems are more likely to be tape based) can come with an access time of up to 12 hours.
Choosing the right storage tier(s) for your content
With the wide array of performance, cost, egress and other considerations available, choosing the right combination of storage for your content is a crucial decision that requires careful planning. Without it, you can find yourself storing large amounts of content far more expensively than you’d anticipated, or without rapid access to mission-critical content.
Here’s a brief cost comparison of two commonly utilised storage services from Amazon (from https://calculator.aws/#/createCalculator, standard pricing, East Ohio):
|Amazon S3 Standard||22,067||Instant|
|S3 Glacier Deep Archive||1,013||3-12 hours|
Therefore choosing the correct storage services / tiers for your business, and for the different types of content you process, is an important decision. You’ll have to pay close attention to how access times, and the time taken to move content between the tiers, will suit your business needs.
Many organisations are using more than one cloud vendor service for their own business platform, or for interaction with other businesses. Going back to the earlier example of a studio and OTT shared platform: the studio might well be using Microsoft cloud storage services, and the OTT Amazon storage services. Cloud interoperability between the cloud vendors, accessing that content and being able to manage storage services across those cloud vendors is another challenge that businesses may face.
The right content for the right tier
Costs vary significantly between the services / tiers, so it is important to store the right content within the right cloud service or storage tier for your needs.
You might only need a small amount of content to be instantly available (and therefore stored most expensively); for example low bitrate proxies of larger high resolution, high bitrate video based content might be suitable for low latency storage with high availability (such as Amazon S3 Standard) that can also take advantage of publication to services such as Amazon’s CDN CloudFront for example. Larger, higher resolution content would be more appropriately stored in longer term storage such as the previously mentioned Amazon Glacier Deep Archive, which offers lower cost but significantly longer times to access the content.
The role of storage management
The benefits of different storage tiers, across multiple cloud vendors, means that the optimal storage ecosystem may have different types of content stored in many storage locations (in a hybrid environment, this also includes local disk and tape systems). Each of these typically requires its own front-end management application, dashboards etc, and none can provide visibility outside of its own storage. You could potentially end up having to administer multiple systems with individual applications, and manage separate access points from the applications that require access to your content.
It is important to have seamless access to all of your assets under storage from all of your applications; adding storage locations (including different cloud storage tiers) should be easy, and indexing the content therein automatic, and shareable to all of your systems. This is where storage management applications, such as Kumulate, become a critical element, managing all storage platforms and tiers in a single application, and using automated lifecycle operations to ensure that content is always stored in the most appropriate and most cost-efficient tier.
Automatic life cycle management
You need to ensure that you obtain the most value from your content. Key to this is ensuring that it’s always stored as cost-effectively as possible. Automated storage management moves and replicates content across different tiers of storage, based on attributes that you can specify, automatically and as a completely background process. For example, content that is current and used regularly over a short time can’t initially be stored somewhere with long egress times. As this content ages and is accessed less frequently, rapid access times become less of a requirement, and it can be moved through successively cheaper storage tiers. The storage management system you choose should manage these processes seamlessly and without requiring user initiation.
Kumulate provides the ability to access not only cloud storage services, but also on premise services such as tape and disk.
Kumulate provides life cycle management for storage of all types, ensuring costs are minimised and access times are applicable to business requirements.
Kumulate provides a single name space access to all content under management as well as the ability to search and browse all content management by the platform
For our calculations and examples, we used AWS services. However, almost all of the other cloud providers offer the same calculators – when you’re looking at costs it’s very important to use these tools to build out comparisons. Here are another couple, for MS Azure and Wasabi: