Welcome!

SDN Journal Authors: Liz McMillan, Yeshim Deniz, Elizabeth White, Pat Romanski, TJ Randall

Related Topics: @CloudExpo, Microservices Expo, Containers Expo Blog, Cloud Security, @DXWorldExpo, SDN Journal

@CloudExpo: Blog Post

The Paradox of Ephemeral Cloud Storage | @CloudExpo [#Cloud]

The moral of the story here is simple: if you put anything beyond your base OS on ephemeral storage, you are at great risk

The very name is kind of ridiculous, don't you think? The word "ephemeral" means it can go away. It's temporary. Fleeting, even. So why would I want to depend on storing something in a medium that can disappear without warning? And why am I forced to buy more of it when all I want is more CPUs or RAM?

Welcome to the paradox of ephemeral storage from cloud computing providers.

Origins and Explanations
Ephemeral storage exists only because of how first-generation cloud providers chunk up servers. The business model is simple: they buy a physical server and try to sell as many virtual machines (VMs) as possible on top of that physical server. Since the VMs are trapped on physical machines in this approach, first-generation providers dictate cookie-cutter sizes that make that stacking game easier for themselves.

In the process, though, these providers can't do anything to improve the redundancy of the disk on the physical servers, and are thus unable to offer guarantees on its availability. Instead they tell you not to trust it. It can evaporate. "Code around it instead" is what we are told.

If I can't trust it, how come I'm forced to buy more of it when I want bigger VM dimensions in other places, seeing as I probably only need 10GB for my operating system anyway? Consider the sizing chart below from PlanForCloud:

Take a look at that largest size. Who wants a 1.6 TB cloud storage liability?

Google Compute Engine and ProfitBricks Bring Sanity
One of the great features of Google Compute Engine is its approach to ephemeral storage. Google refers to this as Scratch Storage and in many cases limits each machine to 10 GB of it. That's just enough to build a base operating system upon, and that's obviously on purpose. Kudos to them.

ProfitBricks takes this a step further by not offering ephemeral storage at all. Instead, the physical servers housing the CPU cores and the RAM are on a separate pool of resources from the disk array that provides the block storage. Good IOPS is maintained by connecting the two with an 80 Gbps InfiniBand network. In the ProfitBricks model, all storage is akin to highly-available redundant block storage.

What You Really Want Is Block Storage
One of the things that public cloud noobs have a hard time getting their heads around at first is the difference between ephemeral storage and block storage. The latter, which every IaaS vendor offers, has some level of redundancy built into it and is where data should really be stored. Below are examples of how several vendors approach that redundancy, with better resulting availability:

Vendor

Block Volume Redundancy

Max Volume Size

AWS

"multiple servers in an Availability Zone"

1 TB

Azure

Offer both locally redundant and geographically redundant

1 TB

GCE

"replicated for additional redundancy"

10 TB

ProfitBricks

Double redundant RAID 10 across two Availability Zones

16 TB

Lessons Learned
The moral of the story here is simple: if you put anything beyond your base OS on ephemeral storage, you are at great risk. That data could go away at any time. You can't depend on it, so don't use it unless you add in an additional form of redundancy at your own engineering expense. Data you care about belongs on block storage: it has built-in redundancy and improved availability, which ensure that the data you care about will be there when you need it.

More Stories By Pete Johnson

Pete Johnson is senior director of product marketing at CliQr Technologies, where he focuses on the support of applications running on OpenStack based clouds. He is interested in the long-term management of applications in public and private clouds, and avoiding vendor lock-in. Prior to joining CliQr, Pete was senior director of platform evangelism at ProfitBricks after spending 19 years with HP as a heads-down developer, technical lead and chief architect.

Comments (1)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


CloudEXPO Stories
All in Mobile is a mobile app agency that helps enterprise companies and next generation startups build the future of digital. We offer mobile development and design for smartphones, tablets and wearables. Our projects cover the latest and most innovative technologies - voice assistants, AI, AR/VR and more. We excel at solutions for sports, fintech and retail industries.
NanoVMs is the only production ready unikernel infrastructure solution on the market today. Unikernels prevent server intrusions by isolating applications to one virtual machine with no users, no shells and no way to run other programs on them. Unikernels run faster and are lighter than even docker containers.
The dream is universal: heuristic driven, global business operations without interruption so that nobody has to wake up at 4am to solve a problem. Building upon Nutanix Acropolis software defined storage, virtualization, and networking platform, Mark will demonstrate business lifecycle automation with freedom of choice and consumption models. Hybrid cloud applications and operations are controllable by the Nutanix Prism control plane with Calm automation, which can weave together the following: database as a service with Era, micro segmentation with Flow, event driven lifecycle operations with Epoch monitoring, and both financial and cloud governance with Beam. Combined together, the Nutanix Enterprise Cloud OS democratizes and accelerates every aspect of your business with simplicity, security, and scalability.
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO Silicon Valley 2019 will cover all of these tools, with the most comprehensive program and with 222 rockstar speakers throughout our industry presenting 22 Keynotes and General Sessions, 250 Breakout Sessions along 10 Tracks, as well as our signature Power Panels. Our Expo Floor will bring together the leading global 200 companies throughout the world of Cloud Computing, DevOps, IoT, Smart Cities, FinTech, Digital Transformation, and all they entail. As your enterprise creates a vision and strategy that enables you to create your own unique, long-term success, learning about all the technologies involved is essential. Companies today not only form multi-cloud and hybrid cloud architectures, but create them with built-in cognitive capabilities.
Darktrace is the world's leading AI company for cyber security. Created by mathematicians from the University of Cambridge, Darktrace's Enterprise Immune System is the first non-consumer application of machine learning to work at scale, across all network types, from physical, virtualized, and cloud, through to IoT and industrial control systems. Installed as a self-configuring cyber defense platform, Darktrace continuously learns what is ‘normal' for all devices and users, updating its understanding as the environment changes.