Storing Data in the Cloud

Every application needs to keep something somewhere. Files, images, logs, customer records, backups, the lot. It is one of those things that sounds simple until you start asking where, exactly, that data lives, and what happens to it when a disk somewhere quietly dies.

The slightly awkward truth is that "cloud storage" is not one thing. It is several quite different services that happen to share a name. The kind of storage you want for a folder of holiday photos is not the kind you want for the disk a virtual machine boots from, and neither of those is what you want for a table of customer orders. Picking the wrong one tends not to fail loudly, it just makes everything a bit harder than it needed to be, which is worse.

So this article is mostly about telling the main kinds apart. We will look at object storage, file storage and block storage, then talk about databases at a high level, and finish with the question that sits underneath all of them: where are the copies, and how many are there.

Object storage

Object storage is the kind of cloud storage many people have used without realising it. If you have ever uploaded a photo to a social media site, saved a file in a cloud drive, or attached an image to an online listing, there is a good chance object storage was involved behind the scenes.

The idea is that you give the storage service a piece of data, which it stores as an object. That object might be an image, a video, a PDF, a backup, or an audit log.

Objects are stored inside something usually called a bucket or a container, depending on the cloud provider. Each object has a unique key, which is the label used to find it later.

A key might look like photos/holidays/beach.jpg. This looks like a normal folder path, but object storage does not necessarily contain real folders. The slashes are usually just part of the key, and the interface uses them to group objects in a familiar way.

Object storage does not work quite like the file system on your laptop. An object is normally treated as a complete item. If you change it, you usually upload a replacement rather than opening it, changing one part, and saving those changes back into the same object.

That makes object storage less suitable for data that needs to be constantly edited in place. What you get in return is enormous scale. An object storage service can hold vast numbers of objects without you having to create or manage disks in advance.

The pricing model is also usually consumption-based. You are commonly charged for how much data you store, how often you access it, and how much data you transfer. You are not normally reserving a fixed disk and paying for the entire capacity whether you use it or not.

Object storage is a strong choice for large amounts of unstructured data. This generally means data that the storage service can treat as a complete object without needing to understand the information inside it. That includes media, backups, archived data, website assets, and logs.

If the thing you are storing is essentially a complete file that you expect to retrieve or replace as a unit, object storage is usually where it belongs.

File storage

File storage is closer to what most people picture when they think about storing things, because it behaves like a shared drive.

The cloud provider creates a file system for you on storage infrastructure in its data centre. This is separate from the disks attached directly to your virtual machines. You can then connect, or mount, that file system on one or more machines.

Mounting a file share means making the remote storage appear as though it were another drive or folder on the machine. On Windows, it might appear as a drive such as Z:\. On Linux, it might appear as a folder such as /mnt/shared. Applications can then open and save files there using familiar paths.

The files are not copied onto every machine. They remain in one shared location, and each machine accesses that same location over the network.

Imagine an application running across three virtual machines. A user uploads annual.pdf through the first machine, which saves it to the shared file system. When the second or third machine looks in the same folder, it can see annual.pdf because all three machines are connected to the same underlying storage.

Without shared file storage, each virtual machine might save uploads to its own local disk. A file uploaded through one machine would then be missing from the others unless the application copied it between them and kept those copies synchronised.

The machines communicate with the file storage using a protocol. A protocol is simply an agreed set of rules for how two systems communicate. It defines how a machine asks to open a folder, read a file, save changes, or create a new file over the network.

Two common file storage protocols are SMB and NFS. SMB is widely associated with Windows file sharing, while NFS is commonly used by Linux and Unix systems. You do not need to understand how they work internally at this stage. The important point is that they allow a remote file system to behave like a familiar drive or folder.

Because several machines can connect to the same file share, file storage is useful when an application needs one common set of files. However, shared access does not automatically make it safe for several machines to edit the same file at exactly the same time. The application may still need a way to prevent conflicting changes.

File storage is a comfortable, familiar model, which is part of why it is popular when teams move older applications into the cloud. A lot of existing software expects to read and write files through normal-looking paths, and file storage lets that software carry on working without being completely rewritten.

It does not usually scale to the same extremes as object storage, and it can cost more per gigabyte depending on the provider and performance level. The trade-off is convenience and compatibility. You get a familiar file system that several machines can use from one shared location.

Block storage

Block storage is the least glamorous of the three and the one beginners notice least, mostly because it tends to be hiding underneath something else.

When we talked about the different types of compute, we mentioned virtual machines: servers you rent with their own operating systems. A virtual machine needs a disk to boot from and somewhere to store its files. Those disks are commonly provided using block storage.

The cloud provider gives you a block volume, which appears to the virtual machine as though a hard drive has been attached to it. A volume is simply an allocated amount of storage, such as 100 GB. The operating system can then prepare that volume, create a file system on it, and use it like a normal disk.

This is different from the shared file storage we looked at earlier. With file storage, the provider gives you a ready-made file system that machines access over the network. With block storage, the provider gives the machine something that looks like a disk, and the operating system decides how that disk is organised.

The block part refers to how the data is handled underneath. The volume is divided into many small, fixed-size sections called blocks. The system can read or change one block without replacing everything else on the volume.

You do not normally work with those blocks yourself. The operating system and applications handle them for you. The important point is that block storage is well suited to data that changes frequently in small ways, rather than being replaced as one complete object.

That makes it useful for operating system disks, application data, and databases. A running database may be updating small parts of its files constantly as records are added or changed.

A block volume is usually attached to one virtual machine at a time. It can often be detached and attached to another machine, but it is not normally used as a shared folder that several servers access together. Some cloud services support more advanced shared attachment options, but the common model is one volume attached to one machine.

A rough way to keep the three straight is that object storage holds complete objects you mostly retrieve or replace, file storage provides a shared file system for several machines, and block storage acts as the disk attached to a particular machine.

Databases, at a high level

The fourth box in that diagram, structured records you need to query, is really a category of its own. Once your data has a clear shape and the different pieces relate to one another, you usually want a database rather than a plain store of files.

Databases come in many forms, but a useful beginner-level split is between relational and NoSQL databases.

A relational database organises data into tables with defined columns. Each row represents one record, and records in different tables can be linked together.

You might have customers in one table and their orders in another. The database can connect those records so that you can ask questions such as which orders belong to a particular customer or how much that customer has spent.

Relational databases are commonly queried using SQL, a language used to store, retrieve, and change data. SQL can also join tables, which means combining related records from several tables into one result.

Relational databases are particularly good when the data has a clear structure and consistency matters. For example, if an order is recorded, you may need to be certain that the customer, payment, and stock information remain in a valid state.

NoSQL is a broad name for databases that do not rely entirely on the traditional table model. Some store document-like records, some store data as simple key-and-value pairs, and others are designed for different kinds of relationships or access patterns.

They are useful when the data does not fit neatly into fixed tables, when its shape changes frequently, or when the system needs to distribute very large amounts of data across many machines.

This is sometimes described as scaling out. Instead of making one machine more powerful, which is scaling up, you spread the work across additional machines.

NoSQL does not automatically mean giving up consistency or reliability. Different databases make different trade-offs, and some support strong consistency and transactions. The important question is not which family is better, but which model fits the shape of the data and the way the application needs to use it.

There is one more idea worth knowing because it changes how databases feel in the cloud, and that is the managed database.

You can run a database yourself on a virtual machine. In that case, you install the database software, configure it, apply updates, arrange backups, monitor it, and deal with failures.

A managed database shifts much of that work to the cloud provider. The provider runs the underlying database system and usually handles areas such as patching, backups, hardware, and routine maintenance. You still design the data, control access, write queries, choose how the database is configured, and make sure it is being used properly.

It is the same shift we keep coming back to in this series. Less of the plumbing is yours to look after, and more of your attention can go towards the part that belongs to your application.

Managed does not mean nothing is your job, mind you. It mostly moves the work rather than removing it.

Redundancy and replication

All of this assumes the storage stays where you put it, which brings us back to the awkward fact from the article on fault tolerance and disaster recovery: everything fails eventually. Disks fail, machines fail, and occasionally whole buildings have a bad day.

This is why cloud storage services rarely rely on one copy of your data in one place. Redundancy means keeping enough additional copies, or equivalent recoverable pieces, so that the failure of one disk or machine does not mean losing the data. Replication is the process of creating and maintaining those copies as the data changes.

The interesting question is how far apart the copies are.

Keeping redundant copies within one cloud location protects you from everyday hardware failures. If a disk or server stops working, the service can use another copy without the application necessarily noticing. This maps neatly onto fault tolerance: a local problem, handled locally, with the spare tyre already fitted.

Keeping copies in separate locations protects against a wider failure. Those locations might be different data centres or entire cloud regions, far enough apart that the same flood, fire, or power cut is unlikely to affect both. If one region becomes unavailable, another copy of the data may still exist elsewhere.

This is often called geographic replication, but it is not something you should simply assume is enabled. Different services offer different redundancy options, and you usually need to choose the level that matches the importance of the data.

There is also a trade-off in how the copies are kept in step. With synchronous replication, a change is confirmed only after it has been written to more than one place. This reduces the chance of losing recent data, but the extra communication can make writes take longer.

With asynchronous replication, the first copy is updated and the distant copy catches up shortly afterwards. This is often faster, particularly across long distances, but the newest changes may not have reached the second location when a failure happens.

It is also important not to confuse replication with a backup. Replication protects you when infrastructure fails, but it may also copy accidental deletions, corrupted data, or unwanted changes to every replica. A backup keeps a separate recoverable version from an earlier point in time.

That extra safety is not free, and it does not remove the need to make decisions. Distant copies, additional backups, and faster recovery options can all cost more. There are also trade-offs in how quickly another location can take over and how much recent data could be lost.

So the useful question is the same one we keep arriving at: not whether the data is safe, but how safe this particular data needs to be. The answer for a few cached thumbnails is not the answer for a company’s financial records. It is worth deciding on purpose rather than discovering the default after something has already gone wrong.

Where this leaves us

The main thing to carry forward is that cloud storage is a small set of distinct tools, not one.

Object storage is designed for complete objects such as images, backups, logs, and other large amounts of unstructured data. File storage provides a shared file system that several machines can use. Block storage behaves like a disk attached to a virtual machine. Databases are designed for data that has structure and needs to be queried, with the option of letting the provider manage much of the database system for you.

Underneath all of them sits the question of resilience. How many usable copies of the data exist, how far apart are they, and what happens when one of them becomes unavailable?

Each provider has its own product names and options for all of this, and the details are exactly the sort of thing the separate Azure and AWS series are for. For now, it is enough to recognise the categories and understand the question each one answers. Once you can tell the shapes apart, the product names are mostly labels attached to ideas you already understand.