Amazon S3 is so common today that it is easy to forget that, at one point, it was a fairly new idea. AWS launched the S3 service in 2006and essentially made modern cloud object storage mainstream. Not a traditional file system, not a block device, but a simple approach: store data as objects, access them via API, add metadata, and that’s it.

That idea has since become a de facto standard.

Today we talk about Amazon S3, Azure Blob Storage, Google Cloud Storage, and many other object storage solutions. Technically, these are not all identical systems, but the principle is similar: data is stored as objects, accessed through APIs and HTTPS, and designed for massive scalability.

The big difference compared to the traditional on-premises world is this: in the public cloud, the service provider builds the platform for many customers, many workloads, and almost unlimited growth. As data volumes grow, the platform also grows in the background. Hardware, lifecycle management, scaling, redundancy, geo-distribution — all of this is taken into account and maintained by the cloud provider, and of course billed granularly and on a daily basis.

On-premises, you usually serve exactly one customer: yourself. And growth usually does not happen “on demand” within a few minutes. New hardware, additional licenses, rack space, network ports, power, cooling, and operational processes must be planned, budgeted, and implemented.

S3 Is Not Always S3

Amazon S3 has established itself as the market leader. Not only as a product, but above all as an API model. Many vendors therefore align with it and offer their own S3-based or S3-compatible solutions.

And this is where the first trap begins.

Because in practice, there can be worlds of difference between:

• S3-compatible
• S3-like
• S3 API compatible
• 100% S3-compatible
• supports S3 API

Many solutions support the basic S3 operations from AWS: writing, reading, and deleting objects, as well as managing buckets. For simple workloads, that is often enough.But this does not automatically mean that all relevant S3 API functions are supported, or that the implementation behaves as expectedunder high load, with many objects, with Object Lock, or during massive delete operations.

As soon as backup workloads, immutability, retention, very high object counts, parallel streams, and many delete operations come into play, you quickly notice whether a platform was really built for this — or whether it merely “speaks S3” but is not optimally suited for the specific backup workload.

Public Cloud Scales Differently Than On-Premises

In the public cloud, object storage is a service. The provider operates huge platforms distributed across regions, availability zones, and global infrastructures. The customer consumes capacity and performance as a service.

On-premises, you have to make these decisions yourself:

• How many nodes do I need? 
• How many disks? 
• HDD, SSD, or all-flash? 
• How does the metadata layer scale? 
• How many objects can the system manage with good performance? 
• How does the platform behave with retention and delete operations? 
• What happens in the event of single-disk, multi-disk, or node failure?
• What happens if an entire site, fire compartment, server room, or failure domain fails? 
• Can I easily migrate or mirror buckets? 
• Should I define or observe certain limits per bucket, site, node, API, or metadata layer? 
• What does the upgrade and lifecycle model look like? 
• Can I represent multiple sites? 
• Does the solution support my security and immutability requirements?

This may sound trivial, but it is not. With object storage, people often think about capacity first — terabytes or petabytes. In practice, however, other things are often more critical: object count, metadata, API performance, and delete behavior.

Especially with backup workloads, this is often underestimated.

Backup Is Not a Normal S3 Workload

When object storage is used for application data, archives, data lakes, or media objects, the workload often consists of large objects, predictable access patterns, and different lifecycle behavior.

However, backup workloads behave differently.

A modern backup solution does not simply write “one large file” into a bucket and call it done. Depending on architecture, block size, job design, retention, and repository type, a very large number of objects can be created very quickly. Backup data then no longer consists of “a few files,” but of millions or even billions of objects.

And these objects must be managed:

• listed per bucket
• written
• read
• versioned 
• protected against modification 
• deleted after retention 
• found in catalogs
• quickly reassembled during restore 
• checked during health 
• efficiently moved during offload or copy jobs

This is a completely different workload from “we are storing a few large archive files.” As mentioned, we are not dealing with a small number of large files here, but with millions or even billions of small objects.

That is why it is not enough to ask:
How much capacity does my object storage have?

The better question is:
How well does my object storage handle my specific backup behavior — especially the significant number of small objects?

Retention Means Not Just Keeping Data, but Also Deleting It

One point that is often considered too late in designs: retention does not only create stored data. Retention also regularly creates delete operations — and on a very LARGE scale.

When backups age out of the retention period, objects must be deleted. Depending on the number of restore points, object size, backup chain, bucket design, and immutability model, this can mean a very large number of API requests.

And then you suddenly realize whether the platform was built for it.

A backup run can work without issues as long as data is only being written. The real problems sometimes appear later:

• when retention starts applying
• when millions of objects need to be deleted 
• when new backups are being written at the same time 
• when immutability expires 
• when lifecycle processes become active 
• when multiple jobs run in parallel 
• when a restore must happen under load

Then “it works in the test lab” can quickly turn into “why is my backup window suddenly twice as long — or why does it even fail?” — and the S3 platform can no longer keep up.

Especially with S3-compatible on-premises solutions, you should therefore not just run a simple write test. You should test the complete lifecycle: write, read, delete, retention, restore, parallelism, and failure scenarios. 

You should also test with realistic test dataand access patterns wherever possible — in other words, in the same way the later backup solution will actually create, modify, protect, and delete data. This includes data volume, change rate, object size, object count, parallelism, and retention behavior.

All-Flash or Tiering? The Answer Is: It Depends

Of course, it would be nice to simply deploy a large all-flash S3 platform and solve the performance topic that way. In some environments, that is possible and exactly the right approach.

But usually there is a budget. And usually there is a lot of backup data. So you quickly end up with hybrid architectures or tiering models:

• metadata on flash
• user data on HDD 
• cache or performance tier on SSD 
• capacity tier on spinning disks 
• possibly cloud tiering for older data 
• possibly mirroring into another availability zone

For backup, it is especially important that metadata management is fast and sized large enough. Because when millions or billions of objects are involved, raw capacity is not the only relevant factor. The platform must also be able to efficiently find, manage, and delete these objects.

A system with a lot of HDD capacity but a weak metadata layer can quickly become a bottleneck in day-to-day backup operations.

That is why you should look very closely at tiering architectures:

• Where is the metadata stored?
• How large is the metadata area? 
• What happens when this area is full or heavily utilized? 
• How does the metadata layer scale? 
• Are there dedicated performance or metadata nodes? 
• How do deletes behave? 
• How many parallel S3 operations are realistically possible? 
• How does the system behave with a lot of small objects? 
• Should backup data be distributed across multiple buckets? 
• If yes, what should be the maximum size of a bucket in terms of GB and object count?

Capacity can often be purchased later — provided budget, rack space, power, cooling, ports, and licenses have been planned. Consistent performance over years is much harder. 

Important: Major design and configuration changes in object storage during production operations can be difficult, time-consuming, and may require additional hardware.

How Is the Data Actually Stored in Object Storage?

One topic that is also often discussed too late in S3 on-premises designs is the actual data placement within the platform.

Object storage is not simply “one big hard drive with an S3 API.” Under the hood, the solution must decide how it protects, distributes, and restores data in the event of failures.

From the traditional storage world, we know terms such as:

• RAID
• mirroring 
• replication 
• erasure coding 
• failure domains 
• node redundancy 
• site redundancy

With object storage, it is particularly important to understand at which level protection takes place.

Some solutions use RAID or local disk groups within individual nodes. Others rely more heavily on distributed protection across multiple nodes. Others combine local protection mechanisms with erasure coding or replication at the cluster level.

RAID protects data only within a server or disk group. For a modern S3 platform, that alone is not enough. If an entire node, rack, or even site fails, the platform must still know where the data is located, how many copies or fragments are still available, and how the data can be fully reconstructed.

Replication: Simple, Fast, but Expensive

The simplest option is replication, often referred to as a data replica.

An object is stored multiple times — for example, two or three times. This is easy to understand, often performant because there is little computational overhead, and relatively easy to restore in the event of a failure.

The disadvantage is obvious: you pay for this simplicity with capacity.

With triple replication, 100 TB of usable data can quickly require 300 TB of raw capacity — plus reserve, metadata, overhead, and growth. For smaller environments or particularly performance-sensitive workloads, this can still make sense. But with large backup data volumes, it quickly becomes expensive. So with a backup workload you should use maximum Replica 2 and think about a mirror to a different availability zone.

Replication can also be interesting across sites. In that case, however, not only capacity is relevant, but also bandwidth, latency, and whether replication is synchronous or asynchronous.

Erasure Coding: More Efficient, but Not Free

Many object storage solutions use erasure coding for reasons of capacity efficiency and fault tolerance.

In simplified terms, an object is not stored multiple times in full, but is split into data and parity fragments. The object can later be reconstructed from a certain number of these fragments.

This is significantly more efficient than traditional replication. Instead of triple storage, you can work with much lower overhead depending on the method and design.

But erasure coding is not “magical capacity savings without side effects.”

There are trade-offs:

• Write operations can be more complex.
• Small objects, usually <= 1 MB, are significantly less favorable than large objects.
• Rebuild or healing processes create additional load.
• Degraded reads can be slower.
• Distribution across nodes, racks, or sites must be planned properly.
• Depending on the encoding scheme, a minimum number of nodes or disks is required.
• Not every erasure-coding design is well suited for backup workloads.

This is especially important for a backup service. If a backup solution creates many small objects, then not only capacity and write throughput matter, but also fragmentation, metadata handling, and rebuild behavior.

A system may look good during normal operation, but during a rebuild, node failure, or larger retention delete cycle, it may suddenly become significantly slower.

Data Placement Is Also a Question of Failure Domains

With object storage, you should not only ask:

How many copies or fragments are there?

But rather:

Where are these copies or fragments located?

That is an important difference.

If several fragments of an object are located on the same server, the same shelf, the same rack, or the same site, redundancy may look good on paper — but not necessarily in practice.

That is why the failure-domain design should be examined closely:

• protection against disk failure
• protection against node failure
• protection against rack failure
• protection against controller or network failures
• protection against site failure
• behavior during maintenance and updates
• behavior during simultaneous failures

This is especially critical on-premises. In the public cloud, many of these architectural decisions are consumed as a service. On-premises, you have to understand, purchase, operate, and regularly test them yourself.

Immutability Does Not Protect Against Poor Storage Design

Another point: immutability is important, but it does not replace a clean data placement concept.

S3 Object Lock or other WORM mechanisms protect objects from being modified or deleted. But they do not automatically protect against poor redundancy design, insufficient reserve capacity, faulty rebuild processes, or an overloaded metadata layer.

An immutable backup is only truly valuable if the underlying platform can also provide the data permanently, consistently, and with good performance.

Or put another way:

Immutable does not automatically mean invulnerable.

Object Size: Backup Software and Storage Must Fit Together

Another point is storage granularity, or object size.

Depending on data type, configuration, and repository type, backup software can create different object sizes. Larger objects are easier for most on-premises object storage solutions to handle because they create less metadata and fewer API operations. At the same time, larger blocks can influence deduplication, storage consumption, restore behavior, or network behavior.

Smaller objects can have advantages from the backup software perspective (they can be compressed more efficiently), but they create a lot more management overhead in the object storage system.

There is no universally correct answer here.

What matters is this:

Backup software and object storage must be designed together.

It is not very useful to select an S3 platform in isolation and then hope that the backup workload will somehow fit.

This is exactly where generic object storage solutions differ from solutions that were specifically built or optimized for backup workloads.

One example in this area is Object First Ootbi, an object storage solution positioned specifically for Veeam backup workloads. That does not automatically mean it is the right choice for every scenario — but it illustrates the important point: backup is its own workload and should be treated as such.

A Bucket Is Not a Design

When it comes to S3, people often ask: “How many buckets do we need?”

Unfortunately, the answer is rarely: “One.”

Bucket design depends heavily on the workload:

• Which backup jobs write to which bucket?
• Are there separate tenants or departments?
• Do different retention policies need to apply?
• Are there different immutability periods?
• Are multiple sites used?
• Are there separate performance requirements?
• Does future migration or deletion need to be easier?
• Are there vendor limits per bucket? Or with the bucket numbers?
• What does monitoring and alerting per bucket look like?

A single large bucket may seem convenient at first. Later, however, it can become confusing or suffer significant performance degradation when billions of objects have been stored and need to be listed, read, or deleted. It becomes especially problematic when different workloads, retention requirements, and security concepts are mixed together.

Conversely, “one bucket per job” is not automatically correct either. Too many buckets can also make management, monitoring, and scaling more difficult.

Here too, the bucket design should not happen by accident, but should be part of the overall design.

Immutability Is Mandatory, but Not Trivial

Ransomware has changed backup designs. In the past, the question was often: “How fast can we back up and restore?” Today, at least as important is: “Can someone delete or manipulate the backups?”

That is why immutability has become a central feature.

In the S3 world, S3 Object Lock is especially relevant here. It allows objects to be made immutable for a defined period of time. For backup solutions, this is extremely important because an attacker does not only want to encrypt production data — they also want to delete or modify backups.

But here too, you need to examine the details carefully:

• Does the solution support Object Lock?
• Is Object Lock required and supported in Governance or Compliance mode?
• How is retention implemented?
• Can an administrator shorten or bypass retention?
• What can be done if retention was configured incorrectly?
• How is it prevented that a compromised admin account manipulates buckets, policies, or credentials?
• Are there protections against accidental or malicious bucket deletion?
• How does the system behave when immutability expires?
• Can the backup software set immutability correctly?
• Are there vendor-specific considerations?
• Does it still work after updates and data migrations?
• How is administrative separation implemented?
• Is there a default, minimum, or maximum retention?

Load Balancing, Network, and Site Design

Object storage is often viewed as “one endpoint.” In reality, however, there is a distributed platform behind it.

If many backup proxies, repository servers, or gateways write data in parallel, access to the object storage must scale accordingly.

Important questions include:

• Are there one or multiple S3 endpoints?
• How is load balancing implemented?
• Is the load balancer itself redundant?
• Does the solution support multiple sites?
• Is replication synchronous or asynchronous?
• What happens in the event of site failure?
• How is DNS designed?
• Are there separate networks for backup, management, and replication?
• What does the bandwidth between sites look like?
• What happens during restore across site boundaries?

For Backup, the Normal State Is Not Enough

Many storage designs are evaluated in the normal state:

• How fast can I write?
• How much capacity do I get?
• How high is the usable storage ratio?
• What is the price per TB?

For backup solutions, that is not enough.

You also have to consider degraded operations:

• What happens during a rebuild?
• What happens when a node fails?
• What happens when backups are running at the same time?
• What happens when retention creates deletes?
• What happens during a restore in a degraded state?
• How long does self-healing take?
• How much reserve capacity is required for healing?
• Can the system still store new immutable objects redundantly and quickly during that time?

Especially with erasure coding, the platform may need to perform significantly more internal work during a failure. That is not bad — it is part of the design. But it must be planned for.

For backup, this means:

Do not only look at the capacity in the data sheet. Test the behavior under failure conditions.

One Solution for Everything — or Multiple Solutions?

Many companies want one central object storage platform for everything:

• backup
• archive
• application data
• logs
• analytics
• Kubernetes
• data lake
• file-to-object
• compliance data

That can make sense. But only if the platform can truly serve these workloads.

The requirements can be very different:

One platform for everything sounds attractive. But if the backup workload suffers, or if other workloads are slowed down by backup jobs, nobody benefits.

Sometimes it makes more sense to use multiple object storage solutions:

• an optimized solution for backup
• a cost-efficient platform for archive
• a performant platform for analytics or applications

This is not always cheaper. But operationally, it can be much cleaner.

What I Would Look for in an On-Premises S3 Design

If I had to evaluate an on-premises S3 solution for backup or mixed workloads, I would check at least the following points:

1. S3 Compatibility
   • Which APIs are supported? 
   • Is there 100% compatibility, or are API integrations missing? If yes, which ones? 
   • If there are additional API commands that are only used with this S3 solution, how do they work, what do they enable, and what do they restrict? 
   • Are there known limitations or specific designs I should follow with the backup software being used?
2. Object Lock / Immutability
   • Is S3 Object Lock supported? 
   • Is Governance and/or Compliance mode supported? 
   • How does the delete process work in Governance mode? 
   • Has the implementation been validated and supported with the backup solution and the intended backup workload size and object count?
3. Object Count
   • How many objects are realistically supported per cluster, node, and bucket? 
   • How can the solution be expanded? 
   • Are there limits per bucket, node, tenant or cluster?
4. Metadata Performance
   • Where is metadata stored? 
   • How does the metadata layer scale? 
   • What happens with millions or billions of small objects, 
     for example <= 1 MB?
5. Delete Performance
   • How does the system behave during retention deletes?
   • Are there real-world figures for many parallel deletes with small objects, for example <= 1 MB?
6. Write- and Restore Performance
   • Do not test only backup, but also restore.
   • Ideally, test under load.
7. Scaling
   • Scale-up or scale-out?
   • How expensive is expansion?
   • How complex is expansion during operations?
8. Data Placement and Redundancy Model
   • How does the platform protect data internally? 
   • At which level does protection take place: disk, node, rack, cluster, or site? 
   • How many simultaneous failures can the system tolerate? 
   • What is the true usable capacity after redundancy, metadata, and reserve? 
   • How does the platform behave during rebuild, healing, and degraded states? 
   • What impact does the redundancy model have on backup, delete, and restore performance? 
   • Are failure domains properly planned? 
   • Does the model fit the specific backup workload?
9. Hardware Architecture
   • All-flash, hybrid, or HDD?
   • Are there separate tiers?
   • How is cache used?
10. Site and Failure Scenarios
    • Single-site or multi-site? 
    • Replication? 
    • Behavior during node, disk, or site failure?
11. Operating Model
    • Updates 
    • Monitoring 
    • Support 
    • Hardware replacement 
    • Lifecycle 
    • Integration into existing processes

Conclusion: S3 Is a Standard, but Design Remains Manual Work

S3 has established itself as a standard. There is almost no way around it anymore. Especially in backup environments, object storage has become a central building block — not least because of immutability, scalability, and good integration into modern backup architectures.

But:

S3 is not magical storage that solves every design problem.

On-premises, you have to look more closely than in the public cloud. You are not simply buying “infinitely scalable object storage,” but a specific platform with specific hardware, specific limits, and specific behavior under load.

For backup, this means:

• Object count matters.
• Metadata matters.
• Deletes matter.
• Immutability matters.
• Restore matters.
• Bucket design matters.
• The specific workload matters more than the data sheet.

And above all:

Just because a solution is S3-compatible does not automatically mean it is a good backup target.

The market is changing quickly. Hardware is getting faster, software is gaining new features, vendors are improving their S3 implementations, and backup products are evolving as well. That is why every design should be reviewed regularly.

In the end, it is not about finding the “best” S3 solution on paper.

It is about building the right solution for your own workload.

And especially with backup, it is better to over-test than to under-test.

So what is your preferred S3 Onprem Storage, what did you change and why?

Sources / Further Reading

• AWS: Amazon S3 20 Years
  https://aws.amazon.com/blogs/aws/twenty-years-of-amazon-s3-and-building-whats-next/
  https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html
• Google Cloud Object Storage
  https://cloud.google.com/storage
• Azure Blob Introduction
  https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction
• Veeam – S3 Limitations
  https://helpcenter.veeam.com/docs/vbr/userguide/s3_compatible_limitations.html
• Veeam: Immutable Backups
  https://www.veeam.com/blog/immutable-backup.html
• Object First: Ootbi
  https://objectfirst.com/ootbi/