When backing up large volumes of data with Veeam, an S3 Object Storage is often the efficient, cost-effective choice, but setting it up and using it also comes with challenges. Choosing the right block size and handling the high number of requests can be decisive in avoiding performance problems and getting the most out of your backups.
In connection with our object store, we regularly come across the backup solution from Veeam. An S3 Object Storage is ideal for offloading large amounts of backup data cost-effectively. Setting it up is also relatively simple – you just enter the «Endpoint», «Access Key» and «Private Key» – but this is exactly where a pitfall lurks that has brought even the largest S3 providers to their knees.
Veeam’s default block size
By default, Veeam uses a «block size» of 512 KB for archive data and 1024 KB for backup data when offloading. This value defines how large the individual chunks are that get uploaded to the Object Storage as objects. Smaller chunks can help achieve a higher compression rate for small source data and thus save storage costs.
After being uploaded to the Object Storage, the many small objects – which can quickly run into the millions depending on the backup size – are managed by Veeam. Due to the nature of backups and their expiry dates, objects that are no longer needed are removed while new objects are continuously uploaded. This results in thousands of GET, PUT and DELETE requests.
Delete operations in particular are very resource-intensive in Object Storage environments, so in larger environments the databases behind the Object Storage may no longer be able to keep up with the clean-up jobs, causing the tables to fill up more and more.
Limits as a safeguard
To prevent Veeam backups from impairing the Object Storage, vendors have implemented various tools and limits. Amazon S3, for example, has introduced a limit of 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second.
Best practice for Netstream Object Storage
Netstream’s Object Storage solution is built on advanced Cloudian software. The latest version, which we use, introduced an innovative protocol that allows requests and block sizes to be negotiated dynamically between the storage and the backup software. This improves overall performance through automated tuning. However, this feature is not yet compatible with all client systems, which makes configuring the settings manually a sensible step to achieve optimal results.
You can find more information on the configuration in our knowledge base.
Recommendation
As a general rule, the guideline still recommends that Veeam’s block size should not fall below 1 MB, as this can otherwise lead to performance problems. A minimum block size of 4 MB is therefore recommended.
In short, the following settings are important:
- Backup job – Storage optimisation settings should be set to “4 MB” or “8 MB” (the latter can be enabled via a registry key)
- Immutability retention period should be < or = the backup retention period
- S3 repository task limit: 4 or 8
- SOBR configuration: a new bucket should be created for every 100 VMs or 200 TB of data
