When backing up large amounts of data with Veeam, you often come across the efficient and cost-saving solution of S3 object storage, but setting it up and using it also poses challenges. Choosing the right block size and handling the high number of requests can be crucial to avoid performance issues and maximize backup efficiency.
In connection with our Object Store, we often come across the backup solution from Veeam. Here, S3 object storage is ideal for the cost-effective storage of a lot of backup data. The setup is also relatively simple with the specification of "Endpoint", "Access Key" and "Private Key" - but there is already a stumbling block here, which has already brought the largest S3 providers to their knees.
Standard Blocksize from Veeam
By default, Veeam uses a "block size" of 512 KB for archive data and 1024 KB for backup data. This value defines how large the individual pieces of data are loaded onto the Object Storage as objects. A smaller block size can help to generate a higher compression rate for small source data and thus save storage costs.
The many small objects, which can quickly number in the millions depending on the size of the backup, are managed by Veeam after being uploaded to the object storage. Due to the nature of backups and their expiration dates, objects that are no longer required are removed while new objects are continuously uploaded. This leads to thousands of GET, PUT and DELETE requests.
Deletion processes in particular are very resource-intensive in object storage environments and so it can happen in larger environments that the databases behind the object storage can no longer keep up with the clean-up jobs quickly enough and so the tables become increasingly full.
Limitations to regulation
To prevent the backup of Veeam environments from affecting object storage, manufacturers have implemented various tools or restrictions. Amazon S3, for example, has introduced a limit of 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second.
Best practice for Netstream Object Storage
Netstream's object storage solution is based on the advanced Cloudian software. In the latest version that we are using, an innovative protocol has been introduced that enables dynamic coordination of requests and block sizes between the storage and the backup software. This improves overall performance through automated reconciliation processes. However, this function is not yet compatible with all client systems, which makes manual configuration of the settings a sensible measure to achieve optimum results.
You can find more information on configuration in our knowledgebase.
Recommendation
According to the guideline, it is still recommended that the block size of Veeam does not fall below 1 MB, as this can otherwise lead to performance problems. A minimum block size of 4 MB is therefore recommended.
In brief, the following settings are important:
- Backup Job - Storage Optimization Settings should be set to "4 MB" or "8MB" (the latter can be activated via registry key)
- Immutability Retention Period sollte < oder = der Backup Retention Period sein
- S3 repository task limit: 4 or 8
- SOBR configuration: A new bucket should be created for every 100 VMs or 200TB of data
