When backing up large amounts of data with Veeam, one often encounters the efficient and cost-saving solution of S3 Object ag, but its setup and use also present challenges. Choosing the right block size and handling the high number of requests can be crucial to avoiding performance issues and maximizing data backup efficiency.
In connection with our Object we repeatedly encounter Backup's store is ideally suited Object storeagfor the cost-effective external storageaglarge amounts of Backupdata. Setup is also relatively simple, requiring only the specification of the "Endpoint," "Access Key," and "Private Key"—but this is where a stumbling block lies, one that has already brought even the largest S3 providers to their knees. Veeam solution. An S3
Standard blocksize by Veeam
By default, Veeam uses a block size of 512 KB for archive data and 1024 KB for Backupdata whenagdata. This value defines how large the individual data pieces are when loaded as objects onto the Object ag. Smaller block sizes can help generate a higher compression rate for small source data and thus save storage costs.
The numerous small objects, which can quickly number in the millions depending on the Backupsize, are managed by Veeam's Object agafter being uploaded. Due to the nature of Backupand their expiration dates, objects that are no longer needed are removed, while new objects are continuously uploaded. This results in thousands of GET, PUT, and DELETE requests.
Deletion operations, in particular, are very resource-intensive in Object-agenvironments, and so it can happen in larger environments that the databases behind the Object storageagno longer keep up with the cleanup jobs quickly enough, and thus the tables become increasingly full.
Limitations to regulation
To prevent Veeam environment backups from impacting Object ag, vendors have implemented various tools and limitations. Amazon S3, for example, has introduced a limit of 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second.
Best practice for Netstream Object Storage
Object solutionagis Netstream storage based on advanced Cloudsoftware. The latest version, which we use, introduces an innovative protocol that enables dynamic coordination of requestsagsizes between the storageagbackup Backup. This improves overall performance through automated reconciliation processes. However, this feature is not yet compatible with all client systems, making manual configuration of the settings a sensible approach to achieve optimal results.
More information about the configuration can be found in our knowledge base.
Recommendation
The guideline still recommends that Veeam's block size should not fall below 1 MB, as this can lead to performance issues. Therefore, a minimum block size of 4 MB is recommended.
In short, the following settings are important:
- Backup Job –agOptimization Settings should be set to “4 MB” or “8 MB” (the latter can be activated via Registry Key)
- The immutability retention period should be less than or equal to the Backup retention period
- S3 Repository Task Limit: 4 or 8
- SOBR configuration: A new bucket should be created for every 100 VMs or 200TB of data
