If you have 1 million files (an average size backup) then this is 20 million bytes which is only 20 MBytes - you'll never notice it. There are moments when the Backblaze client has to keep a SHA-1 checksum (a fixed 20 bytes) in RAM for every file that is stored as part of the backup safely on the Backblaze datacenter side, so that it can quickly look up if the CURRENT file matches it. These are areas like the analysis of what files need to be deleted from the Backblaze datacenter, and the de-duplication steps. Now, there are a couple of OTHER areas unrelated to transmission itself that can cause higher RAM usage in a small subset of customer's backups and is related more to the number of files the customer has (it is entirely unrelated to the file sizes themselves, simply the sheer number of files). Also, the customer controls this, if they set it to 1 thread it will only use about 100 MBytes (plus a few data structures so maybe 150 MBytes) even at the very maximum, and in practice less.Ī side note is if the file is more than 100 MBytes it goes down an entirely different code path of breaking it into 10 MByte 'chunks' and then some of the same logic applies but just at 1/10th the size so this won't cause any usage above say 300 MBytes (30 threads all transmitting 10 MByte chunks at the same time).

In practice, this happens for a short time during the INITIAL UPLOAD but you'll rarely see it ever again like that because it isn't normal to suddenly create 30 files that are all exactly 100 MBytes. So if you have it set to 10 threads and they all line up to transmit at the same time and you have 10 files to transmit that are all up near the 100 MByte limit each, this could use 1 GByte of RAM, and if you configure it to use 30 threads it can get as high as 3 GBytes of RAM at the peak. If the file is 100 MBytes or less in size, Backblaze reads it all from disk in one swoop, compresses it (which may or may not result in any RAM space savings), then encrypts it in RAM, then holds that compressed encrypted version of the entire file in RAM during the transmission. Why is Backblaze taking so much RAM for backup? Is it keeping the transfer copy in RAM? Disclaimer: I work at Backblaze and wrote a lot of the code that uses the most RAM (wow, that sounds bad).

Backblaze m1