I am currently looking for solutions that can optimize network bandwidth usage.
Scenario: Server has a file. Client downloads it through a REST API. Client makes some changes and uploads the changed file back to server through REST. Server will replace the original file with the uploaded file.
Possible approaches I have two possible approaches in mind.
1- Local Diff Before making any changes client will make a copy of original file. After making the changes client will use algorithms like BSDiff or XDelta to extract changes by comparing original and changed files. These changes will be then sent to the server. Server will apply the diff on original file.
2- Use R-Sync Make a REST call to server and ask for initial Rolling Checksum and MD5 Hash. Then based on response generate the diff and POST it to the server. Server will the merge the changes.
I did some rough testing and found out that BSDiff is the most efficient solution (diff size wise - which is the primary optimization target). It generates the smallest possible diff BUT it takes huge memory which makes it impossible to use at client side for large file sizes. On the other hand results of X-Delta and rest of the binary diff tools I tried aren't that great in terms of generated diff size. Local diff also has disadvantage of using extra disk space because of keeping the copy of original file. This can be a problem in case of large files.
Memory issue of BSDiff makes R-Sync the most suitable choice (because rest of the tools are not that efficient in finding the diff). So I have decided to go for R-Sync.
R-Sync works in two steps. First it gets the signatures based on file and then data is sent back based on the signatures sent earlier. I am planning to further optimize the R-Sync by keeping the signatures of original file at client side before making any changes in it. This will remove the need of client requesting server to compute and send signatures at time of upload. Client can just send data based on already computed signatures to server whenever client wants to upload the file.
Question I know this is a bit of odd question that's why I asked a question before actually asking it here. I just would like to know if there are any better alternatives out there to solve this kind of problem? I want to confirm my approach just to make sure that I am on the right path and not missing anything major.