I had an idea to backup my work files so I started looking around for a dropbox/onedrive/googledrive type of solution. None of them had a free account with enough storage. Then I found letsupload.org with FREE UNLIMITED storage. Who knows how long this will last but may as well take advantage. Their only restrictions are 15GB max upload size and if a file has no activity (downloads) in 60 days it will be deleted. Both restrictions are just fine for daily backup. Shoot, I only keep 5 days worth of hmailserver backup, plus a few more days worth on an external drive. 60 days is a YUUUUUGE improvement over that. And its offsite! And its FREE!
For security, 7-zip offers AES 256 encryption with file header encryption. Its basically unbreakable except by the NSA, and I'm not too concerned about their interest in my email server or work files.
Letsupload has an API, for which I scripted in powershell. I coudn't figure out how to do multipart form with curl, which I think would handle the upload part a lot better than powershell. The issue with powershell is that the file needs to be encoded in order to be uploaded via invoke-restmethod, and the encoding must reside in memory ONLY! Therefore, there is a practical limit to file size. According to something I read on github:
https://github.com/PowerShell/PowerShell/issues/4129
When I tried 500mb files, the first would work, but then it would crash/reboot my server on the second one. That's a hard crash. However, 200mb files worked fine. I uploaded fourteen 200mb files to letsupload.org without issue. I watched the memory consumption in task manager when it was encoding the 200mb files and the memory was nearing 2GB for powershell process. In any case, 200mb archive parts worked without crashing.For a 64-bit program, the memory limit for a single .NET object is 2GB, unless you enable gcAllowVeryLargeObjects in the app.config file as follows:
<configuration>
<runtime>
<gcAllowVeryLargeObjects enabled="true" />
</runtime>
</configuration>
For a 32-bit program, the memory limit for a single .NET object is 512MB.
For this to work well as offsite backup, you need to archive your backup with high encryption in parts no greater than 200mb each. 7-zip works well for this. First, install 7-zip and add the program dir to the system path. Here's the command for multi-part archives of 200mb with aes-256 encryption on the archive and file headers:
Code: Select all
7z a -v200m -t7z -m0=lzma2 -mx=9 -mfb=64 -md=32m -ms=on -mhe=on -pSuperSecretPassword C:\Path\To\7-zip\Archive\hMailServer-Backup-YYYY-MM-DD.7z C:\Path\To\Source\File\YourFile.zip
Ultimately, I want to incorporate this into Jimi's backup script and make it part of my nightly backup. Encryption notwithstanding, this may not be legal in Europe due to privacy laws. I don't know. Something to look into.
If anyone knows more about curl than me, I'm pretty sure this can be greatly improved because *I think* curl will obviate the 2GB memory issue. Any help would be greatly appreciated.
Here's the script (proof of concept right now):
Powershell:
Code: Select all
<#
7-zip command for maximum encryption (AES-256)
7z a -v200m -t7z -m0=lzma2 -mx=9 -mfb=64 -md=32m -ms=on -mhe=on -pSuperSecretPassword C:\Path\To\7-zip\Archive\hMailServer-Backup-YYYY-MM-DD.7z C:\Path\To\Source\File\YourFile.zip
7z a -v200m -t7z -m0=lzma2 -mx=9 -mfb=64 -md=32m -ms=on -mhe=on -pSuperSecretPassword C:\Path\To\7-zip\Archive\hMailServer-Backup-YYYY-MM-DD.7z C:\Path\To\Source\Files\In\Dir\
#>
<### USER VARIABLES ###>
$APIKey1 = "ASixtyFourCharacterStringAPIKeyNumberOne"
$APIKey2 = "ASixtyFourCharacterStringAPIKeyNumberTwo"
$BackupFolder = "C:\HMS-BACKUP\test"
<# Begin Script #>
<# Clear out error variable #>
$Error.Clear()
<# Authorize and get access token #>
$AuthBody = @{
'key1' = $APIKey1;
'key2' = $APIKey2;
}
$URIAuth = "https://letsupload.org/api/v2/authorize"
$Auth = Invoke-RestMethod -Method GET $URIAuth -Body $AuthBody -ContentType 'application/json; charset=utf-8'
$AccessToken = $Auth.data.access_token
$AccountID = $Auth.data.account_id
<# Create Folder #>
$URICF = "https://letsupload.org/api/v2/folder/create"
$FolderName = "Backup-EM-$((Get-Date).ToString('yyyy-MM-dd'))"
$CFBody = @{
'access_token' = $AccessToken;
'account_id' = $AccountID;
'folder_name' = $FolderName;
'is_public' = 0; # 0=private, 1=unlisted, 2=public
}
Write-Host "Creating Folder"
$CreateFolder = Invoke-RestMethod -Method GET $URICF -Body $CFBody -ContentType 'application/json; charset=utf-8'
$CreateFolder.response
$FolderID = $CreateFolder.data.id
Write-Host "Folder ID: $FolderID"
<# Upload File #>
$Count = (Get-ChildItem $BackupFolder).Count
Write-Host "There are $Count files to upload"
$N = 1
Get-ChildItem $BackupFolder | ForEach {
$FileName = $_.Name;
$FilePath = "$BackupFolder\$FileName";
$UploadURI = "https://letsupload.org/api/v2/file/upload";
Write-Host "----------------------------"
Write-Host "Encoding $FileName - $N of $Count"
$FileBytes = [System.IO.File]::ReadAllBytes($FilePath);
$FileEnc = [System.Text.Encoding]::GetEncoding('UTF-8').GetString($FileBytes);
$Boundary = [System.Guid]::NewGuid().ToString();
$LF = "`r`n";
$BodyLines = (
"--$Boundary",
"Content-Disposition: form-data; name=`"access_token`"",
'',
$AccessToken,
"--$Boundary",
"Content-Disposition: form-data; name=`"account_id`"",
'',
$AccountID,
"--$Boundary",
"Content-Disposition: form-data; name=`"folder_id`"",
'',
$FolderID,
"--$Boundary",
"Content-Disposition: form-data; name=`"upload_file`"; filename=`"$FileName`"",
"Content-Type: application/json",
'',
$FileEnc,
"--$Boundary--"
) -join $LF
Write-Host "Uploading $FileName - $N of $Count"
$Upload = Invoke-RestMethod -Uri $UploadURI -Method POST -ContentType "multipart/form-data; boundary=`"$boundary`"" -Body $BodyLines
$UResponse = $Upload.response
$UURL = $Upload.data.url
$USize = $Upload.data.size
$UStatus = $Upload._status
Write-Host "Response: $UResponse"
Write-Host "URL : $UURL"
Write-Host "Size : $USize"
Write-Host "Status : $UStatus"
$N++
}
