Free Unlimited Offsite Backup

Use this forum if you have problems with a hMailServer script, such as hMailServer WebAdmin or code in an event handler.
Post Reply
palinka
Senior user
Senior user
Posts: 2180
Joined: 2017-09-12 17:57

Free Unlimited Offsite Backup

Post by palinka » 2020-10-19 03:31

No, this is not spam. :lol:

I had an idea to backup my work files so I started looking around for a dropbox/onedrive/googledrive type of solution. None of them had a free account with enough storage. Then I found letsupload.org with FREE UNLIMITED storage. Who knows how long this will last but may as well take advantage. Their only restrictions are 15GB max upload size and if a file has no activity (downloads) in 60 days it will be deleted. Both restrictions are just fine for daily backup. Shoot, I only keep 5 days worth of hmailserver backup, plus a few more days worth on an external drive. 60 days is a YUUUUUGE improvement over that. And its offsite! And its FREE!

For security, 7-zip offers AES 256 encryption with file header encryption. Its basically unbreakable except by the NSA, and I'm not too concerned about their interest in my email server or work files.

Letsupload has an API, for which I scripted in powershell. I coudn't figure out how to do multipart form with curl, which I think would handle the upload part a lot better than powershell. The issue with powershell is that the file needs to be encoded in order to be uploaded via invoke-restmethod, and the encoding must reside in memory ONLY! Therefore, there is a practical limit to file size. According to something I read on github:

https://github.com/PowerShell/PowerShell/issues/4129
For a 64-bit program, the memory limit for a single .NET object is 2GB, unless you enable gcAllowVeryLargeObjects in the app.config file as follows:

<configuration>
<runtime>
<gcAllowVeryLargeObjects enabled="true" />
</runtime>
</configuration>
For a 32-bit program, the memory limit for a single .NET object is 512MB.
When I tried 500mb files, the first would work, but then it would crash/reboot my server on the second one. That's a hard crash. However, 200mb files worked fine. I uploaded fourteen 200mb files to letsupload.org without issue. I watched the memory consumption in task manager when it was encoding the 200mb files and the memory was nearing 2GB for powershell process. In any case, 200mb archive parts worked without crashing.

For this to work well as offsite backup, you need to archive your backup with high encryption in parts no greater than 200mb each. 7-zip works well for this. First, install 7-zip and add the program dir to the system path. Here's the command for multi-part archives of 200mb with aes-256 encryption on the archive and file headers:

Code: Select all

7z a -v200m -t7z -m0=lzma2 -mx=9 -mfb=64 -md=32m -ms=on -mhe=on -pSuperSecretPassword C:\Path\To\7-zip\Archive\hMailServer-Backup-YYYY-MM-DD.7z C:\Path\To\Source\File\YourFile.zip
Next you need to create a free account at letsupload. After you login, go to the settings and create 2 api keys and save the keys. Then use the keys in the script user variables. The script first authenticates with the two api keys, then creates a folder to store the uploads. The folder name is backupblahblah-(today's date). The json response includes the folder ID which is used when uploading the files. Then it uploads the files in a folder. Easy peasy. :mrgreen:

Ultimately, I want to incorporate this into Jimi's backup script and make it part of my nightly backup. Encryption notwithstanding, this may not be legal in Europe due to privacy laws. I don't know. Something to look into.

If anyone knows more about curl than me, I'm pretty sure this can be greatly improved because *I think* curl will obviate the 2GB memory issue. Any help would be greatly appreciated.

Here's the script (proof of concept right now):

Powershell:

Code: Select all

<#
	7-zip command for maximum encryption (AES-256)

	7z a -v200m -t7z -m0=lzma2 -mx=9 -mfb=64 -md=32m -ms=on -mhe=on -pSuperSecretPassword C:\Path\To\7-zip\Archive\hMailServer-Backup-YYYY-MM-DD.7z C:\Path\To\Source\File\YourFile.zip

	7z a -v200m -t7z -m0=lzma2 -mx=9 -mfb=64 -md=32m -ms=on -mhe=on -pSuperSecretPassword C:\Path\To\7-zip\Archive\hMailServer-Backup-YYYY-MM-DD.7z C:\Path\To\Source\Files\In\Dir\

#>

<###   USER VARIABLES   ###>
$APIKey1 = "ASixtyFourCharacterStringAPIKeyNumberOne"
$APIKey2 = "ASixtyFourCharacterStringAPIKeyNumberTwo"
$BackupFolder = "C:\HMS-BACKUP\test"

<#  Begin Script  #>

<#  Clear out error variable  #>
$Error.Clear()

<#  Authorize and get access token  #>

$AuthBody = @{
	'key1' = $APIKey1;
	'key2' = $APIKey2;
}
$URIAuth = "https://letsupload.org/api/v2/authorize"
$Auth = Invoke-RestMethod -Method GET $URIAuth -Body $AuthBody -ContentType 'application/json; charset=utf-8' 
$AccessToken = $Auth.data.access_token
$AccountID = $Auth.data.account_id

<#  Create Folder  #>

$URICF = "https://letsupload.org/api/v2/folder/create"
$FolderName = "Backup-EM-$((Get-Date).ToString('yyyy-MM-dd'))"

$CFBody = @{
	'access_token' = $AccessToken;
	'account_id' = $AccountID;
	'folder_name' = $FolderName;
	'is_public' = 0;               # 0=private, 1=unlisted, 2=public
}

Write-Host "Creating Folder"

$CreateFolder = Invoke-RestMethod -Method GET $URICF -Body $CFBody -ContentType 'application/json; charset=utf-8' 
$CreateFolder.response
$FolderID = $CreateFolder.data.id
Write-Host "Folder ID: $FolderID"

<#  Upload File  #>

$Count = (Get-ChildItem $BackupFolder).Count
Write-Host "There are $Count files to upload"

$N = 1

Get-ChildItem $BackupFolder | ForEach {

	$FileName = $_.Name;
	$FilePath = "$BackupFolder\$FileName";
	
	$UploadURI = "https://letsupload.org/api/v2/file/upload";

	Write-Host "----------------------------"
	Write-Host "Encoding $FileName - $N of $Count"

	$FileBytes = [System.IO.File]::ReadAllBytes($FilePath);
	$FileEnc = [System.Text.Encoding]::GetEncoding('UTF-8').GetString($FileBytes);
	$Boundary = [System.Guid]::NewGuid().ToString(); 
	$LF = "`r`n";

	$BodyLines = (
		"--$Boundary",
		"Content-Disposition: form-data; name=`"access_token`"",
		'',
		$AccessToken,
		"--$Boundary",
		"Content-Disposition: form-data; name=`"account_id`"",
		'',
		$AccountID,
		"--$Boundary",
		"Content-Disposition: form-data; name=`"folder_id`"",
		'',
		$FolderID,
		"--$Boundary",
		"Content-Disposition: form-data; name=`"upload_file`"; filename=`"$FileName`"",
		"Content-Type: application/json",
		'',
		$FileEnc,
		"--$Boundary--"
	) -join $LF
		
	Write-Host "Uploading $FileName - $N of $Count"
	$Upload = Invoke-RestMethod -Uri $UploadURI -Method POST -ContentType "multipart/form-data; boundary=`"$boundary`"" -Body $BodyLines

	$UResponse = $Upload.response
	$UURL = $Upload.data.url
	$USize = $Upload.data.size
	$UStatus = $Upload._status
	
	Write-Host "Response: $UResponse"
	Write-Host "URL     : $UURL"
	Write-Host "Size    : $USize"
	Write-Host "Status  : $UStatus"

	$N++
}
Letsupload API: https://letsupload.org/api.html

User avatar
katip
Senior user
Senior user
Posts: 779
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Free Unlimited Offsite Backup

Post by katip » 2020-10-19 09:31

I would suggest to have a look at this project: https://www.duplicati.com
It helps me a lot to backup gigs of data to off-site drives.
Open source & very active.
Katip
--
HMS 5.7.0 x64, MariaDB 10.4.10 x64, SA 3.4.2, ClamAV 0.101.2 + SaneS

palinka
Senior user
Senior user
Posts: 2180
Joined: 2017-09-12 17:57

Re: Free Unlimited Offsite Backup

Post by palinka » 2020-10-19 11:50

I will. Thanks for the tip!

palinka
Senior user
Senior user
Posts: 2180
Joined: 2017-09-12 17:57

Re: Free Unlimited Offsite Backup

Post by palinka » 2020-10-20 15:56

Fixed up a bit. This version creates the archive as well as upload.

Code: Select all

<#

.SYNOPSIS
	LetsUpload Backup Utility

.DESCRIPTION
	Compresses and uploads folder contents to LetsUpload.io

.FUNCTIONALITY
	Compresses and uploads folder contents to LetsUpload.io

.PARAMETER UploadFolder
	Specifies the folder on local filesystem to compress and upload
	DO NOT include trailing slash "\"
	
.PARAMETER UploadName
	Specifies the name (description) of the archive to be created as well as letsupload folder name
	
.NOTES
	Create account and get API keys from https://www.letsupload.io, then fill in $APIKey variables under USER VARIABLES.
	Run from task scheduler daily.
	Windows only.
	API: https://letsupload.io/api.html
	Install latest 7-zip and put into system path.
	
.EXAMPLE
	PS C:\Users\username> C:\scripts\LetsUpload.ps1 "C:\Path\To\Folder\To\Backup" "Backup Description (email, work, etc)"

#>

Param(
	[Parameter(Mandatory=$True)]
	[ValidatePattern("^[A-Z]\:\\")]
	[String]$UploadFolder,

	[Parameter(Mandatory=$False)]
	[String]$UploadName
)

<###   USER VARIABLES   ###>
$APIKey1           = "1QFMyGCDgCH7BKG6ZKhxmUvAl98abP4bYiJ16iJTtLYZopqycRZJpndpca6ZgByT"
$APIKey2           = "Fky8b24HpzuYhPeXmZO8m1pe6vqcxluodasRtF1C6dnShutYkpguAlJYAWd7JgiB"
$ArchivePassword   = "supersecretpassword" # Password to 7z archive
$BackupLocation    = "C:\LetsUpload"       # Location archive files will be stored
$VolumeSize        = "100m"                # Size of archive volume parts - maximum 200m recommended - valid suffixes for size units are (b|k|m|g)
$IsPublic          = 0                     # 0 = Private, 1 = Unlisted, 2 = Public in site search
$VerboseConsole    = $True                 # If true, will output debug to console
$VerboseFile       = $True                 # If true, will output debug to file

<###   EMAIL VARIABLES   ###>
$EmailFrom         = "notify@mydomain.tld"
$EmailTo           = "admin@mydomain.tld"
$Subject           = "Offsite Backup $UploadName"
$SMTPServer        = "mail.mydomain.tld"
$SMTPAuthUser      = "notify@mydomain.tld"
$SMTPAuthPass      = "supersecretpassword"
$SMTPPort          =  587
$SSL               = $True
$HTML              = $False
$AttachDebugLog    = $True                 # If true, will attach debug log to email report - must also select $VerboseFile
$MaxAttachmentSize = 1                     # Size in MB

<###   FUNCTIONS   ###>
Function Debug ($DebugOutput) {
	If ($VerboseFile) {Write-Output "$(Get-Date -f G) $DebugOutput" | Out-File $DebugLog -Encoding ASCII -Append}
	If ($VerboseConsole) {Write-Host "$(Get-Date -f G) $DebugOutput"}
}

Function Email ($EmailOutput) {
	Write-Output $EmailOutput | Out-File $VerboseEmail -Encoding ASCII -Append
}

Function EmailResults {
	Try {
		$Body = (Get-Content -Path $VerboseEmail | Out-String )
		If (($AttachDebugLog) -and (Test-Path $DebugLog) -and (((Get-Item $DebugLog).length/1MB) -lt $MaxAttachmentSize)){$Attachment = New-Object System.Net.Mail.Attachment $DebugLog}
		$Message = New-Object System.Net.Mail.Mailmessage $EmailFrom, $EmailTo, $Subject, $Body
		$Message.IsBodyHTML = $HTML
		If (($AttachDebugLog) -and (Test-Path $DebugLog) -and (((Get-Item $DebugLog).length/1MB) -lt $MaxAttachmentSize)){$Message.Attachments.Add($DebugLog)}
		$SMTP = New-Object System.Net.Mail.SMTPClient $SMTPServer,$SMTPPort
		$SMTP.EnableSsl = $SSL
		$SMTP.Credentials = New-Object System.Net.NetworkCredential($SMTPAuthUser, $SMTPAuthPass); 
		$SMTP.Send($Message)
	}
	Catch {
		Debug "Email ERROR : `n$Error[0]"
	}
}

<###   BEGIN SCRIPT   ###>
$StartScript = Get-Date

<#  Clear out error variable  #>
$Error.Clear()

<#  Use UploadName (or not)  #>
$UploadName = $UploadName -Replace '\s','-'
$UploadName = $UploadName -Replace '[^a-zA-Z0-9-]',''
If ($UploadName) {
	$BackupName = "$UploadName-$((Get-Date).ToString('yyyy-MM-dd'))"
} Else {
	$BackupName = "Backup-$((Get-Date).ToString('yyyy-MM-dd'))"
}

<#  Delete old debug file and create new  #>
$VerboseEmail = "$PSScriptRoot\VerboseEmail.log"
If (Test-Path $VerboseEmail) {Remove-Item -Force -Path $VerboseEmail}
New-Item $VerboseEmail
$DebugLog = "$PSScriptRoot\LetsUploadDebug.log"
If (Test-Path $DebugLog) {Remove-Item -Force -Path $DebugLog}
New-Item $DebugLog

<#  Create archive  #>
$StartArchive = Get-Date
Debug "Create archive : $BackupName"
Debug "Archive folder : $UploadFolder"
$VolumeSwitch = "-v$VolumeSize"
$PWSwitch = "-p$ArchivePassword"
Try {
	& cmd /c 7z a $VolumeSwitch -t7z -m0=lzma2 -mx=9 -mfb=64 -md=32m -ms=on -mhe=on $PWSwitch "$BackupLocation\$BackupName\$BackupName.7z" "$UploadFolder\*"
}
Catch {
	Debug "Archive Creation ERROR : `n$Error[0]"
	Email "Archive Creation ERROR : Check Debug Log"
	Email "Archive Creation ERROR : `n$Error[0]"
	EmailResults
	Exit
}
Debug "Archive creation finished in $([int]((New-Timespan $StartArchive).TotalMinutes)) minutes."

<#  Authorize and get access token  #>
Debug "Getting access token from LetsUpload"
$URIAuth = "https://letsupload.io/api/v2/authorize"
$AuthBody = @{
	'key1' = $APIKey1;
	'key2' = $APIKey2;
}
Try{
	$Auth = Invoke-RestMethod -Method GET $URIAuth -Body $AuthBody -ContentType 'application/json; charset=utf-8' 
}
Catch {
	Debug "LetsUpload Authentication ERROR : `n$Error[0]"
	Email "LetsUpload Authentication ERROR : Check Debug Log"
	Email "LetsUpload Authentication ERROR : `n$Error[0]"
	EmailResults
	Exit
}
$AccessToken = $Auth.data.access_token
$AccountID = $Auth.data.account_id
Debug "Access Token : $AccessToken"
Debug "Account ID   : $AccountID"

<#  Create Folder  #>
Debug "Creating Folder $BackupName at LetsUpload"
$URICF = "https://letsupload.io/api/v2/folder/create"
$CFBody = @{
	'access_token' = $AccessToken;
	'account_id' = $AccountID;
	'folder_name' = $BackupName;
	'is_public' = $IsPublic;
}
Try {
	$CreateFolder = Invoke-RestMethod -Method GET $URICF -Body $CFBody -ContentType 'application/json; charset=utf-8' 
}
Catch {
	Debug "LetsUpload Folder Creation ERROR : `n$Error[0]"
	Email "LetsUpload Folder Creation ERROR : Check Debug Log"
	Email "LetsUpload Folder Creation ERROR : `n$Error[0]"
	EmailResults
	Exit
}
$CreateFolder.response
$FolderID = $CreateFolder.data.id
$FolderURL = $CreateFolder.data.url_folder
Debug "Folder ID  : $FolderID"
Debug "Folder URL : $FolderURL"

<#  Upload Files  #>
$StartUpload = Get-Date
Debug "Begin uploading files to LetsUpload"
$Count = (Get-ChildItem "$BackupLocation\$BackupName").Count
Debug "There are $Count files to upload"
Email "There are $Count files to upload"
$N = 1

Try {
	Get-ChildItem "$BackupLocation\$BackupName" | ForEach {

		$FileName = $_.Name;
		$FilePath = "$BackupLocation\$BackupName\$FileName";
		
		$UploadURI = "https://letsupload.io/api/v2/file/upload";
		Debug "----------------------------"
		Debug "Encoding file $FileName"
		$FileBytes = [System.IO.File]::ReadAllBytes($FilePath);
		$FileEnc = [System.Text.Encoding]::GetEncoding('UTF-8').GetString($FileBytes);
		$Boundary = [System.Guid]::NewGuid().ToString(); 
		$LF = "`r`n";

		$BodyLines = (
			"--$Boundary",
			"Content-Disposition: form-data; name=`"access_token`"",
			'',
			$AccessToken,
			"--$Boundary",
			"Content-Disposition: form-data; name=`"account_id`"",
			'',
			$AccountID,
			"--$Boundary",
			"Content-Disposition: form-data; name=`"folder_id`"",
			'',
			$FolderID,
			"--$Boundary",
			"Content-Disposition: form-data; name=`"upload_file`"; filename=`"$FileName`"",
			"Content-Type: application/json",
			'',
			$FileEnc,
			"--$Boundary--"
		) -join $LF
			
		Debug "Uploading $FileName - $N of $Count"
		$Upload = Invoke-RestMethod -Uri $UploadURI -Method POST -ContentType "multipart/form-data; boundary=`"$boundary`"" -Body $BodyLines

		$UResponse = $Upload.response
		$UURL = $Upload.data.url
		$USize = $Upload.data.size
		$UStatus = $Upload._status

		Debug "Response : $UResponse"
		Debug "URL      : $UURL"
		Debug "Size     : $USize"
		Debug "Status   : $UStatus"

		If ($UResponse -NotMatch "File uploaded") {
			Debug "Error in uploading file number $N. Check the log for errors."
			Email "Error in uploading file number $N. Check the log for errors."
			EmailResults
			Exit
		}

		$N++
	}
}
Catch {
		Debug "Upload ERROR : `n$Error[0]"
		Email "Upload ERROR : Check Debug Log"
		Email "Upload ERROR : `n$Error[0]"
		EmailResults
		Exit
}

Debug "Upload finished in $([int]((New-Timespan $StartUpload).TotalMinutes)) minutes."

<#  Email results  #>
Debug "Upload sucessful. $Count files uploaded to $FolderURL"
Email "Upload sucessful. $Count files uploaded to $FolderURL"
Debug "Script completed in $([int]((New-Timespan $StartScript).TotalMinutes)) minutes."
Email "Script completed in $([int]((New-Timespan $StartScript).TotalMinutes)) minutes."
Debug "Sending Email"
EmailResults

palinka
Senior user
Senior user
Posts: 2180
Joined: 2017-09-12 17:57

Re: Free Unlimited Offsite Backup

Post by palinka » 2020-10-20 16:50


palinka
Senior user
Senior user
Posts: 2180
Joined: 2017-09-12 17:57

Re: Free Unlimited Offsite Backup

Post by palinka » 2020-10-23 20:03

Fixed a bug that caused file encoding to corrupt the archive to be uploaded. Took a while to figure that one out.

Clue was here: https://stackoverflow.com/questions/254 ... -form-data
#ISO-8859-1 is only encoding where byte value == code point value
$value = [System.Text.Encoding]::GetEncoding("ISO-8859-1").GetString($value)
I was using UTF-8 per the letsupload API instructions. Anyway, its working like a charm now. Free unlimited (and uncorrupted) backup. :D

There's also a download script that looks for the latest backup and puts all the archive volumes in one folder.

https://github.com/palinkas-jo-reggelt/ ... and-Upload

Post Reply