hMailServer Offsite Backup

palinka · Post by **palinka** » 2020-10-27 17:51

First of all, I have to give Jimimaseye 100% credit for this. Except for the uploading part, its all from his backup script here: https://hmailserver.com/forum/viewtopic ... 21&t=28139 which I've been using with great success for over a year. I wasn't planning on touching it at all, but I couldn't figure out how to modify the original to do what I wanted. So I re-wrote the whole thing in powershell, which I know a lot better than plain old windows command line.

You know what they say - its easier to write code than to read code.

Anyway, it does exactly the same thing, except for two things - it compresses the backup files to a password protected archive and it uploads the archive to LetsUpload.io.

I recently discovered letsupload when I was looking for a solution to back up work files offsite. Its free and unlimited and has an API. The only caveat is that undownloaded files are deleted after 60 days. For backup purposes, that's no big deal. 60 days is plenty as far as I'm concerned.

The only requirement is installing 7-zip and putting it in the system path. Also, the database must be mysql -or- internal db (same as the original script).

One note about 7-zip: It takes A LOT longer to compress with encryption than unencrypted. However, I believe this is a necessary action because I wouldn't feel comfortable throwing up my personal files into a barely known storage space. My last run took 34 minutes 45 seconds to create a 2,549 MB encrypted archive from 5,117 MB backup data dir (as reported by 7-zip). My hardware is not exactly cutting edge, so you can probably do better than that.

The sequence of events goes like this:

1. Stops hMailServer and (if in use) Spamassassin Services
2. Updates Spamassassin (if in use)
3. Cycles hMailServer and (if in use) Spamassassin logs
4. Backs up hMailServer data directory using RoboCopy
5. Dumps MySQL database -or- internal database and adds to backup directory
6. Restarts Spamassassin (if in use) and hMailServer
7. Deletes messages older than specified number of days in specified folders (eg Trash, Spam, etc)
8. Compresses the backup directory into a multi-volume 7z archive with AES-256 encryption (including header encyrption)
9. Creates a folder at LetsUpload.io and uploads the archive files
10. Sends success/fail email with debug log attached

You can see the sequence of events is exactly the same as the original script except for the type of zip archive and uploading to letsupload. I split the archive into 100MB volumes and each takes < 20 seconds to upload. The last run was 26 volumes and took 8.5 minutes in total.

Although the original backup script is fantastic, one thing I tried to improve on was the email and log output. The things I looked for in the log every morning was the number of spam & viruses, and robocopy stats for how many messages were added (received since the last backup) and deleted, so I included that information in the email body. Also, I included a bunch of timers so you can see how long each operation takes. I think that's useful information.

I also "improved" the cleardown script so that it only reports messages deleted and the total count - also included in the email body. "Improved" in quotation marks because this part is yet to be heavily tested and my skills in this area can't match Jimi's, although if you want to just try it out, its a function in a separate file, so just comment out the DeleteByDBID lines and give it a go. Its working for me without errors so far.

Files here: https://github.com/palinkas-jo-reggelt/ ... ite_Backup

Enjoy.

jimimaseye · Post by **jimimaseye** » 2020-10-27 18:11

palinka wrote: ↑
2020-10-27 17:51
First of all, I have to give Jimimaseye 100% credit for this. Except for the uploading part, its all from his backup script here: https://hmailserver.com/forum/viewtopic ... 21&t=28139

Arrr..... ....memories. 🥰

palinka · Post by **palinka** » 2020-10-27 19:35

Probably the most useful script in the whole place.

palinka · Post by **palinka** » 2020-10-30 15:20

I added better error handling for the upload part.

* If there is a failure during upload, it will try again N times (specified in the config).
* If there's a successful upload, but the remote file size does not match the size of the file uploaded, it will throw an error, delete the uploaded file and try again.
* At completion, it counts the remote files and compares to the local backup files. If there's a mismatch, it NOTIFIES of an error, but doesn't try the whole upload process over again.

I've been running it for the past week and tweaking the debug output as I go along.

Here's a sample email notification. I don't know why, but it took about twice as long last night as previous nights. Part of that is due to slower upload speeds than normal, which have been <20 seconds per 100mb volume.