Page 1 of 3

Automatic whitelisting

Posted: 2008-12-17 22:10
by andyp
This script adds all known external email addresses to the hm spam whitelist.

How does it work?
If an authenticated user sends an email to an external address, this address will be stored in a file for a defined time. You can also add addresses manually to this file. The addresses in the file and all external addresses of all distribution lists of your server will be added to hm spam whitelist. This way, trusted email addresses won't be marked or deleted as spam and won't have to wait the delay time of the greylist until they pass the first time. Since version5 you can also have a separate whitelist file.

Basically this is the same as in v4 section, but you can now add addresses manually and the script should be faster.
It works with v5, but it should also be compatible with v4. However you can use the old emailaddressfile of v4.

Installation manual and script itself is in the zip.

Version2: Separate thread for update of whitelist, which is executed in the eventhandler.
Advantages: The SMTP session is a lot shorter and you can slow the script down to reduce CPU activity. Should be a lot better for large installation or long distribution lists. You can also execute this script manually.

Version3: Increased performance

Version4: Bugfix

Version5: Bugfix - Existing email addresses were not update correctly.
New feature - separate whitelist in a file, e.g. provided by another application.

Version6: Instability fix in write_log function; improved register_emailadress function

Version7: Some functions had been missing in version6

Version8: Now all IPs of the mailservers used by registered email addresses are whitelisted as well. The old emailaddressfile will be converted.

Version9: You can now set, what you like to whitelist or not.

Re: Automatic whitelisting

Posted: 2008-12-31 17:57
by M*I*B
... hmmm ... Maybe I'm tooooo stupid ...

After editing and copying the file to the Folder where the EventHandlers.vbs sleeps and name it AutoWhiteList.vbs and after reload ... must be a file called emailaddresses.txt exist since the 1st eMails send out from me to external receipient?!? Is'nt it?!?

Some of my config:

hMailserver work in Folder C:\Program Files (x86)\hMailServer
I edit the INI-fite to match my own need, so the other Folder are under C:\Documents and Settings\Administrator\My Documents\Mailserver\. I edit the AutoWhiteList also to match the right folder.

1st lines of my hMailServer.INI

Code: Select all

[Directories]
ProgramFolder=C:\Program Files (x86)\hMailServer
DataFolder=C:\Documents and Settings\Administrator\My Documents\Mailserver
DatabaseFolder=C:\Documents and Settings\Administrator\My Documents\Mailserver
LogFolder=C:\Documents and Settings\Administrator\My Documents\Mailserver\~logz~
EventFolder=C:\Documents and Settings\Administrator\My Documents\Mailserver\ereignis
TempFolder=C:\temp
[GUILanguages]
ValidLanguages=english,german,swedish
1st lines of the AutoWhiteList.vbs

Code: Select all

'General
Public obApp
Public Const ipslocalhost = "0.0.0.0#87.230.89.37#127.0.0.1"  'separated by #
Public Const user = "[me myself and I]"
Public Const pw = "[what else *g*]"
Public Const logspath = "C:\Documents and Settings\Administrator\My Documents\Mailserver\~logz~\"
Public Const write_log_active = true

'Automatic whitelisting
Public Const remember_wl_address = 365
Public Const emailaddressesfile = "C:\Documents and Settings\Administrator\My Documents\Mailserver\ereignis\emailaddresses.txt"
So what's wrong?

Will be fine if someone can help...

Re: Automatic whitelisting

Posted: 2008-12-31 18:32
by martin
> and name it AutoWhiteList.vbs and after reload

You shouldn't do that. Why do you make up your own script file name?

Re: Automatic whitelisting

Posted: 2008-12-31 18:40
by M*I*B
... öhem ... Why not? Is not it does'nt matter how the script called?

Re: Automatic whitelisting

Posted: 2009-01-01 09:31
by mattg
Unless it is named 'EventHandlers.vbs' it won't be called by hMailServer scripts.
http://www.hmailserver.com/documentatio ... _scripting

Re: Automatic whitelisting

Posted: 2009-01-07 13:10
by westdam
using it on my v.5.321 installation and work like a charm.

great work as the v4 :)

bye
marco

Re: Automatic whitelisting

Posted: 2009-01-08 00:10
by DFitch
Very Nice Script, Thankyou!

However seems I'm having some performance issues. When I try to send at times it gives me a timeout error(but the emails are still sent)

35+domains and whitelist currently 1440.

Any ideas?

Re: Automatic whitelisting

Posted: 2009-01-08 00:58
by martin
I think you will see performance problems if the number of white listing records gets too big. It's not really tested with more than 100-150 white listing records. I expect that you will see slow downs if you have more than a few hundreds. :-\

Maybe it's not the issue you see though, since your email messages are still sent. Also, I'm a bit surprised that you see a timeout already after 1500 records. Maybe if you turn on SMTP logging and post the error here the next time it happens I can see what's going on.

Re: Automatic whitelisting

Posted: 2009-01-08 18:46
by andyp
I had the same problem with large dist lists and am now using a separate script file to update the whitelist. By using a separate thread the smtp session continues within seconds and the long update process is done afterwards. It also doesn't use 100% of CPU capacity any more, because I have added breaks.

Will post it as version2 in the next days. Works fine for 2 weeks...

Re: Automatic whitelisting

Posted: 2009-01-08 22:36
by DFitch
sounds great look forward to trying it.
Thanks again.

Re: Automatic whitelisting

Posted: 2009-01-09 20:34
by andyp
Posted the new script...

Re: Automatic whitelisting

Posted: 2009-01-09 20:50
by martin
Nice work. Just a generic hint which can improve script performance quite a bit. When you do obApp.Domains, a list of domains are fetched from the database ad loaded into the collection. This means that if you execute the following code, the entire domain list is read three times from the database:

Code: Select all

a = obapp.Domains.Item(i) 
b = obapp.Domains.Item(i) 
c= obapp.Domains.Count
Which means that when you execute the following code, the entire domain list is read <number of domains> * 2 times from the database. The same thing applies to dom.DistributionLists and other collections. With your version of the script, if the distribution list contains 100 members, the entire list will be read 200 times from the database.

Code: Select all

Do While i <= obapp.Domains.Count - 1
  Set dom = obapp.Domains.Item(i)
Loop
To get much better performance, you can do

Code: Select all

set obDomains = obApp.Domains
Do While i <= obDomains.Count - 1
Set dom = obDomains.Item(i)
and

Code: Select all

set obDistLists = obApp.DistributionLists
for i = 0 to obDistLists.Count -1

next
and so on. For this specific script, in larger installations, I guess doing this would improve script performance ~10000% or something. :)

It's not obvious if you see it from a "API client" perspective, but when you call obapp.Domains.Item(i) , the script engine will first execute obApp.Domains which will cause the entire domain list to be read from the database. On the object returned by the Domains property, it will then call the Item property. So this single line causes two different items to be created, a Domains collection, and a Domains object.

The benefit of having it this way is that a user of the API doens't need to call Refresh() all the time before trying to access a collection. I've thought about changing this, but at the same time I'm trying to get hMailServer to keep less things in memory (less caching) and more rely on the database...

Re: Automatic whitelisting

Posted: 2009-01-09 22:02
by andyp
Thanks, I assume this applies to all other collections as well. I guess there will be a version3

Re: Automatic whitelisting

Posted: 2009-01-09 22:23
by martin
Yep, pretty much all collections (with maybe some exception).

Re: Automatic whitelisting

Posted: 2009-01-11 05:44
by DFitch
Thanks for update, it seems to be working fine.

However, I am seeing this error over and over.

Script Error: Source: (null) - Error: 80070002 - Description: (null) - Line: 135 Column: 2 - Code: (null)"

line 135 contains: WshShell.run(vbs_update_wl)

Re: Automatic whitelisting

Posted: 2009-01-11 13:11
by andyp
I think the variable in
Public const vbs_update_wl = ...
is not properly defined.

Replace line 135 by and you will see if script can find the file

Code: Select all

		if fs.FileExists(vbs_update_wl) then
			WshShell.run(vbs_update_wl)
		else
			write_log ("      Error, cannot find file " & vbs_update_wl)
		end if
Then I would check the WshShell object is properly created. Lines

Code: Select all

	Dim WshShell
	Set WshShell = CreateObject("WScript.Shell")
Just doubleclick the EventHandlers_awl_update.vbs file to see if it works. Update process will start after 2 seconds.

Re: Automatic whitelisting

Posted: 2009-01-11 16:42
by DFitch
It only gives error when adding a new email to file, otherwise no errors.

Re: Automatic whitelisting

Posted: 2009-01-11 16:55
by andyp
How do you add the email to the file, could you give me an example.
Does it throw an error when executed manually or executed by hmailserver?

Re: Automatic whitelisting

Posted: 2009-01-11 17:11
by DFitch
when executed by hMail

Re: Automatic whitelisting

Posted: 2009-01-11 18:20
by andyp
Can you execute EventHandlers_awl_update.vbs manually without error?

Re: Automatic whitelisting

Posted: 2009-01-11 18:33
by DFitch
object required: obApp

Re: Automatic whitelisting

Posted: 2009-01-11 18:42
by andyp
Have you

Public obApp
and
Set obApp = CreateObject("hMailServer.Application")

in EventHandlers_awl_update.vbs

Re: Automatic whitelisting

Posted: 2009-01-11 19:01
by DFitch
The EventHandlers_awl_update.vbs was missing Set obApp = CreateObject("hMailServer.Application")

I added it and still same error

Re: Automatic whitelisting

Posted: 2009-01-11 19:57
by andyp
Ups, my mistake, version3 contains an error.
add to main part.

Set obApp = CreateObject("hMailServer.Application")
Call obApp.Authenticate(user, pw)

Re: Automatic whitelisting

Posted: 2009-01-11 20:05
by DFitch
OK thx

Re: Automatic whitelisting

Posted: 2009-01-11 20:19
by DFitch
still same error with version4.

There's nothing after execute update in logs. Is second script being called?

"1/11/2009 1:17:02 PM" " execute update"

Re: Automatic whitelisting

Posted: 2009-01-11 21:16
by DFitch
update.....

It works manual now with v4.

"1/11/2009 2:11:26 PM" " Start updating whitelist"
"1/11/2009 2:12:18 PM" " Cleared whitelist."

but Still errors with hMail running it. (hangs at execute)

"1/11/2009 2:13:48 PM" " Registering recipient addresses."
"1/11/2009 2:13:49 PM" " adding new line with xxxx@gmail.com"
"1/11/2009 2:13:49 PM" " writing emailaddressesfile"
"1/11/2009 2:13:49 PM" " execute update"

Re: Automatic whitelisting

Posted: 2009-01-11 23:17
by andyp
Have you tried to create a second vbs file which calls the update vbs? If this doesn't work, the error has nothing to do with hmailserver.

Code: Select all

Public Const vbs_update_wl = "C:\hMailServer\Events\EventHandlers_awl_update.vbs"

Dim WshShell
Set WshShell = CreateObject("WScript.Shell")
if fs.FileExists(vbs_update_wl) then
	WshShell.run(vbs_update_wl)
else
	wscript.echo("      Error, cannot find: " & vbs_update_wl)
end if

Re: Automatic whitelisting

Posted: 2009-01-11 23:23
by DFitch
no, just the scripts that you created.

If i call it with your code i get error for object required: fs

when I set fs: it didn't error.

Re: Automatic whitelisting

Posted: 2009-01-11 23:32
by martin
Add
Set fs = CreateObject("scripting.filesystemobject")
just after the other line containing CreateObject...

Re: Automatic whitelisting

Posted: 2009-01-11 23:34
by DFitch
thx martin, just edited that while you were posting, it didn't error then.

However in his main script it has those variables, but errors.

Re: Automatic whitelisting

Posted: 2009-01-11 23:39
by martin
So have you enabled logging in the EventHandlers_awl_update.vbs?
Does it output anything?

Re: Automatic whitelisting

Posted: 2009-01-11 23:42
by DFitch
Just whatever logging he has coded already. Seems it never gets past that update part from the first script.

I have no scripting knowledge so no clue what to look for.

Re: Automatic whitelisting

Posted: 2009-01-11 23:59
by martin
Try adding
wscript.echo("hello")
just before
WshShell.run(vbs_update_wl)
in your main script.

Then in hMailServer Administrator, refresh script, send a message, see if the "hello" is added to the log.

Re: Automatic whitelisting

Posted: 2009-01-12 00:05
by DFitch
wasn't added

Re: Automatic whitelisting

Posted: 2009-01-12 00:09
by martin
Doesn't make any sense. Mayby andyp has some idea. Could be some permission issue in the file system, or something completely different...

Re: Automatic whitelisting

Posted: 2009-01-12 00:15
by DFitch
Tell me about it.

This is from main script.
-----------------------------------------
If upd = True Then
write_log (" execute update")
if fs.FileExists(vbs_update_wl) Then
WshShell.run(vbs_update_wl)
else
write_log (" Error, cannot find: " & vbs_update_wl)
end if
End If

--------------------------------------

Where in logs it writes "execute update" then stops.

Re: Automatic whitelisting

Posted: 2009-01-12 00:32
by DFitch
When i execute another script with just what Andy suggested, it works fine!
------------------------
Public Const vbs_update_wl = "c:\hMailServer\Events\EventHandlers_awl_update.vbs"

Dim WshShell
Set WshShell = CreateObject("WScript.Shell")
Set fs = CreateObject("scripting.filesystemobject")
if fs.FileExists(vbs_update_wl) Then
WshShell.run(vbs_update_wl)
else
wscript.echo(" Error, cannot find: " & vbs_update_wl)
end if
---------------------

I don't see the difference and why it doesn't execute in main script

Re: Automatic whitelisting

Posted: 2009-01-12 00:37
by martin
Yes, you said that.

Is the vbs_update_wl variable exactly the same in both scripts?
If so, this is probably a permission issue. Maybe the hMailServer service is unable to access the directory containing the file referenced by vbs_update_wl, or it's unable to read the file. Or something similar to that.

Re: Automatic whitelisting

Posted: 2009-01-12 00:44
by DFitch
yes there the same.

Re: Automatic whitelisting

Posted: 2009-01-12 00:52
by martin
There's no error log created, right?

Re: Automatic whitelisting

Posted: 2009-01-12 00:54
by DFitch
yes when hMail executes i get error: Script Error: Source: (null) - Error: 80070002 - Description: (null) - Line: 135 Column: 2 - Code: (null)"

Re: Automatic whitelisting

Posted: 2009-01-12 02:07
by DFitch
update...

Ran test using absolute path of directory and the script fired away and worked properly.

As far as paths to logs, etc were fine if they contained spaces. (program files\hmailserver)

However, when trying to execute the second script it didn't like spaces so correcting with progra~1\hmailserver fixed the issues.

Thanks Andy and Martin for your time!

Re: Automatic whitelisting

Posted: 2009-01-20 10:24
by luci
How about a version of this script that is not executed each time a message is sent ?
The whitelist can be updated by parsing the SMTP log file.
This can be executed with task scheduller, will require less CPU and will avoid the crashes I experience caused by the VBScript runtime.

This version of the script does not work well with hMailServer 4, as it crashes the server from time to time.

Re: Automatic whitelisting

Posted: 2009-01-20 10:36
by ^DooM^
This version of the script is for hMailServer V5 not V4.

Re: Automatic whitelisting

Posted: 2009-01-20 10:43
by luci
Yes, I know, but the author said:
t works with v5, but it should also be compatible with v4.
I was just reporting a problem with v4

Re: Automatic whitelisting

Posted: 2009-01-20 12:07
by andyp
By
but it should also be compatible with v4
I ment I didn't test it, because I don't use v4 any more. Either way I tried to keep it compatible to v4. Please post your error message with version v4, in case you like help.

If you want to run a task with the os scheduler, just execute EventHandlers_awl_update.vbs by scheduler and comment the line

Code: Select all

WshShell.run(vbs_update_wl)

Re: Automatic whitelisting

Posted: 2009-01-20 12:25
by luci
In v4 I get "Source: SocketConnection::OnDataReceived, Code: HM4377, Description: An error occured while parsing data." and the server crashes after a while. I know this is not a VBScript error, but it seems related to the script, because when disabled there are no more errors. The errors occur once or several times a day. When they are occuring they persist until the service is restarted. Sometimes the machines needs to be rebooted.

I will try running that script manually to test it. Thanks for the tip.
I don't move to v5 until I am sure this is stable enough for production use.

Re: Automatic whitelisting

Posted: 2009-01-20 12:40
by luci
So what script I execute manually ? EventHandlers_awl_update.vbs or awl_script ?
Your modification reffers to awl_script. I tried to execute both but the whitelist is not updated.
In fact awl_script relies on OnAcceptMessage to execute.

I need to update the whitelist once at the end of the day based on the SMTP log. Are you sure the script parses the SMTP log ? It does not seem to me.

Re: Automatic whitelisting

Posted: 2009-01-20 15:03
by andyp
Execute EventHandlers_awl_update.vbs with the os task. Awl_script needs to be integrated in the hm eventhandler.vbs, check instruction in the zip.
I need to update the whitelist once at the end of the day based on the SMTP log. Are you sure the script parses the SMTP log ? It does not seem to me.
No and it is not designed to do that. It is best to write a skript on your own, because 80% will change then.

Error: Which script did you execute? In what line occurs the error?

Re: Automatic whitelisting

Posted: 2009-01-20 16:10
by luci
I got errors using awl_script, executed with hmail event handlers. The errors are shown in the hmail log. I am just assuming the script is responsible, even if there are no runtime errors, because when I disable it, the errors go away.
The problem is that I don't have crashes at every script run, but from time to time (from several times in a day to maybe twice a week)

Re: Automatic whitelisting

Posted: 2009-01-20 17:25
by andyp
Still cannot help you, if you don't post me the line code where the error happens. This information is also in the error log.

If you have a large installation and many scripts are executed at the same time by different smtp session. You will get into trouble if 2 or more scripts try to access the file of whitelist storage.

Re: Automatic whitelisting

Posted: 2009-01-20 17:53
by luci
If you have a large installation and many scripts are executed at the same time by different smtp session. You will get into trouble if 2 or more scripts try to access the file of whitelist storage.
I can't give any line number because no line number is shown. The script does not return an error, but the server crashes when using your script from time to time.
Probably that's the thing - concurent opening of the whitelist file. This can explain the impredictible nature of the problem.
I assume the problem persists in hmailserver v5 ?

Re: Automatic whitelisting

Posted: 2009-01-20 18:41
by andyp
Well I don't think it's the access problem, because the error message would say something about "access denied". I have that problem with the log file and am currently trying to avoid that by a lock file.

The error msg indicates the socket connection with the smtp partner might be to long and is disturbed somehow. Afterwards the data cannot be parsed. This could be the result of a long smtp session caused by the script during the update of whitelist. The last version above should has that problem as it executes the update in a separate thread. So it seems to be an internal hm problem. Maybe Martin can help you with that.

Re: Automatic whitelisting

Posted: 2009-03-05 06:22
by tymewyse
BTW, Thanks for the script, it works quite well. When this script is being used, does it interfere at all with using the internal hMS white list?


Jeff

hMS v5.0 B326
SpamAssassin 3.25 (internal)
Win2k AS

Re: Automatic whitelisting

Posted: 2009-03-05 12:29
by andyp
No, it doesn't interfere. But it does delete all entries with zvbs and zlst at the beginning on each update.

Re: Automatic whitelisting

Posted: 2009-03-05 17:30
by chrislees
andyp - This is excellent. THANKS!

Worked first time once I'd stopped rushing and concentrated! Now that I have this I feel that the other users on the box I run will accept grey listing much better. Plan to allow it to build a white list for a while then switch on grey listing.

Slightly off topic:
I understand most concepts (I think) but cannot find any forum discussions or anything in the documentation to describe the white listing tab within grey listing in the hMailServer Administrator interface itself. I realise you are not responsible but reckon you probably know the answer.

Can you direct me to anything or perhaps answer yourself:

I understand that the addresses listed in the 'White listing' section in the hMailServer Administrator interface that stands alone (that your script utilises) are not subject to the setup in Spam protection at all. However, does the grey listing's white list override grey listing or override Spam protection in the same way?

Thanks

Re: Automatic whitelisting

Posted: 2009-03-05 17:37
by andyp
The whitelist I am adding the addresses applies to all spam filters. The tab greylisting whitelist only to the greylisting spam protection, at least I would guess that. Never really tried the other whitelist.

Re: Automatic whitelisting

Posted: 2009-03-06 12:31
by chrislees
Having been confident yesterday that I had this all running nicely I have spotted what might be nothing or might just be a small problem.

I had assumed that the 'last email' date in the emailaddresses.txt file would update to reflect the last date an email was sent addressed to a user. Yesterday an outgoing mail wrote the address in successfully as you can see below, but today a mail from the same internal user to the same external user has triggered a "3/6/2009 10:17:57 AM" " writing emailaddressesfile" in the log, but the date has not changed in the emailaddresses.txt file.

The user that was mailed to yesterday AND today is the top line - chris.lees@...

Am I wrong in what I expected to happen or is there a problem with my setup?

emailaddresses.txt is included from just a few minutes ago (ie after the log shows the 'writing' entry

Code: Select all

# email	nr_emails	last_email	int_date
# add new addresses by writing every single address in a separate line

chris.lees@externaldomain.net	1	3/5/2009	39877
stephen.cooper@externaldomain.co.uk	1	3/5/2009	39877
osyjavied@externaldomain.com	1	3/5/2009	39877
m.longden-jones@externaldomain.net	1	3/5/2009	39877
clees@externaldomain.sch.uk	1	3/5/2009	39877
cliffordaleigh@externaldomain.com	1	3/5/2009	39877
sarahcalvey@externaldomain.co.uk	1	3/5/2009	39877
john.gregory@externaldomain.co.uk	1	3/6/2009	39878

Re: Automatic whitelisting

Posted: 2009-03-06 22:56
by andyp
It might be possible more hm threads try to access the file simultaneously. Avoiding this has not been implemented, yet. In this case the second thread would throw an error which would be definetly in hm log. Something like "access denied".