Automatic whitelisting

This section contains scripts that hMailServer has contributed with. hMailServer 5 is needed to use these.
chrislees
Normal user
Normal user
Posts: 63
Joined: 2006-11-04 18:30
Location: UK

Re: Automatic whitelisting

Post by chrislees » 2009-03-11 13:09

OK,

Firstly, excellent script - does just what I needed and for me to write this would have taken ages and been very inefficient I would think.

I have got my setup working the way I expect it was designed to and now I know why.

Either I installed it wrong or there is a bit of code missing (or I have an old version???).

My message count and last email date were not updating and I spotted that the code below (starting at line 48 from 'Automatic Whitelisting) could never run (I think):

Code: Select all

If fd = True then
	write_log ("    adding to line " & LCase(arr(0)))
	content = content & LCase(arr(0)) & Chr(9) & arr(1) + 1 & Chr(9) & Date() & Chr(9) & CLng(Date()) & nl
Else....
As I saw it fd was never going to equal True at this point in the script as it has been set at false under all conditions applying to this If block only 6 lines before.

In the following block (starting at line 41) I added 'fd = True' at line 46

Code: Select all

If UBound(arr) = 3 then
	fd = false
	For j = 0 To UBound(toarr) - 1
		If LCase(arr(0)) = LCase(toarr(j)) Then
			fndadr = fndadr & "#" & LCase(toarr(j))
			fd = True
		End If
	Next
...
and now everything appears to be doing what I thought it would.

Is this correct or am I still misunderstanding?
hMailServer 5.4 (1931)

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-03-11 17:40

You are right, I will post the new script in the evening. The line you added is in one of my earliest versions, somehow I lost it.

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-03-14 09:08

Did the bugfix to the script and added a new feature. You can define a text file, which contains a separate list of addresses to whitelist. On my server addresses can be added by the users sending a specific mail to a specific address.

chrislees
Normal user
Normal user
Posts: 63
Joined: 2006-11-04 18:30
Location: UK

Re: Automatic whitelisting

Post by chrislees » 2009-03-26 14:42

How have you acheived that?

I have been toying with the idea of developing a script to let users add to the whitelist without my intervention as the mail admin.

I hoped to accept mail addressed to white@domain.tld from an authenticated user and extract any email addresses from the subject line in order to add them to your emailaddresses.txt file so that your existing setup would take it from there.

Is this what you are describing?
hMailServer 5.4 (1931)

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-03-26 15:05

Yes that is one way to do it. I have an emailaddress to which authenticated users can sent a list of addresses to whitelist. These addresses will be added in the whitelist.txt file. Addresses stored there will be always updated in the whitelist of hm. The difference is that addresses in whitelist.txt will not be deleted after x days like in emailaddresses.txt. The script adding the addresses in one of the files is not provided with the above scripts.

chrislees
Normal user
Normal user
Posts: 63
Joined: 2006-11-04 18:30
Location: UK

Re: Automatic whitelisting

Post by chrislees » 2009-04-14 11:57

I have been having a bit of trouble with one particular spam issue and it stems from having whitelisted various addresses.

I have domains on two hM5 boxes (mx01 and mx02); half on each. The domains which are primary on mx01 have mx02 as a backup mx (and vice versa). On mx01 there is a TCP/IP route for mx02 (and vice versa) which has AntiSpam disabled - ie I trust ALL mail from each box destined for the other box.

I have various AntiSpam measures enabled in hMailServer and use andyp's Automatic whitelisting script on both boxes.

Lots of the users on each box have connections (business relationships, friends, etc) with users on the other and so send each other mail. Consequently there are mailboxes on mx01 with addresses which are whitelisted on mx02.

Suppose domain.tld is hosted on mx01 and mx02 is a backup mx for domain.tld. If mx02 receives a spam addressed TO AND FROM user@domain.tld it doesn't do AntiSpam checks for it because the FROM address is in the white list on mx02. The spam is delivered and doesn't get checked by mx01 when relayed to it because of the TCP/IP range with AntiSpam disabled.

In order to get around this I have now modified the script to stop it adding addresses for domains for which 'Routes' exist and deleted all addresses from the whitelist for these domains.

Mail which is sent by a user via mx02 to a user which exists in a domain which is primary on mx01 is still delivered as normal without AntiSpam checks as a result of the TCP/IP range on mx01. (Obviously the same is true in the other direction). So these users do not suffer the greylisting delay as a result of my having removed their whitelisting entries.

The script modification seems to be working OK, I'd be grateful if anyone can see any errors I've missed that might rear their heads under certain circumstances I've not foreseen.

Basically the script now treats domains that are 'Routes' as if they were local (using the function andyp already included), IF the domain is flagged as 'Treat this route as a local domain' in the route settings.

The modifications are as follows:

Inserted within the Const block in EventHandlers.vbs (to allow easy switching on and off of the feature):

Code: Select all

Public Const treat_mx_backup_domains_as_local = 1
Inserted in EventHandlers.vbs in the is_local_domain function at the end of the 'If domain_buffer = "" Then' block:

Code: Select all

	Set objRoutes = obapp.Settings.Routes
	
	If treat_mx_backup_domains_as_local Then
		For i = 0 to objRoutes.Count - 1
			If objRoutes.Item(i).TreatSecurityAsLocalDomain Then
				domain_buffer = domain_buffer & "#" & objRoutes.Item(i).DomainName
			End If
		Next
	End If
hMailServer 5.4 (1931)

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-05-12 20:30

This should work. But this
Suppose domain.tld is hosted on mx01 and mx02 is a backup mx for domain.tld. If mx02 receives a spam addressed TO AND FROM user@domain.tld it doesn't do AntiSpam checks for it because the FROM address is in the white list on mx02. The spam is delivered and doesn't get checked by mx01 when relayed to it because of the TCP/IP range with AntiSpam disabled.
should be resolved in 5.1 and above if I understand you correctly.

User avatar
martin
Developer
Developer
Posts: 6834
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: Automatic whitelisting

Post by martin » 2009-05-12 21:45

If the sender of an email message is authenticated, hMailServer doesn't do any spam protection.
If the sender is not authenticated, hMailServer 5.1 does all forms of spam protection. In version 5.0 and previous, hMailServer did all spam protection except for grey listing.

chrislees
Normal user
Normal user
Posts: 63
Joined: 2006-11-04 18:30
Location: UK

Re: Automatic whitelisting

Post by chrislees » 2009-05-12 22:09

I think I may be misunderstanding

Are you saying that in v5.0, for a message with a from address which is whitelisted, from a sender who is not authenticated, then all spam checks except greylisting are performed?

And in V5.1, with from whitelisted, sender not authenticated ALL spam checks are performed?

I thought that the whitelisting tab within greylisting achieved that and that the other whitelisting screen circumvented ALL spam checks.
hMailServer 5.4 (1931)

djamell
New user
New user
Posts: 15
Joined: 2009-04-12 03:55

Re: Automatic whitelisting

Post by djamell » 2009-05-19 18:43

What is the format for address lines added to the whitelist.txt file?

Also, what relationship does the whitelist.txt file have to the data stored in the hm_greylisting_whiteaddresses table in the DB?

Thanks in advance.

djamell
New user
New user
Posts: 15
Joined: 2009-04-12 03:55

Re: Automatic whitelisting

Post by djamell » 2009-05-23 15:43

Bump please.

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-05-24 12:55

Each address in one line without any other characters.

Addresses in this file will be permanently updated in the whitelist.

djamell
New user
New user
Posts: 15
Joined: 2009-04-12 03:55

Re: Automatic whitelisting

Post by djamell » 2009-05-26 17:27

I am just now deploying my new hMailServer to my clients and had an issue this morning with a customer that had a copying machine/scanner that sends email. It would hang while sending mail the first time to an address. Subsequent sends to the same address went fine. Other users running Outlook are having no issues. I am also seeing the same issue from client websites that send email through the Web Server using a Contact Form.

Looking at the hMailServer Logs, the mail is accepted for edlivery, but never gets to the intended recipient on the first try.

Then I noticed that the emailaddresses.txt file contained addresses from local hMailServer Domains.

I guess I have two questions.

1. Is the script supposed to add local domains to the emailaddresses.txt file?

2. Is there some delay that's causing mail-enabled devices and Web Server Contact Forms to stumble?

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-05-26 19:35

djamell wrote:1. Is the script supposed to add local domains to the emailaddresses.txt file?
No, it should not in newer versions
djamell wrote:2. Is there some delay that's causing mail-enabled devices and Web Server Contact Forms to stumble?
I think version1 and 2 is doing this.

So make an update to version5

djamell
New user
New user
Posts: 15
Joined: 2009-04-12 03:55

Re: Automatic whitelisting

Post by djamell » 2009-05-26 22:57

Version 5 is the only one I've used. Is there some configuration for local domains that I need to do?

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-06-02 10:26

Insert the markes lines. Check the log to see what the script is doing ...

Code: Select all

	source = get_smtp_recipient(oMessage, oClient.IPAddress)
	sourcearr = Split(source,"#")
	For k = 0 To UBound(sourcearr)
		If Not is_local_domain(sourcearr(k)) Then
			toarr = toarr & sourcearr(k) & "#"
'From the line afterwards
			write_log("    Adding external address: " & sourcearr(k))
		Else
			write_log("    Ignoring local address: " & sourcearr(k))
'To the line before
		End If
	Next
	toarr = Split(toarr,"#")

djamell
New user
New user
Posts: 15
Joined: 2009-04-12 03:55

Re: Automatic whitelisting

Post by djamell » 2009-06-02 15:24

OK, now I'm really confused. I even re-downloaded the package from the first post in this thread to make sure I was right.

I can't seem to find this block of code anywhere in either awl_script.txt or EventHandlers_awl_update.vbs.

I tried searching for "source = get_smtp_recipient" and "sourcearr = Split" to no avail.

Sorry in advance if I'm doing something wrong.

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-06-03 10:22

We had the problem that I continously improve the code without post every change. I had the script already changed so you could recognized the code. Posted an update above. No adding of local domains in version6 any more.

djamell
New user
New user
Posts: 15
Joined: 2009-04-12 03:55

Re: Automatic whitelisting

Post by djamell » 2009-06-03 13:47

I'm going out of town for a week, so I won't get a chance to try this until I get back.

Thanks for all your efforts. This is a valuable script for anyone using the HMail Greylist feature.

User avatar
DFitch
Senior user
Senior user
Posts: 258
Joined: 2006-09-16 20:40

Re: Automatic whitelisting

Post by DFitch » 2009-06-04 16:26

I updated with version 6 and now see this error.

Script Error: Source: Microsoft VBScript runtime error - Error: 800A000D - Description: Type mismatch: 'get_smtp_recipient' - Line: 76 Column: 1 - Code: (null)"
hMailServer 5.3.3: External MySql
Win2k3 Server | eWall 4.0 Anti-Spam Anti-Virus SMTP Proxy {http://sssolutions.net/}
SpamAssassin 3.31 - ClamAV on backend Ubuntu Server 10.04(VMware)

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-06-05 10:17

Yeah, these and some other stuff was missing, posted the updated file in the first post. Sorry.

User avatar
DFitch
Senior user
Senior user
Posts: 258
Joined: 2006-09-16 20:40

Re: Automatic whitelisting

Post by DFitch » 2009-06-05 13:44

no problem, thank you for a great contrib!
Working like a charm again.
hMailServer 5.3.3: External MySql
Win2k3 Server | eWall 4.0 Anti-Spam Anti-Virus SMTP Proxy {http://sssolutions.net/}
SpamAssassin 3.31 - ClamAV on backend Ubuntu Server 10.04(VMware)

izero
New user
New user
Posts: 26
Joined: 2007-01-19 00:29
Contact:

Re: Automatic whitelisting

Post by izero » 2009-07-09 10:38

Hi mans, I help you.

If you need better performance (and better scripts :) ), use ADO Recordset at "offline" mode. You obtain very fast searching throught records, more better procedures for searching (look at Find procedure in Recordset) and of course, you can save and load recordset to/from file (csv or I prefer xml). You can speedup your code by this "feature" about 10000% :wink:

Here is a small pieces of code:

Code: Select all


Class DBIDCollection

	Private colTest
	
	Private Sub Class_Initialize()
	
		Set colTest = CreateObject("ADODB.Recordset")
		
		colTest.Fields.Append "MyID", 200, 25
		colTest.Fields.Append "CodeID", 200, 1024
		colTest.open
	
	End Sub
	
	Private Sub Class_Terminate()
	
		colTest.Close
		
		Set colTest = Nothing
		
	End Sub
	
	Public Property Get Rc
	
		'direct access to Recordset
		Set Rc = colTest
	
	End property
	
	Public Function Append(ByVal itemID, ByVal itemCode)
		
		colTest.AddNew Array("MyID", "CodeID"), Array(itemID, itemCode)
		
		Append = itemID
		
	End Function
	
	Public Function FindValueByItem(ByVal itemCode)
	
		colTest.Find "CodeID = '" & itemCode & "'"

		If Not colTest.EOF Then
			FindValueByItem = CStr(colTest("MyID").Value)
		Else
			FindValueByItem = "-1"
		End If
	
	End Function
	
	Public Function FindValueByItemEx(ByVal keyID, ByVal fieldName)
	
		Dim strRet
		
		colTest.Filter = "MyID = '" & keyID & "'"

		If Not colTest.EOF Then
			strRet = CStr(colTest(CStr(fieldName)).Value)
		Else
			strRet = ""
		End If
		
		If colTest.RecordCount > 1 Then strRet = "-1"
		
		FindValueByItemEx = strRet
	
	End Function
	
End Class


Set colTest = New DBIDCollection

colTest.Append "a100", "b100"

strReturn = colTest.FindValueByItemEx("a100", "CodeID") 'get CodeID field where MyID = a100

colTest.Rc.Save "card.xml", 1

'colTest.Rc.Open "card.xml"

Set colTest = Nothing

cheers :mrgreen:

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-07-09 17:16

I already thought of using a database or something like you provided. Just hadn't the time to implement it.

User avatar
DFitch
Senior user
Senior user
Posts: 258
Joined: 2006-09-16 20:40

Re: Automatic whitelisting

Post by DFitch » 2009-08-11 15:13

Hi Andy,

This script is working great, I was gonna suggest or see how hard it would be to implement an add-on feature (DNS Query and Whitelist IPS for extended time).

When the authenticated local user sends message to external user not only whitelist that user BUT also run a DNS query on that external domain for mx records and whitelist all IPS for say 30days or whatever the admin chooses?

I currently have a script in eWall that does this, however would like the option in hMail too. ^_^

Is this possible?

Thanks Again,
D
hMailServer 5.3.3: External MySql
Win2k3 Server | eWall 4.0 Anti-Spam Anti-Virus SMTP Proxy {http://sssolutions.net/}
SpamAssassin 3.31 - ClamAV on backend Ubuntu Server 10.04(VMware)

izero
New user
New user
Posts: 26
Joined: 2007-01-19 00:29
Contact:

Re: Automatic whitelisting

Post by izero » 2009-08-11 17:14

DFitch wrote: Is this possible?
It is simple ... use GetMailServer function from Utilities object, you can obtain all IPs separated by comma. I use this function in my scripts for my "antispam" script, see user contributed scripts for hMail 5 in this forum.

izero

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-08-11 22:36

DFitch wrote:Is this possible?
Yes, it is possible.
izero wrote:It is simple
Yes, too.
But not done in a few minutes. I like it and I will take some time in the next weeks to do it.

User avatar
DFitch
Senior user
Senior user
Posts: 258
Joined: 2006-09-16 20:40

Re: Automatic whitelisting

Post by DFitch » 2009-08-12 00:19

I like it and I will take some time in the next weeks to do it.
Great, look forward to the addition.

May add some check for current whitelisted users and check If IP is already listed if not then add.

Thanks
D
hMailServer 5.3.3: External MySql
Win2k3 Server | eWall 4.0 Anti-Spam Anti-Virus SMTP Proxy {http://sssolutions.net/}
SpamAssassin 3.31 - ClamAV on backend Ubuntu Server 10.04(VMware)

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-09-05 02:54

I have added the new feature.

Now all IPs of the mailservers used by registered email addresses are whitelisted as well. The old emailaddressfile will be converted.
Attachments
awl_v5_version8.zip
(6.94 KiB) Downloaded 540 times

User avatar
DFitch
Senior user
Senior user
Posts: 258
Joined: 2006-09-16 20:40

Re: Automatic whitelisting

Post by DFitch » 2009-09-06 16:34

Thankyou :mrgreen:

Just a note thou: If you have many users whitelisted and send an email after updating the script the email will most likely timeout, let the process finish updating, it took probably 5mins+ to finish(approx.1200+whitelisted). Then everything seems to work properly.

I have to check the database entry too, think there are many duplicates if you don't let it finish.

Maybe way to check for duplicates and remove?

Thanks again.
D
hMailServer 5.3.3: External MySql
Win2k3 Server | eWall 4.0 Anti-Spam Anti-Virus SMTP Proxy {http://sssolutions.net/}
SpamAssassin 3.31 - ClamAV on backend Ubuntu Server 10.04(VMware)

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: Automatic whitelisting

Post by westdam » 2009-10-09 09:57

mm
with this new feature if I write to an hotmail user with my local auth account all the hotmail MX would be whitelisted too??
remember me please, the whitelisted ip is "global" or only for the account just have sent the mail?

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-10-09 19:36

Yes, the ips of all server of the mx entry will be generally whitelisted.

westdam
Senior user
Senior user
Posts: 728
Joined: 2006-08-01 21:24
Location: Padova, Italy
Contact:

Re: Automatic whitelisting

Post by westdam » 2009-10-12 15:02

mm i like your script so much. but i've bumped your feature request to add a simple version of this into hmailserver.

what do you think?

joksi
Normal user
Normal user
Posts: 76
Joined: 2006-10-15 12:29

Re: Automatic whitelisting

Post by joksi » 2009-10-29 08:53

You should make it possible in the settings to disable the automatic whitelisting of senders server(s).
This feature is potentally bad, its one thing sending email to a company and then whitelist the companys own mailserver (because probably the company is legit if you mail them), but, its not OK whitelisting ie Hotmails and Gmails, or another big ISPs mailservers, with thousands or even million random users.

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-10-31 13:09

new provided version

joksi
Normal user
Normal user
Posts: 76
Joined: 2006-10-15 12:29

Re: Automatic whitelisting

Post by joksi » 2009-11-01 00:19

Also it would be great if ipslocalhost variable could handle ie 192.168.1.0/24 or 192.168.1.1-254, or something similar.

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-11-01 05:28

the ip of the localhost is just used for the determination whether the client has authenticated. Usually you can send without auth from the localhost, so you probably what to whitelist these addresses as well. I don't think that a mailserver has that many ips. But feel free to customize this part of the script.

joksi
Normal user
Normal user
Posts: 76
Joined: 2006-10-15 12:29

Re: Automatic whitelisting

Post by joksi » 2009-11-01 11:33

What I meant was, lets say the localhost is 10.1.1.10, and all clients in 10.1.1.1/24 is sending without authentication. Because of this the script wont add external mail from the local network, one would need to add all 254 local IPs delimited with #.

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-11-01 15:20

I understood your request well, but I would assume most people have a server in the web and are not using authentication for local net. So am I, except my backupserver.

joksi
Normal user
Normal user
Posts: 76
Joined: 2006-10-15 12:29

Re: Automatic whitelisting

Post by joksi » 2009-11-04 20:37

Well that means that local LAN senders will not populate the external address whitelist, cause the script only populates it with external addresses with authenticated users as senders.

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-11-05 22:36

No, that means that authenticated user and senders from the ip configured by variable ipslocalhost populate the list. As I already said: Feel free to change the script.

joksi
Normal user
Normal user
Posts: 76
Joined: 2006-10-15 12:29

Re: Automatic whitelisting

Post by joksi » 2009-11-05 23:25

That is exaclty what I was saying, and that is why it should be possible to enter a subnet in the ipslocalhost variable instead of a single IP, cause the local LAN probably dont have to use authentication which means that clients in the local LAN will not populate the whitelist with external addresses.

andyp
Normal user
Normal user
Posts: 191
Joined: 2008-01-18 21:00

Re: Automatic whitelisting

Post by andyp » 2009-11-06 19:23

I am not saying your argumentation is wrong or not logical. I also told you, your proposal should not be required for people running hmailserver on a usual webserver with one or several IPs. But I simply don't have the need for this feature and am not going to change the script. So the third time: Feel free to change script and provide us the code!!!

joksi
Normal user
Normal user
Posts: 76
Joined: 2006-10-15 12:29

Re: Automatic whitelisting

Post by joksi » 2009-11-06 19:59

I am not telling You that you should modify the script, its your script, im just giving a suggestion.
You may be right that most people have the mailserver on a host external to their local LAN (and therefore using authentication), on the internet, even if my personal opinion is that this may not be the case,cause im thinking it should be more usual to have a mailserver in your own LAN. If not, probably you have an ISP provider providing you the mail services, instead of runnin your own server with Hmail.

Its still a fact though, and a bad one, meaning that only authenticated senders will popolate the thrusted external whitelist with the destionation address. So for all of you having not authentication for your local mailusers, this script would prove useless because no whitelist will ever be built.

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: Automatic whitelisting

Post by ^DooM^ » 2009-11-06 20:45

All of my Lan users authenticate except for web services.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

joksi
Normal user
Normal user
Posts: 76
Joined: 2006-10-15 12:29

Re: Automatic whitelisting

Post by joksi » 2009-11-06 22:47

Okey, well there is actually no need for authentication when your on the same LAN,but maybe its a questions of preference.

^DooM^
Site Admin
Posts: 13861
Joined: 2005-07-29 16:18
Location: UK

Re: Automatic whitelisting

Post by ^DooM^ » 2009-11-06 22:53

I agree it's definitely preference related.
If at first you don't succeed, bomb disposal probably isn't for you! ヅ

philg
Normal user
Normal user
Posts: 38
Joined: 2008-05-03 15:49

Re: Automatic whitelisting

Post by philg » 2011-01-18 22:13

Hi Andy.

I have been using auto-whitelist, and several other scripts (including list-server) for some time now, and every now and then, I get a strange DEP error in hMailServer. The error was repeatable, under certain load conditions (many emails being sent to new addresses in short time intervals, such as generated by a mailing list application.

Anyway after several days of tinkering and experimentation I figured out what the problem was. Every time a mail is sent, the autowhitelist script maintains its local files of whitelist and then it updates the hMailServer whitelist (in sub update_whitelist())

When updating the hMailserver white list, it removes ALL addresses and then re-adds them all.

The problem is, if another mail is sent during this process, the list can be cleared while the first message thread is still using it, causing all sorts of nasty COM errors eventually leading to the DEP crash!

I have managed to protect against this by adding a lock file in the awl update process EventHandlers_awl_update.vbs which forces the script to wait if another instance is running (by checking for the existence and failure to open the "open for non-shared write" lock file). Whilst this has made the scripts a little bit slower, it has completely eradicated any DEP crashes I was getting in hMailserver.

Due to the complex nature of the changes its not particularly feasible to post the updates here. I can, if required provide the complete EventHandlers_awl_update.vbs script file if people are interested/having the same issues I raised in this post.

I wish there were a way of adding more advanced scripting API, maybe C++ or .NET support because writing this sort of thing in C# would be so much more productive and error protection/recovery would be a lot easier than vbscript!

Apologies if this has already been covered.

Thanks,
Phil

bathroomfloormatt
New user
New user
Posts: 15
Joined: 2011-01-19 16:39

Re: Automatic whitelisting

Post by bathroomfloormatt » 2011-01-19 16:54

Hi All,

This is my first post and i have registered as i have a problem getting the auto whitlist on hmailserver version 5.3.3. I have it working fine on an older installation (4.4.1).

Firstly a bit about the server, its a hp ml110 running windows 2003 std r2 with latest updates. have clamwin and mcafee virusscan 8.7 installed, running hmail 5.3.3.

hmail is configured mainly as a relay to our exchange server however does have a few imap accounts for another domain we have.

anyway, i have loaded in the auto whitelist script from andyp and set it up correctly (i think).

when i check syntax it says ok.

However it does and does not seem to working. when i send emails through the server using either an authenticated user or from an ip address i have added to both the vbs files in the events folder it does not add them to the whitelist in the hmail admin console.

the script writes to the event log and the address i sent to is added by the script to the "emailaddresses.txt"file, but it does not add it to hmail. ie if i go to the admin console and to whitelist it is not in there.

I wasnt sure if this was a feature, so tried replying to an email sent through hmail, but got a please try again later from th greylisting. however that addess was listed in the emailaddresses.txt file.

so it seems the script is capturing the outgoing email addresses and logging them, but not adding them into hmail.

any ideas as i really would like to upgrade our older hmail installs.

PS: why is auto whitlist not part of hmailserver by default, combined with greylisting it makes for the best antispam solution.

Thanks
Matt.

edit:

have just looked at the error log and i get the following:

"ERROR" 2568 "2011-01-19 14:35:09.439" "Script Error: Source: (null) - Error: 80070002 - Description: (null) - Line: 155 Column: 3 - Code: (null)"

User avatar
mattg
Moderator
Moderator
Posts: 21044
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Automatic whitelisting

Post by mattg » 2011-01-20 08:43

Did you use the version 5 version of this script, not one that you had used previously with version 4 of hMailserver? The COM API changed significantly between version 4 and and version 5, and most scripts will need to be updated...
bathroomfloormatt wrote:have just looked at the error log and i get the following:

"ERROR" 2568 "2011-01-19 14:35:09.439" "Script Error: Source: (null) - Error: 80070002 - Description: (null) - Line: 155 Column: 3 - Code: (null)"
Also, what is line 155 of your script?

Unrelated to this problem, but as an aside, you may want to look at this port of ClamAV rather than ClamWIN
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

bathroomfloormatt
New user
New user
Posts: 15
Joined: 2011-01-19 16:39

Re: Automatic whitelisting

Post by bathroomfloormatt » 2011-01-20 14:02

Thanks for you reply mattg.

I downloaded "awl_v5_version9.zip" and started from scratch. i have also tried using the zip that conatins the older versions but still the same problem.

its now line 122 on the version of the script im using that is getting the error (i installed an older version to try).

it falls in this:

If upd = True Then
write_log (" execute update")
WshShell.run(vbs_update_wl)
End If

and line 122 is:

WshShell.run(vbs_update_wl)

thanks for the advice about clamav, will give that a look.

Matt.

PS. not sure if it matters, but the install of hmailserver 5.3.3 was not clean, as i upgraded a version 4 install. - may have left some of 4 behind????

philg
Normal user
Normal user
Posts: 38
Joined: 2008-05-03 15:49

Re: Automatic whitelisting

Post by philg » 2011-01-20 18:09

I think I had a simular problem.

Try replacing the line

Code: Select all

WshShell.run(vbs_update_wl)
with:

Code: Select all

WshShell.run vbs_update_wl, 1, false
EDIT: Actually thinking about it, that probably wont make any difference.

If that does not work, then I suggest something inside the script "EventHandlers_awl_update.vbs" may be the problem.
You could try running that vbs script file from the command line (I found it does not have to be run from hMailserver) and see if it throws any obvious errors - thats how I managed to debug the copy I have.

Thanks,
Phil

bathroomfloormatt
New user
New user
Posts: 15
Joined: 2011-01-19 16:39

Re: Automatic whitelisting

Post by bathroomfloormatt » 2011-01-24 14:11

Thanks philg,

Tried changing that line as suggested but no luck im afraid, still has the error on the same line.

ran eventhandlers_awl_update.vbs from the cmd line and got this error:

Image

welcome any help i can get

Thanks
Matt.

philg
Normal user
Normal user
Posts: 38
Joined: 2008-05-03 15:49

Re: Automatic whitelisting

Post by philg » 2011-01-24 15:14

OK, thought that might be the case.

Open up the eventhandlers_awl_update.vbs and see if at the top it says "Option Explicit" (mine does but I cant remember if that's a change I added myself). If it does, then I'm guessing that obApp is not declared at the top...


At this point, rather than try and fix the original script (which Ive already done for myself) I think I will just attach my modified version.

But please be aware:
  • It has been heavily modified to try and reduce multi-threading problems when 2 mails come in very close together (of course thats a good thing to have - but its a big difference from the original). Please don't forget to fix the lines marked with CHANGE THIS (passwords and file paths).
  • It logs to a different file than the original script. This writes to its own log file, hmailserver_awl_update_<date>.log and implements locking to prevent two threads writing to the file at the same time (this slows it down a little but stops errors).
  • It also implements locking around the hMailserver whitelist updates which I found could cause hMailserver to crash under load (which again may slow it down a little but it should be a lot more stable). (see my message above for more details about that one).
It should however work fine (I've been using this updated version for months now with no errors on hMailServer 5.3 B1879).
Let me know how you get on :)

Thanks,
Phil
Attachments
EventHandlers_awl_update.zip
My updated whitelist updater script file
(2.91 KiB) Downloaded 441 times

bathroomfloormatt
New user
New user
Posts: 15
Joined: 2011-01-19 16:39

Re: Automatic whitelisting

Post by bathroomfloormatt » 2011-01-24 17:04

Yipeeeee - well sort of!!!

Thanks phil, uploaded that version of "eventhandlers_awl_update.vbs" you sent and changed the bits you said to.

bad news first:

if i sent an email through hmail it doesnt get added to the whitlist in hmail and i get the same line in the error log.

good news:

however if i run the script (eventhandlers_awl_update.vbs) from cmd line like you said then it takes all the addresses in the "emailaddresses.txt" file and adds them to the hmail whitelist - yes i go into the hmail admin console and they are all there. whereas if i ran the old version i got the obapp error, but dont on your version.

so any ideas on why hmail cant run the script, but i can manually?

thanks so far.
Matt.

philg
Normal user
Normal user
Posts: 38
Joined: 2008-05-03 15:49

Re: Automatic whitelisting

Post by philg » 2011-01-24 17:27

I guess that's something to do with the original error....
Were there any errors logged?

Where you have WshShell.run vbs_update_wl, 1, false in the main eventhandler vbs, you could try adding (on the line before or thereabouts):

Code: Select all

write_log ("    spawning new awl update process ("& vbs_update_wl &")")
before it (as I have) then you can see if it is even trying to run the second VBS script.

The thing is, if that does not work, you should get an error which you didn't mention.

Lets have a look at the relevant bits of hmailserver_event_ & hmailserver_ log files - is there any indication of an attempt to call the awl update script? Also is there anything in the hmailserver_awl_update_ log if it exists (if you ran it manually then there will be some log but is there anything from the run that didnt work)?

Also are there any ERROR_hmailserver logs from today?

Its possible that I have missed something in my rush to provide my modified version for you to test! My main event handler script is modded even more so I'd rather not attempt to post that publicly (maybe one day!)

Thanks,
Phil
Last edited by philg on 2011-01-24 17:32, edited 1 time in total.

philg
Normal user
Normal user
Posts: 38
Joined: 2008-05-03 15:49

Re: Automatic whitelisting

Post by philg » 2011-01-24 17:31

Just to check, you do have the correct path and name set in vbs_update_wl at the top of the main script don't you?

For me its "D:\Progra~1\hMailServer\Events\EventHandlers_awl_update.vbs"
I just noticed that I've used short paths (progra~1) - maybe there is a long file name issue. You could check that too.

I modded my script at about line 155 - where your wshShell.run is as follows (basically more logging)

Code: Select all

		If upd = True Then
			write_log ("    spawning new awl update process ("& vbs_update_wl &")")
			if fs.FileExists(vbs_update_wl) then
				WshShell.run vbs_update_wl, 1, false
			else
				write_log ("      Error, cannot find: " & vbs_update_wl)
			end if
		End If
Thanks,
Phil

philg
Normal user
Normal user
Posts: 38
Joined: 2008-05-03 15:49

Re: Automatic whitelisting

Post by philg » 2011-01-24 21:43

I just realised I missed a part of your message saying:
and i get the same line in the error log.
In some ways that might be good. There is one thing that springs to mind immediately. Can you check if setting CScript to be the default script processor helps (as opposed to WScript). to do this, open a command prompt, and then type

Code: Select all

CScript /H:CScript
It should come back saying the default is not CScript. Then try sending that mail again. If you get the same error (and my previous two messages didn't help ;)) let me know and I will dig a bit deeper into the changes I made here.

Thanks,
Phil

bathroomfloormatt
New user
New user
Posts: 15
Joined: 2011-01-19 16:39

Re: Automatic whitelisting

Post by bathroomfloormatt » 2011-01-25 14:00

Thanks Phil,

im going to try what you have suggested and will report back.

Thanks again.
Matt.

bathroomfloormatt
New user
New user
Posts: 15
Joined: 2011-01-19 16:39

Re: Automatic whitelisting

Post by bathroomfloormatt » 2011-01-25 14:20

philg wrote:I guess that's something to do with the original error....
Were there any errors logged?

Where you have WshShell.run vbs_update_wl, 1, false in the main eventhandler vbs, you could try adding (on the line before or thereabouts):

Code: Select all

write_log ("    spawning new awl update process ("& vbs_update_wl &")")
before it (as I have) then you can see if it is even trying to run the second VBS script.

The thing is, if that does not work, you should get an error which you didn't mention.

Lets have a look at the relevant bits of hmailserver_event_ & hmailserver_ log files - is there any indication of an attempt to call the awl update script? Also is there anything in the hmailserver_awl_update_ log if it exists (if you ran it manually then there will be some log but is there anything from the run that didnt work)?

Also are there any ERROR_hmailserver logs from today?

Its possible that I have missed something in my rush to provide my modified version for you to test! My main event handler script is modded even more so I'd rather not attempt to post that publicly (maybe one day!)

Thanks,
Phil
ok have added in the bit to make it log when it tries to run the update and i know get this:

"25/01/2011 12:05:45" "-->User has authenticated. User , Client 10.59.1.203"
"25/01/2011 12:05:45" " Registering recipient addresses."
"25/01/2011 12:05:45" " adding new line with xxx@xxx.myzen.co.uk"
"25/01/2011 12:05:45" " writing emailaddressesfile"
"25/01/2011 12:05:45" " execute update"
"25/01/2011 12:05:45" " spawning new awl update process (C:\Program Files\hMailServer\Events\EventHandlers_awl_update.vbs)"

and obviously i get the entry in the error log:

"ERROR" 1432 "2011-01-25 12:05:45.761" "Script Error: Source: (null) - Error: 80070002 - Description: (null) - Line: 123 Column: 2 - Code: (null)"

but on line 123 as i added in another line of code to do the logging.

here is a print screen of the script which shows the correct location of eventhandlers_awl_update.vbs - i think, or im going to look a fool!!!

Image

will reply to your post about cscript next.

Thanks
Matt.

Post Reply