Simple Auto Whitelist

Use this forum if you have problems with a hMailServer script, such as hMailServer WebAdmin or code in an event handler.
Post Reply
palinka
Senior user
Senior user
Posts: 2475
Joined: 2017-09-12 17:57

Simple Auto Whitelist

Post by palinka » 2020-12-30 01:51

I've been giving auto whitelisting some thought. I looked at one in the scripting forum and thought, dang, that's complicated. It doesn't need to be.

The concept behind auto whitelisting is if a sender was a recipient in the past (of the current recipient), then the sender must be valid. This information is available in the hmailserver database. Why create new tables or text files to hold this info?

Here's my go at it. I recycled Soren's whitelisting function to make it work. The function deletes any reference to spam score, therefore bringing the spam score to 0 if there was a score.

All my function does is check hm_message_metadata to see if current recipient has sent N messages to the current sender in the past X days. If true, it calls Soren's whitelist function.

It does NOT modify hMailServer whitelist. Relies on MySQL database, so other databases would need a different query and connection.

Code: Select all

Sub WhiteList(oMessage, strMatch)
    Dim i
    If (oMessage.HeaderValue("X-hMailServer-Spam") = "YES") Then
        oMessage.HeaderValue("X-hMailServer-WhiteList") = strMatch
        oMessage.Headers.ItemByName("X-hMailServer-Spam").Delete
        For i = 0 To 10
            If (oMessage.HeaderValue("X-hMailServer-Reason-" & i) <> "") Then oMessage.Headers.ItemByName("X-hMailServer-Reason-" & i).Delete
        Next
        oMessage.Headers.ItemByName("X-hMailServer-Reason-Score").Delete
        oMessage.Save
    End If
End Sub

Function IsSenderAutoWhitelisted(m_Sender, m_Recipient) : IsSenderAutoWhitelisted = False
	Dim LookBackDays : LookBackDays = 90  '<--- Number of days to look back to see if sender was recipient
	Dim MinSent : MinSent = 3             '<--- Min number of messages previously sent to sender in order to whitelist sender
	Dim CountSent : CountSent = 0
	Dim oRecord, oConn : Set oConn = CreateObject("ADODB.Connection")
	oConn.Open "Driver={MariaDB ODBC 3.1 Driver}; Server=localhost; Database=hmailserver; User=hmailserver; Password=supersecretpassword;"

	If oConn.State <> 1 Then
		EventLog.Write( "Sub WhitelistSender - ERROR: Could not connect to database" )
		CountSent = 0
		Exit Function
	End If

	Set oRecord = oConn.Execute("SELECT COUNT(*) AS count FROM hm_message_metadata WHERE metadata_dateutc > NOW() - INTERVAL " & LookBackDays & " DAY AND metadata_from REGEXP '" & m_Recipient & "' AND metadata_to REGEXP '" & m_Sender & "';")
	Do Until oRecord.EOF
		CountSent = oRecord("count")
		oRecord.MoveNext
	Loop
	oConn.Close
	Set oRecord = Nothing

	If CInt(CountSent) > MinSent Then IsSenderAutoWhitelisted = True
End Function

Sub OnAcceptMessage(oClient, oMessage)

	REM - Auto Whitelisting
	If oClient.Username = "" Then
		If IsSenderAutoWhitelisted(oMessage.FromAddress, oMessage.Recipients(0).OriginalAddress) Then
			Call WhiteList(oMessage, "AutoWhitelist From = '" & oMessage.FromAddress & "'")
			Exit Sub
		End If
	End If

End Sub
Last edited by palinka on 2020-12-30 02:08, edited 1 time in total.

User avatar
jimimaseye
Moderator
Moderator
Posts: 8917
Joined: 2011-09-08 17:48

Re: Simple Auto Whitelist

Post by jimimaseye » 2020-12-30 02:01

There is a script around here somewhere that does as you describe. I think it was written by soren.

Maybe worth comparing.

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 2475
Joined: 2017-09-12 17:57

Re: Simple Auto Whitelist

Post by palinka » 2020-12-30 02:09

jimimaseye wrote:
2020-12-30 02:01
There is a script around here somewhere that does as you describe. I think it was written by soren.

Maybe worth comparing.

[Entered by mobile. Excuse my spelling.]
I looked. Didn't see it. My google-fu is poor. :oops:

User avatar
SorenR
Senior user
Senior user
Posts: 4199
Joined: 2006-08-21 15:38
Location: Denmark

Re: Simple Auto Whitelist

Post by SorenR » 2020-12-30 03:01

jimimaseye wrote:
2020-12-30 02:01
There is a script around here somewhere that does as you describe. I think it was written by soren.

Maybe worth comparing.

[Entered by mobile. Excuse my spelling.]
IIRC it was a dynamic Greylist Whitelisting ... https://www.hmailserver.com/forum/viewt ... 49#p205049
SørenR.

Algorithm (noun.)
Word used by programmers when they do not want to explain what they did.

palinka
Senior user
Senior user
Posts: 2475
Joined: 2017-09-12 17:57

Re: Simple Auto Whitelist

Post by palinka » 2020-12-30 04:15

SorenR wrote:
2020-12-30 03:01
jimimaseye wrote:
2020-12-30 02:01
There is a script around here somewhere that does as you describe. I think it was written by soren.

Maybe worth comparing.

[Entered by mobile. Excuse my spelling.]
IIRC it was a dynamic Greylist Whitelisting ... https://www.hmailserver.com/forum/viewt ... 49#p205049
So it is kinda sorta original! :mrgreen:

User avatar
katip
Senior user
Senior user
Posts: 815
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Simple Auto Whitelist

Post by katip » 2020-12-30 05:03

jimimaseye wrote:
2020-12-30 02:01
There is a script around here somewhere that does as you describe. I think it was written by soren.

Maybe worth comparing.

[Entered by mobile. Excuse my spelling.]
probably this by Percepts here : https://www.hmailserver.com/forum/viewt ... ts#p184106
Katip
--
HMS 5.7.0 x64, MariaDB 10.4.10 x64, SA 3.4.2, ClamAV 0.101.2 + SaneS

User avatar
mattg
Moderator
Moderator
Posts: 21268
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Simple Auto Whitelist

Post by mattg » 2020-12-30 05:19

palinka wrote:
2020-12-30 01:51
The concept behind auto whitelisting is if a sender was a recipient in the past (of the current recipient), then the sender must be valid. This information is available in the hmailserver database. Why create new tables or text files to hold this info?
I like the simplicity of this concept
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
katip
Senior user
Senior user
Posts: 815
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Simple Auto Whitelist

Post by katip » 2020-12-30 05:57

mattg wrote:
2020-12-30 05:19
I like the simplicity of this concept
concept is useful and serves me fine since many years.
however in case of BATV (fortunately not so common yet) it's useless. i add such Froms to SA whitelist - just as a workaround, not wise.
Katip
--
HMS 5.7.0 x64, MariaDB 10.4.10 x64, SA 3.4.2, ClamAV 0.101.2 + SaneS

User avatar
jimimaseye
Moderator
Moderator
Posts: 8917
Joined: 2011-09-08 17:48

Re: Simple Auto Whitelist

Post by jimimaseye » 2020-12-30 10:15

jimimaseye wrote:
2020-12-30 02:01
There is a script around here somewhere that does as you describe. I think it was written by soren.

Maybe worth comparing.

[Entered by mobile. Excuse my spelling.]
Perhaps I was thinking of this but you will already be aware of it : https://www.hmailserver.com/forum/viewt ... 20&t=33602

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

palinka
Senior user
Senior user
Posts: 2475
Joined: 2017-09-12 17:57

Re: Simple Auto Whitelist

Post by palinka » 2020-12-30 16:17

jimimaseye wrote:
2020-12-30 10:15
jimimaseye wrote:
2020-12-30 02:01
There is a script around here somewhere that does as you describe. I think it was written by soren.

Maybe worth comparing.

[Entered by mobile. Excuse my spelling.]
Perhaps I was thinking of this but you will already be aware of it : https://www.hmailserver.com/forum/viewt ... 20&t=33602

[Entered by mobile. Excuse my spelling.]
This one is manual white list (and blacklist, and others, etc). Not automatic.

palinka
Senior user
Senior user
Posts: 2475
Joined: 2017-09-12 17:57

Re: Simple Auto Whitelist

Post by palinka » 2020-12-30 16:18

katip wrote:
2020-12-30 05:57
mattg wrote:
2020-12-30 05:19
I like the simplicity of this concept
concept is useful and serves me fine since many years.
however in case of BATV (fortunately not so common yet) it's useless. i add such Froms to SA whitelist - just as a workaround, not wise.
What is BATV?

palinka
Senior user
Senior user
Posts: 2475
Joined: 2017-09-12 17:57

Re: Simple Auto Whitelist

Post by palinka » 2020-12-30 16:29

katip wrote:
2020-12-30 05:03
jimimaseye wrote:
2020-12-30 02:01
There is a script around here somewhere that does as you describe. I think it was written by soren.

Maybe worth comparing.

[Entered by mobile. Excuse my spelling.]
probably this by Percepts here : https://www.hmailserver.com/forum/viewt ... ts#p184106
This is a good one, but it adds every single recipient to the whitelist. I figured more than one message sent should be required before adding to the whitelist.

Adding my multiple sent check to percept's script would be a snap. But i didn't want to even touch the whitelist, with the idea that a) it should be reserved for "real" or problematic entries, and b) hey, it's not as simple and this is the ez-bake version. :D

User avatar
SorenR
Senior user
Senior user
Posts: 4199
Joined: 2006-08-21 15:38
Location: Denmark

Re: Simple Auto Whitelist

Post by SorenR » 2020-12-30 17:04

palinka wrote:
2020-12-30 16:18
katip wrote:
2020-12-30 05:57
mattg wrote:
2020-12-30 05:19
I like the simplicity of this concept
concept is useful and serves me fine since many years.
however in case of BATV (fortunately not so common yet) it's useless. i add such Froms to SA whitelist - just as a workaround, not wise.
What is BATV?
https://en.wikipedia.org/wiki/Bounce_Ad ... Validation
SørenR.

Algorithm (noun.)
Word used by programmers when they do not want to explain what they did.

palinka
Senior user
Senior user
Posts: 2475
Joined: 2017-09-12 17:57

Re: Simple Auto Whitelist

Post by palinka » 2020-12-30 20:34

SorenR wrote:
2020-12-30 17:04
palinka wrote:
2020-12-30 16:18
katip wrote:
2020-12-30 05:57


concept is useful and serves me fine since many years.
however in case of BATV (fortunately not so common yet) it's useless. i add such Froms to SA whitelist - just as a workaround, not wise.
What is BATV?
https://en.wikipedia.org/wiki/Bounce_Ad ... Validation
Ahhh... yeah, no bounce addresses would work with this, so marketing email would be out. But who replies to that anyway?

User avatar
katip
Senior user
Senior user
Posts: 815
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Simple Auto Whitelist

Post by katip » 2020-12-31 05:36

palinka wrote:
2020-12-30 20:34
Ahhh... yeah, no bounce addresses would work with this, so marketing email would be out. But who replies to that anyway?
not only marketing crap. some of our customers (particularly biggers such as P&G, Skoda, Electrolux...) use it too.
recently i noticed one message from an agent with whom we exchange mail since 10 years (so in W/L of course), suddenly blocked by SA due to an attachment. it came out they changed their ISP who stamps mails with BATV.
i maintain a small section of 10-15 "whitelist_from_received" in SA W/L for such cases. other than that i'm very satisfied with auto-whitelist concept. it saves lots of CPU.
Katip
--
HMS 5.7.0 x64, MariaDB 10.4.10 x64, SA 3.4.2, ClamAV 0.101.2 + SaneS

Post Reply