Issue related to hMailServer_CatchSpam

Use this forum for discussions about SpamAssassin and anti-spam in general.
Post Reply
ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-08-22 08:21

Hi,

I found this thread viewtopic.php?f=22&p=232225#p232225 which I think worth to try and implement but I was having difficulties when running the PublicSuffixLoad.ps1 on windows powershell from Palinka's script https://github.com/palinkas-jo-reggelt/ ... _CatchSpam.

Got some error:

Code: Select all

PS C:\Users\Administrator> C:\Users\Administrator\Documents\PublicSuffixLoad.ps1
Set-Content : A parameter cannot be found that matches parameter name 'NoNewline'.
At C:\Users\Administrator\Documents\PublicSuffixLoad.ps1:51 char:70
+ (Get-Content -Path $CondensedDatList) -Replace '$','|' | Set-Content -NoNewline  ...
+                                                                      ~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Set-Content], ParameterBindingException
    + FullyQualifiedErrorId : NamedParameterNotFound,Microsoft.PowerShell.Commands.SetContentCommand

Set-Content : A parameter cannot be found that matches parameter name 'NoNewline'.
At C:\Users\Administrator\Documents\PublicSuffixLoad.ps1:53 char:102
+ ...  | Set-Content -NoNewline -Path $CondensedDatList
+                    ~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Set-Content], ParameterBindingException
    + FullyQualifiedErrorId : NamedParameterNotFound,Microsoft.PowerShell.Commands.SetContentCommand

Set-Content : A parameter cannot be found that matches parameter name 'NoNewline'.
At C:\Users\Administrator\Documents\PublicSuffixLoad.ps1:54 char:71
+ (Get-Content -Path $CondensedDatList) -Replace '\|$','' | Set-Content -NoNewline ...
+                                                                       ~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Set-Content], ParameterBindingException
    + FullyQualifiedErrorId : NamedParameterNotFound,Microsoft.PowerShell.Commands.SetContentCommand

Set-Content : A parameter cannot be found that matches parameter name 'NoNewline'.
At C:\Users\Administrator\Documents\PublicSuffixLoad.ps1:55 char:70
+ (Get-Content -Path $CondensedDatList) -Replace '$','"' | Set-Content -NoNewline  ...
+                                                                      ~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Set-Content], ParameterBindingException
    + FullyQualifiedErrorId : NamedParameterNotFound,Microsoft.PowerShell.Commands.SetContentCommand

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-08-23 09:39

You have old powershell? I think NoNewLine only works with ver 5+.

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-08-23 09:46

https://github.com/danielbohannon/Revok ... -320339554

Possible fix. I'm on vacation so I can’t test anything.

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-08-23 18:18

I upgraded powershell to version 5 and was able to run the script. However, I got an error from my logs related to the script on EventHandlers.vbs

Code: Select all

"ERROR"	4900	"2021-08-23 19:14:52.483"	"Script Error: Source: Microsoft VBScript runtime error - Error: 800A000D - Description: Type mismatch: 'Include' - Line: 601 Column: 1 - Code: (null)"
"ERROR"	4896	"2021-08-23 19:06:05.615"	"Script Error: Source: Microsoft VBScript runtime error - Error: 800A01F4 - Description: Variable is undefined: 'Include' - Line: 601 Column: 1 - Code: (null)"
It is referring to this code:

Code: Select all

Function GetMainDomain(strDomain)
	Dim strRegEx, Match, Matches
	Dim TestDomain, DomainParts, a, i, PubSuffMatch, Include
	Include("C:\scripts\hmailserver\FWBan\PublicSuffix\public_suffix_list.vbs")
	
	DomainParts = Split(strDomain,".")
	a = UBound(DomainParts)
	If a > 1 Then
		TestDomain = DomainParts(1)
		For i = 2 to a
			TestDomain = TestDomain & "." & DomainParts(i)
		Next
	ElseIf a = 1 Then
		TestDomain = DomainParts(1)
	Else
		Exit Function
	End If

	Set Matches = oLookup(PubSufRegEx, TestDomain, False)
	For Each Match In Matches
		PubSuffMatch = True
	Next

	If PubSuffMatch Then 
		GetMainDomain = DomainParts(0) & "." & TestDomain
	Else
		GetMainDomain = GetMainDomain(TestDomain)
	End If
End Function

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-08-23 22:30

ashtec014 wrote:
2021-08-23 18:18
I upgraded powershell to version 5 and was able to run the script. However, I got an error from my logs related to the script on EventHandlers.vbs

Code: Select all

"ERROR"	4900	"2021-08-23 19:14:52.483"	"Script Error: Source: Microsoft VBScript runtime error - Error: 800A000D - Description: Type mismatch: 'Include' - Line: 601 Column: 1 - Code: (null)"
"ERROR"	4896	"2021-08-23 19:06:05.615"	"Script Error: Source: Microsoft VBScript runtime error - Error: 800A01F4 - Description: Variable is undefined: 'Include' - Line: 601 Column: 1 - Code: (null)"
Looks like i forgot to include function include. You'll have to find it here somewhere (because I'm still on vacation). Goolag "site:hmailserver.com sorenr function include"

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-08-24 08:10

I found the include function and error has gone:

Code: Select all

Function Include(sInstFile)
      Dim f, s, oFSO
      Set oFSO = CreateObject("Scripting.FileSystemObject")
      On Error Resume Next
      If oFSO.FileExists(sInstFile) Then
         Set f = oFSO.OpenTextFile(sInstFile)
         s = f.ReadAll
         f.Close
         ExecuteGlobal s
      End If
      On Error Goto 0
      Set f = Nothing
      Set oFSO = Nothing
   End Function
I got another error after running it:

Code: Select all

"ERROR"	9592	"2021-08-24 09:01:43.871"	"Script Error: Source: Microsoft VBScript runtime error - Error: 800A000D - Description: Type mismatch: 'oLookup' - Line: 352 Column: 3 - Code: (null)"
"ERROR"	9516	"2021-08-24 09:01:47.362"	"Script Error: Source: Microsoft OLE DB Provider for ODBC Drivers - Error: 80004005 - Description: [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified - Line: 580 Column: 4 - Code: (null)"
The first error looks like from this code:

Code: Select all

Dim strRegEx, Match, Matches, PTR_Record

	REM	- Grab PTR-Record
	PTR_Record = PTRLookup(oClient.IPAddress)

	REM - Test Whitelist (0 = Not Listed, 1 = Whitelisted)
	Dim IsWhitelisted : IsWhitelisted = Whitelisted(oClient.IPAddress)

	REM - Record entries for CatchSpam
	Dim spamDomain, SAScore, oLookup
	If IsWhitelisted = 0 Then
		If oMessage.HeaderValue("X-hMailServer-Reason-Score") <> "" Then 
			strRegEx = "[0-9]{1,3}"
			Set Matches = oLookup(strRegEx, oMessage.HeaderValue("X-hMailServer-Reason-Score"), False)
			For Each Match In Matches
				SAScore = Match.Value
			Next
			
			REM - If SAScore greater than DELETE THRESHOLD - use hMailServer delete threshold score
			REM - CatchSpam should only be applied to messages that should be deleted : its used to reject the client outright
			If (CInt(SAScore) > 6) Then 
				spamDomain = GetMainDomain(PTR_Record)
				' EventLog.Write( "Spam Received: Score = " & CInt(SAScore) & ", PTR = " & PTR_Record & " Domain = " & spamDomain )
				If spamDomain <> "" Then 
					Call CatchSpam(spamDomain)
					'
					' Anything else you want to do
					'
				End If
			End If
		End If
	End If
The 2nd one is related to ODBC connection which I already changed from MariaDB to MySQL.

Code: Select all

Function IsCatchSpam(spamDomain) : IsCatchSpam = False
	Dim m_CountDomain, m_SafeDomain
    Dim oRecord, oConn : Set oConn = CreateObject("ADODB.Connection")
    oConn.Open "Driver={MySQL ODBC 5.5 Driver}; Server=localhost; Database=mydb; User=user; Password=myPassword;"

    If oConn.State <> 1 Then
		EventLog.Write( "Function IsCatchSpam - ERROR: Could not connect to database" )
        Exit Function
    End If

    Set oRecord = oConn.Execute("SELECT hits,safe FROM hm_catchspam WHERE domain = '" & spamDomain & "'")
    Do Until oRecord.EOF
        m_CountDomain = oRecord("hits")
        m_SafeDomain = oRecord("safe")
        oRecord.MoveNext
    Loop
    oConn.Close
    Set oRecord = Nothing
	If (CInt(m_CountDomain) > 2) And (m_SafeDomain = 0) Then IsCatchSpam = True
End Function

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-08-24 08:27

ashtec014 wrote:
2021-08-24 08:10
I got another error after running it:

Code: Select all

"ERROR"	9592	"2021-08-24 09:01:43.871"	"Script Error: Source: Microsoft VBScript runtime error - Error: 800A000D - Description: Type mismatch: 'oLookup' - Line: 352 Column: 3 - Code: (null)"
"ERROR"	9516	"2021-08-24 09:01:47.362"	"Script Error: Source: Microsoft OLE DB Provider for ODBC Drivers - Error: 80004005 - Description: [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified - Line: 580 Column: 4 - Code: (null)"
For Pete's sake! I forgot the lookup functions too? :oops:

Get Soren's function lookup *AND* function olookup.

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-08-24 09:14

Thank you Palinka. Error has gone for oLookup. But the error for Microsoft OLE DB Provider for ODBC drivers are still present.

Code: Select all

"ERROR"	9840	"2021-08-24 10:13:11.888"	"Script Error: Source: Microsoft OLE DB Provider for ODBC Drivers - Error: 80004005 - Description: [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified - Line: 589 Column: 4 - Code: (null)"

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-08-24 09:21

It's very difficult for me to help you troubleshoot the odbc error because so much is in your environment. Some general suggestions:

* If you have a working "select" function in your eventhandlers.vbs then copy the connection string from that.

* You need a working ODBC connector for your database.

* Make sure the DSN is character-for-character exactly the same as what's shown in the windows ODBC dialog box.

* Remember, there are 64 and 32 bit odbc connections. Try both if one doesn't work as expected.

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-08-24 10:23

I got it working.
* Make sure the DSN is character-for-character exactly the same as what's shown in the windows ODBC dialog box.
I forgot to include the word 'ANSI'.

Code: Select all

"Driver={MySQL ODBC 5.3 ANSI Driver}; Server=localhost; Database=myDB; User=user; Password=myPassword;"
No errors shown after running it. Looks like everything is okay. So far no catch yet on my web file. Thanks much Palinka! Appreciate your help 🙏

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-08-24 13:36

Ok cool. You may not get any hits for a while, or maybe never.

I had a run of these particular type of spammers: valid domain, PTR, SPF, etc. They would pump spam until spamhaus listed them usually days later. The other thing is there would be several subdomains of the same domain also with valid everything (hence the "getmaindomain" function). Spamassassin would pick up on it usually, but the other, regular defences. That's why i created this script - so i could identify and block them.

I've had these guys try to blast me with spam, get blocked and then nothing for months. Then different ones start up and get blocked, then nothing for months. That's how they go.

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-08-24 17:27

I've not seen any data yet as of this writing. But surely will keep monitoring and check if there's any hit and how it works. I've check yours from this link https://firewallban.dynu.net/ and looks like last hit was May 2021. So, I'm guessing it would take that long for me to see it as well. 😊

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-08-25 07:25

Screenshot_20210825-071404_Brave.jpg

Actually, i haven't implemented the firewall ban since i rebuilt my server after a HD crash a few months ago. In July i had a run of these bans.

There's a "safe" option in case you get hits from legit MTAs. So far, I've only had to mark sendgrid as safe. Safe = ignore. To further prevent false positives, i have the whitelist function: if whitelisted, then skip catchspam checks. Sendgrid is kind of unique in that regard. There's a lot of spam on dedicated servers with domains ending with sendgrid.net. They could be blacklisted while other sendgrid.net servers are not. Since catchspam works with domains and not IPs, that's an important distinction.

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-08-25 10:00

Nice catch! But how to mark "safe or ignore" the domain in case of false positive? Is there an option as well to the web UI? I can't check it for now since I haven't seen any data coming in yet.

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-08-25 10:33

Yes, in the web UI. Click on the domain and a window pops up with option to mark safe or unsafe.

Also in the web UI, yellow is 2 hits, red is blocked (3 hits). :D

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-08-28 07:33

I got an error from logs related to MySQL. Do you have any idea what it mean?

Code: Select all

"ERROR"	14580	"2021-08-27 09:09:20.941"	"Severity: 2 (High), Code: HM5032, Source: DALConnection::Execute, Description: MySQL: Field 'safe' doesn't have a default value (Additional info: INSERT INTO hm_catchspam (domain,hits) VALUES ('ingesoft.net',1) ON DUPLICATE KEY UPDATE hits=(hits+1),timestamp=NOW();)"
"ERROR"	14580	"2021-08-27 09:09:20.941"	"Script Error: Source: hMailServer COM library - Error: 800403E9 - Description: Execution of SQL statement failed. Error: MySQL: Field 'safe' doesn't have a default value (Additional info: INSERT INTO hm_catchspam (domain,hits) VALUES ('ingesoft.net',1) ON DUPLICATE KEY UPDATE hits=(hits+1),timestamp=NOW();) - Line: 584 Column: 1 - Code: (null)"
"ERROR"	14580	"2021-08-27 17:07:26.663"	"Severity: 2 (High), Code: HM5032, Source: DALConnection::Execute, Description: MySQL: Field 'safe' doesn't have a default value (Additional info: INSERT INTO hm_catchspam (domain,hits) VALUES ('inep.co.rs',1) ON DUPLICATE KEY UPDATE hits=(hits+1),timestamp=NOW();)"
"ERROR"	14580	"2021-08-27 17:07:26.663"	"Script Error: Source: hMailServer COM library - Error: 800403E9 - Description: Execution of SQL statement failed. Error: MySQL: Field 'safe' doesn't have a default value (Additional info: INSERT INTO hm_catchspam (domain,hits) VALUES ('inep.co.rs',1) ON DUPLICATE KEY UPDATE hits=(hits+1),timestamp=NOW();) - Line: 584 Column: 1 - Code: (null)"

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-08-28 20:56

ashtec014 wrote:
2021-08-28 07:33
I got an error from logs related to MySQL. Do you have any idea what it mean?

Code: Select all

"ERROR"	14580	"2021-08-27 09:09:20.941"	"Severity: 2 (High), Code: HM5032, Source: DALConnection::Execute, Description: MySQL: Field 'safe' doesn't have a default value (Additional info: INSERT INTO hm_catchspam (domain,hits) VALUES ('ingesoft.net',1) ON DUPLICATE KEY UPDATE hits=(hits+1),timestamp=NOW();)"
"ERROR"	14580	"2021-08-27 09:09:20.941"	"Script Error: Source: hMailServer COM library - Error: 800403E9 - Description: Execution of SQL statement failed. Error: MySQL: Field 'safe' doesn't have a default value (Additional info: INSERT INTO hm_catchspam (domain,hits) VALUES ('ingesoft.net',1) ON DUPLICATE KEY UPDATE hits=(hits+1),timestamp=NOW();) - Line: 584 Column: 1 - Code: (null)"
"ERROR"	14580	"2021-08-27 17:07:26.663"	"Severity: 2 (High), Code: HM5032, Source: DALConnection::Execute, Description: MySQL: Field 'safe' doesn't have a default value (Additional info: INSERT INTO hm_catchspam (domain,hits) VALUES ('inep.co.rs',1) ON DUPLICATE KEY UPDATE hits=(hits+1),timestamp=NOW();)"
"ERROR"	14580	"2021-08-27 17:07:26.663"	"Script Error: Source: hMailServer COM library - Error: 800403E9 - Description: Execution of SQL statement failed. Error: MySQL: Field 'safe' doesn't have a default value (Additional info: INSERT INTO hm_catchspam (domain,hits) VALUES ('inep.co.rs',1) ON DUPLICATE KEY UPDATE hits=(hits+1),timestamp=NOW();) - Line: 584 Column: 1 - Code: (null)"
Use phpmyadmin or other mysql administrator to set the default value of column "safe" to 0. I'll look at changing the sql later.

I'm surprised NOT NULL doesn't automatically define a default value of 0 since the column is an integer.

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-08-29 07:31

palinka wrote:
2021-08-28 20:56
Use phpmyadmin or other mysql administrator to set the default value of column "safe" to 0. I'll look at changing the sql later.

I'm surprised NOT NULL doesn't automatically define a default value of 0 since the column is an integer.
Tried to insert value '0' under column safe but got an error when executing it:

Code: Select all

Executing:
INSERT INTO `MyMailDB`.`hm_catchspam` (`safe`) VALUES ('0');

Operation failed: There was an error while applying the SQL script to the database.
ERROR 1364: 1364: Field 'domain' doesn't have a default value
SQL Statement:
INSERT INTO `MyMailDB`.`hm_catchspam` (`safe`) VALUES ('0')

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-08-29 08:10

Insert statement won't work because there has to be a value for each column. Use update instead.

UPDATE `MyMailDB`.`hm_catchspam` SET `safe` = 0;

Better yet, use ALTER to define a default value for 'safe'. You'll have to look that up. I don't know the syntax of the top of my head and i won't be near a computer for a couple weeks still. :mrgreen:

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-08-29 08:15

ALTER TABLE `MyMailDB`.`hm_catchspam` MODIFY COLUMN safe INT(1) NOT NULL DEFAULT 0;

Actually after a quick goolag search, this should do the trick.

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-08-29 08:21

Awesome! it works now. The column under safe will now be filled automaticaly right?
Image

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-08-29 10:00

Yes, insert commands that omit safe should default safe to 0.

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-08-29 11:06

palinka wrote:
2021-08-29 10:00
Yes, insert commands that omit safe should default safe to 0.
Thanks much Palinka! I appreciate it.

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-09-25 07:48

Hi,

After almost a month of monitoring the logs as well as the status of spam captured by this script, I can say that it is very much helpful. In fact there are 9 domains captured already and I really appreciate for sharing this script. However, this past few days, I've seen this on my logs, looks like related to one of the code from the script.

Error logs:

Code: Select all

"ERROR"	15508	"2021-09-23 15:39:42.753"	"Severity: 2 (High), Code: HM5032, Source: DALConnection::Execute, Description: MySQL: Data too long for column 'domain' at row 1 (Additional info: INSERT INTO hm_catchspam (domain,hits) VALUES ('imohsohealthy.com[nl]mail.innovativepkg.com.ph',1) ON DUPLICATE KEY UPDATE hits=(hits+1),timestamp=NOW();)"
"ERROR"	15508	"2021-09-23 15:39:42.799"	"Script Error: Source: hMailServer COM library - Error: 800403E9 - Description: Execution of SQL statement failed. Error: MySQL: Data too long for column 'domain' at row 1 (Additional info: INSERT INTO hm_catchspam (domain,hits) VALUES ('imohsohealthy.com[nl]mail.innovativepkg.com.ph',1) ON DUPLICATE KEY UPDATE hits=(hits+1),timestamp=NOW();) - Line: 667 Column: 1 - Code: (null)"
Code from line 667:

Code: Select all

Function CatchSpam(spamDomain)
	Dim strSQL, oDB : Set oDB = GetDatabaseObject
	strSQL = "INSERT INTO hm_catchspam (domain,hits) VALUES ('" & spamDomain & "',1) ON DUPLICATE KEY UPDATE hits=(hits+1),timestamp=NOW();"
	Call oDB.ExecuteSQL(strSQL)
End Function

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-09-25 13:40

ashtec014 wrote:
2021-09-25 07:48

Code: Select all

'imohsohealthy.com[nl]mail.innovativepkg.com.ph'
'imohsohealthy.com[nl]mail.innovativepkg.com.ph'
That's very strange. There are 2 domains separated by new lines. Any chance you can find the message and post the received headers? I think its probably not an issue with the script, but rather an incoming message is malformed. I've never seen a situation like that before.

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-09-25 15:48

palinka wrote:
2021-09-25 13:40
That's very strange. There are 2 domains separated by new lines. Any chance you can find the message and post the received headers? I think its probably not an issue with the script, but rather an incoming message is malformed. I've never seen a situation like that before.
Unfortunately. I can't recover the message anymore as it was deleted already by a global rule because the message was a spam and exceeded threeshold I set. Since this is not an issue with the script I'll just leave it as it is for now and will continue monitoring if I found one in the future, I'll let you know. Thank you so much.

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-09-25 18:31

ashtec014 wrote:
2021-09-25 15:48
palinka wrote:
2021-09-25 13:40
That's very strange. There are 2 domains separated by new lines. Any chance you can find the message and post the received headers? I think its probably not an issue with the script, but rather an incoming message is malformed. I've never seen a situation like that before.
Unfortunately. I can't recover the message anymore as it was deleted already by a global rule because the message was a spam and exceeded threeshold I set. Since this is not an issue with the script I'll just leave it as it is for now and will continue monitoring if I found one in the future, I'll let you know. Thank you so much.
It only happened for that "domain", correct?

imohsohealthy.com[nl]mail.innovativepkg.com.ph

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-09-25 18:39

palinka wrote:
2021-09-25 18:31

It only happened for that "domain", correct?

imohsohealthy.com[nl]mail.innovativepkg.com.ph
That is correct.

Here's the spam domain captured so far:
Capture.PNG

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-09-25 18:49

ashtec014 wrote:
2021-09-25 18:39
palinka wrote:
2021-09-25 18:31

It only happened for that "domain", correct?

imohsohealthy.com[nl]mail.innovativepkg.com.ph
That is correct.

Here's the spam domain captured so far:
Capture.PNG
OK good. Then the most likely issue is the [nl] in the domain name. That must have been presented to hmailserver that way, which is a clear violation of the rfc.

You may be able to find the message in your smtp logs. Look for the ehlo entry and see if the [nl] is in the domain.

Also, you can force hmailserver to validate helo: viewtopic.php?p=209546#p209546

Code: Select all

   '
   '   Validate HELO/EHLO greeting
   '
   Const strFQDN = "^(?=^.{1,254}$)(^(?:(?!\.|-)([a-z0-9\-\*]{1,63}|([a-z0-9\-]{1,62}[a-z0-9]))\.)+(?:[a-z]{2,})$)$"
   Const strIPv4 = "^\[(?:[0-9]{1,3}\.){3}[0-9]{1,3}\]$"
   Const strIPv6 = "^\[(IPv6)((?:[0-9A-Fa-f]{0,4}:){1,7}(?:(?:(>25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)|[0-9A-Fa-f]{1,4}))\]$"
   strRegEx = strFQDN & "|" & strIPv4 & "|" & strIPv6
   If (Lookup(strRegEx, oClient.HELO) = False) Then
      Result.Value = 2
      Result.Message = "5.7.1 CODE03 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means."
      Call AutoBan(oClient.IPAddress, "Bad HELO - " & oClient.HELO, 7, "d")
      Exit Sub
   End If
This would boot a helo with a [nl] in it. It goes in Sub OnHELO if you're using a version that supports it.

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-09-26 09:40

palinka wrote:
2021-09-25 18:49
You may be able to find the message in your smtp logs. Look for the ehlo entry and see if the [nl] is in the domain.
Here's the SMTP logs at that time and can't find any [nl] during transmission.

Code: Select all

"TCPIP"	8144	"2021-09-23 15:37:52.501"	"TCP - 43.224.191.18 connected to 192.168.1.8:25."
"DEBUG"	8144	"2021-09-23 15:37:52.501"	"Executing event OnClientConnect"
"DEBUG"	8144	"2021-09-23 15:38:18.971"	"Event completed"
"DEBUG"	8144	"2021-09-23 15:38:18.971"	"TCP connection started for session 1344"
"SMTPD"	8144	1344	"2021-09-23 15:38:18.971"	"43.224.191.18"	"SENT: 220 mail.mydomain.com"
"SMTPD"	11104	1344	"2021-09-23 15:38:19.237"	"43.224.191.18"	"RECEIVED: EHLO mail.innovativepkg.com.ph"
"DEBUG"	11104	"2021-09-23 15:38:19.237"	"Executing event OnHELO"
"DEBUG"	11104	"2021-09-23 15:38:39.234"	"Event completed"
"SMTPD"	11104	1344	"2021-09-23 15:38:39.234"	"43.224.191.18"	"SENT: 250-mail.mydomain.com[nl]250-SIZE 64423936[nl]250-STARTTLS[nl]250 HELP"
"SMTPD"	15636	1344	"2021-09-23 15:38:39.499"	"43.224.191.18"	"RECEIVED: STARTTLS"
"SMTPD"	15636	1344	"2021-09-23 15:38:39.499"	"43.224.191.18"	"SENT: 220 Ready to start TLS"
"DEBUG"	11104	"2021-09-23 15:38:39.499"	"Performing SSL/TLS handshake for session 1344. Verify certificate: False"
"TCPIP"	11104	"2021-09-23 15:38:40.302"	"TCPConnection - TLS/SSL handshake completed. Session Id: 1344, Remote IP: 43.224.191.18, Version: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384, Bits: 256"
"SMTPD"	11104	1344	"2021-09-23 15:38:40.552"	"43.224.191.18"	"RECEIVED: EHLO mail.innovativepkg.com.ph"
"DEBUG"	11104	"2021-09-23 15:38:40.552"	"Executing event OnHELO"
"DEBUG"	8144	"2021-09-23 15:38:50.442"	"Pre-creating session 1348"
"TCPIP"	8144	"2021-09-23 15:38:50.442"	"TCP - 193.56.29.27 connected to 192.168.1.8:25."
"DEBUG"	8144	"2021-09-23 15:38:50.442"	"Executing event OnClientConnect"
"DEBUG"	11104	"2021-09-23 15:39:10.126"	"Event completed"
"SMTPD"	11104	1344	"2021-09-23 15:39:10.142"	"43.224.191.18"	"SENT: 250-mail.mydomain.com[nl]250-SIZE 64423936[nl]250 HELP"
"SMTPD"	15636	1344	"2021-09-23 15:39:10.392"	"43.224.191.18"	"RECEIVED: MAIL FROM:<officebackup198@gmail.com> SIZE=15345"
"TCPIP"	15636	"2021-09-23 15:39:10.720"	"DNS lookup: 18.191.224.43.zen.spamhaus.org, 0 addresses found: (none), Match: False"
"TCPIP"	15636	"2021-09-23 15:39:10.923"	"DNS lookup: 18.191.224.43.bl.spamcop.net, 0 addresses found: (none), Match: False"
"TCPIP"	15636	"2021-09-23 15:39:11.345"	"DNS lookup: 18.191.224.43.b.barracudacentral.org, 0 addresses found: (none), Match: False"
"TCPIP"	15636	"2021-09-23 15:39:11.345"	"DNS lookup: 18.191.224.43.hostkarma.junkemailfilter.com, 0 addresses found: (none), Match: False"
"TCPIP"	15636	"2021-09-23 15:39:11.751"	"DNS lookup: 18.191.224.43.bl.spameatingmonkey.net, 0 addresses found: (none), Match: False"
"TCPIP"	15636	"2021-09-23 15:39:12.048"	"DNS lookup: 18.191.224.43.cbl.abuseat.org, 0 addresses found: (none), Match: False"
"TCPIP"	15636	"2021-09-23 15:39:12.439"	"DNS lookup: 18.191.224.43.dnsbl.sorbs.net, 0 addresses found: (none), Match: False"
"TCPIP"	15636	"2021-09-23 15:39:12.751"	"DNS lookup: 18.191.224.43.sbl.spamhaus.org, 0 addresses found: (none), Match: False"
"DEBUG"	8144	"2021-09-23 15:39:13.095"	"Event completed"
"DEBUG"	8144	"2021-09-23 15:39:13.095"	"TCP connection started for session 1347"
"SMTPD"	8144	1347	"2021-09-23 15:39:13.095"	"193.56.29.27"	"SENT: 220 mail.mydomain.com"
"DEBUG"	11104	"2021-09-23 15:39:13.095"	"The read operation failed. Bytes transferred: 0 Remote IP: 193.56.29.27, Session: 1347, Code: 2, Message: End of file"
"DEBUG"	11104	"2021-09-23 15:39:13.095"	"Ending session 1347"
"TCPIP"	15636	"2021-09-23 15:39:13.173"	"DNS lookup: 18.191.224.43.dnsbl.spamdonkey.com, 1 addresses found: 127.0.0.3, Match: False"
"TCPIP"	15636	"2021-09-23 15:39:13.173"	"DNS lookup: 18.191.224.43.dnsbl.spamdonkey.com, 1 addresses found: 127.0.0.3, Match: False"
"TCPIP"	15636	"2021-09-23 15:39:13.189"	"DNS lookup: 18.191.224.43.dnsbl.spamdonkey.com, 1 addresses found: 127.0.0.3, Match: True"
"TCPIP"	15636	"2021-09-23 15:39:13.189"	"DNS lookup: 18.191.224.43.dnsbl.spamdonkey.com, 1 addresses found: 127.0.0.3, Match: False"
"TCPIP"	15636	"2021-09-23 15:39:13.189"	"DNS lookup: 18.191.224.43.dnsbl.spamdonkey.com, 1 addresses found: 127.0.0.3, Match: False"
"TCPIP"	15636	"2021-09-23 15:39:13.564"	"DNS lookup: 18.191.224.43.dnsbl-1.uceprotect.net, 0 addresses found: (none), Match: False"
"TCPIP"	15636	"2021-09-23 15:39:13.908"	"DNS lookup: 18.191.224.43.dnsbl-2.uceprotect.net, 0 addresses found: (none), Match: False"
"TCPIP"	15636	"2021-09-23 15:39:14.392"	"DNS lookup: 18.191.224.43.dnsbl-3.uceprotect.net, 0 addresses found: (none), Match: False"
"TCPIP"	15636	"2021-09-23 15:39:14.845"	"DNS lookup: 18.191.224.43.all.spamrats.com, 0 addresses found: (none), Match: False"
"DEBUG"	15636	"2021-09-23 15:39:14.845"	"Spam test: SpamTestDNSBlackLists, Score: 2"
"DEBUG"	15636	"2021-09-23 15:39:14.861"	"Spam test: SpamTestMXRecords, Score: 0"
"DEBUG"	15636	"2021-09-23 15:39:14.892"	"Spam test: SpamTestSPF, Score: 0"
"DEBUG"	15636	"2021-09-23 15:39:14.892"	"Total spam score: 2"
"SMTPD"	15636	1344	"2021-09-23 15:39:14.892"	"43.224.191.18"	"SENT: 250 OK"
"SMTPD"	11104	1344	"2021-09-23 15:39:15.142"	"43.224.191.18"	"RECEIVED: RCPT TO:<email@mydomain.com>"
"SMTPD"	11104	1344	"2021-09-23 15:39:15.142"	"43.224.191.18"	"SENT: 250 OK"
"SMTPD"	15636	1344	"2021-09-23 15:39:15.408"	"43.224.191.18"	"RECEIVED: DATA"
"DEBUG"	15636	"2021-09-23 15:39:15.408"	"Executing event OnSMTPData"
"DEBUG"	15636	"2021-09-23 15:39:35.200"	"Event completed"
"SMTPD"	15636	1344	"2021-09-23 15:39:35.200"	"43.224.191.18"	"SENT: 354 OK, send."
"DEBUG"	15636	"2021-09-23 15:39:35.981"	"Adding task AsynchronousTask to work queue Asynchronous task queue"
"DEBUG"	15508	"2021-09-23 15:39:35.981"	"Executing task AsynchronousTask in work queue Asynchronous task queue"
"DEBUG"	15508	"2021-09-23 15:39:35.981"	"SURBL: Execute"
"DEBUG"	15508	"2021-09-23 15:39:35.981"	"SURBL: Found URL: googleapis.com"
"DEBUG"	15508	"2021-09-23 15:39:35.981"	"SURBL: Found URL: w3.org"
"DEBUG"	15508	"2021-09-23 15:39:35.981"	"SURBL: 2 unique addresses found."
"DEBUG"	15508	"2021-09-23 15:39:35.981"	"SURBL: Lookup: googleapis.com.multi.surbl.org"
"DEBUG"	15508	"2021-09-23 15:39:36.153"	"SURBL: Lookup: w3.org.multi.surbl.org"
"DEBUG"	15508	"2021-09-23 15:39:36.325"	"SURBL: Match not found"
"DEBUG"	15508	"2021-09-23 15:39:36.325"	"SURBL: Execute"
"DEBUG"	15508	"2021-09-23 15:39:36.325"	"SURBL: Found URL: googleapis.com"
"DEBUG"	15508	"2021-09-23 15:39:36.325"	"SURBL: Found URL: w3.org"
"DEBUG"	15508	"2021-09-23 15:39:36.325"	"SURBL: 2 unique addresses found."
"DEBUG"	15508	"2021-09-23 15:39:36.325"	"SURBL: Lookup: googleapis.com.dbl.spamhaus.org"
"DEBUG"	15508	"2021-09-23 15:39:36.497"	"SURBL: Lookup: w3.org.dbl.spamhaus.org"
"DEBUG"	15508	"2021-09-23 15:39:36.684"	"SURBL: Match not found"
"DEBUG"	15508	"2021-09-23 15:39:36.684"	"SURBL: Execute"
"DEBUG"	15508	"2021-09-23 15:39:36.684"	"SURBL: Found URL: googleapis.com"
"DEBUG"	15508	"2021-09-23 15:39:36.684"	"SURBL: Found URL: w3.org"
"DEBUG"	15508	"2021-09-23 15:39:36.684"	"SURBL: 2 unique addresses found."
"DEBUG"	15508	"2021-09-23 15:39:36.684"	"SURBL: Lookup: googleapis.com.uribl.spameatingmonkey.net"
"DEBUG"	15508	"2021-09-23 15:39:37.075"	"SURBL: Lookup: w3.org.uribl.spameatingmonkey.net"
"DEBUG"	15508	"2021-09-23 15:39:37.450"	"SURBL: Match not found"
"DEBUG"	15508	"2021-09-23 15:39:37.450"	"Spam test: SpamTestSURBL, Score: 0"
"DEBUG"	15508	"2021-09-23 15:39:37.466"	"Pre-creating session 1349"
"TCPIP"	15508	"2021-09-23 15:39:37.466"	"Connecting to 127.0.0.1:783..."
"DEBUG"	15636	"2021-09-23 15:39:37.466"	"TCP connection started for session 1349"
"DEBUG"	15636	"2021-09-23 15:39:37.466"	"Sending message to SpamAssassin. Session 1349, File: C:\Program Files\hMailServer\Data\{72CF3C8E-AFA6-432E-82E1-E5F1E87613AE}.eml"
"DEBUG"	15636	"2021-09-23 15:39:40.002"	"Parsing response from SpamAssassin. Session 1349"
"DEBUG"	15636	"2021-09-23 15:39:40.018"	"SA - Copy+Delete used"
"DEBUG"	15636	"2021-09-23 15:39:40.018"	"Ending session 1349"
"DEBUG"	15508	"2021-09-23 15:39:40.018"	"Spam test: SpamTestSpamAssassin, Score: 5"
"DEBUG"	15508	"2021-09-23 15:39:40.018"	"Total spam score: 5"
"DEBUG"	15508	"2021-09-23 15:39:40.018"	"Executing event OnAcceptMessage"
"ERROR"	15508	"2021-09-23 15:39:42.753"	"Severity: 2 (High), Code: HM5032, Source: DALConnection::Execute, Description: MySQL: Data too long for column 'domain' at row 1 (Additional info: INSERT INTO hm_catchspam (domain,hits) VALUES ('imohsohealthy.com[nl]mail.innovativepkg.com.ph',1) ON DUPLICATE KEY UPDATE hits=(hits+1),timestamp=NOW();)"
"ERROR"	15508	"2021-09-23 15:39:42.799"	"Script Error: Source: hMailServer COM library - Error: 800403E9 - Description: Execution of SQL statement failed. Error: MySQL: Data too long for column 'domain' at row 1 (Additional info: INSERT INTO hm_catchspam (domain,hits) VALUES ('imohsohealthy.com[nl]mail.innovativepkg.com.ph',1) ON DUPLICATE KEY UPDATE hits=(hits+1),timestamp=NOW();) - Line: 667 Column: 1 - Code: (null)"
"DEBUG"	15508	"2021-09-23 15:39:42.799"	"Event completed"
"DEBUG"	15508	"2021-09-23 15:39:42.799"	"Saving message: {72CF3C8E-AFA6-432E-82E1-E5F1E87613AE}.eml"
"DEBUG"	15508	"2021-09-23 15:39:42.799"	"Requesting SMTPDeliveryManager to start message delivery"
"SMTPD"	15508	1344	"2021-09-23 15:39:42.799"	"43.224.191.18"	"SENT: 250 Queued (4.864 seconds)"
"DEBUG"	8104	"2021-09-23 15:39:42.799"	"Adding task DeliveryTask to work queue SMTP delivery queue"
"DEBUG"	7776	"2021-09-23 15:39:42.799"	"Executing task DeliveryTask in work queue SMTP delivery queue"
"DEBUG"	7776	"2021-09-23 15:39:42.799"	"Delivering message..."
"APPLICATION"	7776	"2021-09-23 15:39:42.815"	"SMTPDeliverer - Message 265807: Delivering message from officebackup198@gmail.com to email@mydomain.com. File: C:\Program Files\hMailServer\Data\{72CF3C8E-AFA6-432E-82E1-E5F1E87613AE}.eml"
"DEBUG"	7776	"2021-09-23 15:39:42.815"	"Connecting to ClamAV virus scanner..."
"SMTPD"	15636	1344	"2021-09-23 15:39:43.065"	"43.224.191.18"	"RECEIVED: QUIT"
"SMTPD"	15636	1344	"2021-09-23 15:39:43.065"	"43.224.191.18"	"SENT: 221 goodbye"
"DEBUG"	8144	"2021-09-23 15:39:43.065"	"Ending session 1344"
"DEBUG"	7776	"2021-09-23 15:39:43.815"	"Connecting to ClamAV stream port..."
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"No virus detected: stream: OK"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Applying rules"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Applying rule ExternalScore7"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Performing rule action"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Applying rule X-hMailServer-Reason-Score"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Applying rule ScanZipArchive"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Performing rule action"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Executing event Unknown"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Event completed"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Applying rule RemoveVirusEmail"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Applying rule sa-learn"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Performing rule action"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Performing rule action"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Copying mail contents"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Saving message: {9F12DA1E-AF69-4B23-BF5C-548343807BA1}.eml"
"APPLICATION"	7776	"2021-09-23 15:39:44.893"	"SMTPDeliverer - Message 265807: Message deleted. Action was taken by a global rule (Rule name: ExternalScore7, ID: 53). "
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"AWStats::LogDeliveryFailure"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Deleting message"
"DEBUG"	7776	"2021-09-23 15:39:44.893"	"Deleting message file."
palinka wrote:
2021-09-25 18:49

Also, you can force hmailserver to validate helo: viewtopic.php?p=209546#p209546

Code: Select all

   '
   '   Validate HELO/EHLO greeting
   '
   Const strFQDN = "^(?=^.{1,254}$)(^(?:(?!\.|-)([a-z0-9\-\*]{1,63}|([a-z0-9\-]{1,62}[a-z0-9]))\.)+(?:[a-z]{2,})$)$"
   Const strIPv4 = "^\[(?:[0-9]{1,3}\.){3}[0-9]{1,3}\]$"
   Const strIPv6 = "^\[(IPv6)((?:[0-9A-Fa-f]{0,4}:){1,7}(?:(?:(>25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)|[0-9A-Fa-f]{1,4}))\]$"
   strRegEx = strFQDN & "|" & strIPv4 & "|" & strIPv6
   If (Lookup(strRegEx, oClient.HELO) = False) Then
      Result.Value = 2
      Result.Message = "5.7.1 CODE03 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means."
      Call AutoBan(oClient.IPAddress, "Bad HELO - " & oClient.HELO, 7, "d")
      Exit Sub
   End If
This would boot a helo with a [nl] in it. It goes in Sub OnHELO if you're using a version that supports it.
I tried to add this on my EventHandlers but one of the dynamic IP's from users are being blocked/banned in the server.
Here's the transmission logs:

Code: Select all

"TCPIP"	4312	"2021-09-26 10:08:59.047"	"TCP - IP.IP.IP.IP connected to 192.168.1.8:587."
"DEBUG"	4312	"2021-09-26 10:08:59.047"	"Executing event OnClientConnect"
"DEBUG"	4312	"2021-09-26 10:08:59.978"	"Event completed"
"DEBUG"	4312	"2021-09-26 10:08:59.978"	"TCP connection started for session 261"
"SMTPD"	4312	261	"2021-09-26 10:08:59.978"	"IP.IP.IP.IP"	"SENT: 220 mail.mydomain.com"
"SMTPD"	7220	261	"2021-09-26 10:09:00.025"	"IP.IP.IP.IP"	"RECEIVED: EHLO PCUserName"
"DEBUG"	7220	"2021-09-26 10:09:00.025"	"Executing event OnHELO"
"DEBUG"	7220	"2021-09-26 10:09:00.072"	"Event completed"
"SMTPD"	7220	261	"2021-09-26 10:09:00.072"	"IP.IP.IP.IP"	"SENT: 554 5.7.1 CODE03 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means."
"DEBUG"	7220	"2021-09-26 10:09:00.134"	"The read operation failed. Bytes transferred: 0 Remote IP: IP.IP.IP.IP, Session: 261, Code: 2, Message: End of file"
"DEBUG"	7220	"2021-09-26 10:09:00.134"	"Ending session 261"
I also try to add and configure this but some false positive IP's are being blocked.

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-09-26 12:22

ashtec014 wrote:
2021-09-26 09:40
palinka wrote:
2021-09-25 18:49
You may be able to find the message in your smtp logs. Look for the ehlo entry and see if the [nl] is in the domain.
Here's the SMTP logs at that time and can't find any [nl] during transmission.

Code: Select all

"SMTPD"	11104	1344	"2021-09-23 15:38:19.237"	"43.224.191.18"	"RECEIVED: EHLO mail.innovativepkg.com.ph"

.......


"DEBUG"	15508	"2021-09-23 15:39:40.018"	"Executing event OnAcceptMessage"
"ERROR"	15508	"2021-09-23 15:39:42.753"	"Severity: 2 (High), Code: HM5032, Source: DALConnection::Execute, Description: MySQL: Data too long for column 'domain' at row 1 (Additional info: INSERT INTO hm_catchspam (domain,hits) VALUES ('imohsohealthy.com[nl]mail.innovativepkg.com.ph',1) ON DUPLICATE KEY UPDATE hits=(hits+1),timestamp=NOW();)"
"ERROR"	15508	"2021-09-23 15:39:42.799"	"Script Error: Source: hMailServer COM library - Error: 800403E9 - Description: Execution of SQL statement failed. Error: MySQL: Data too long for column 'domain' at row 1 (Additional info: INSERT INTO hm_catchspam (domain,hits) VALUES ('imohsohealthy.com[nl]mail.innovativepkg.com.ph',1) ON DUPLICATE KEY UPDATE hits=(hits+1),timestamp=NOW();) - Line: 667 Column: 1 - Code: (null)"
"DEBUG"	15508	"2021-09-23 15:39:42.799"	"Event completed"
Whatever it was, it happened in OnAcceptMessage. Somehow, a variable got stuck and the second domain was added to it. For now, I'd consider that an anomaly. If it happens again, we'll have a look at it more closely. I wouldn't worry about it for now. I've never seen that happen before on my system. The HELO presented correctly.
palinka wrote:
2021-09-25 18:49

Also, you can force hmailserver to validate helo: viewtopic.php?p=209546#p209546

Code: Select all

   '
   '   Validate HELO/EHLO greeting
   '
   Const strFQDN = "^(?=^.{1,254}$)(^(?:(?!\.|-)([a-z0-9\-\*]{1,63}|([a-z0-9\-]{1,62}[a-z0-9]))\.)+(?:[a-z]{2,})$)$"
   Const strIPv4 = "^\[(?:[0-9]{1,3}\.){3}[0-9]{1,3}\]$"
   Const strIPv6 = "^\[(IPv6)((?:[0-9A-Fa-f]{0,4}:){1,7}(?:(?:(>25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)|[0-9A-Fa-f]{1,4}))\]$"
   strRegEx = strFQDN & "|" & strIPv4 & "|" & strIPv6
   If (Lookup(strRegEx, oClient.HELO) = False) Then
      Result.Value = 2
      Result.Message = "5.7.1 CODE03 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means."
      Call AutoBan(oClient.IPAddress, "Bad HELO - " & oClient.HELO, 7, "d")
      Exit Sub
   End If
This would boot a helo with a [nl] in it. It goes in Sub OnHELO if you're using a version that supports it.
I tried to add this on my EventHandlers but one of the dynamic IP's from users are being blocked/banned in the server.
Sorry, I forgot to mention you should exclude local and LAN like this (at the very top of OnHELO or OnAcceptMessage or both):

Code: Select all

   '   Exclude Backup-MX & local LAN from test
   '
   If (oClient.IPAddress = "127.0.0.1") Then Exit Sub
   If (Left(oClient.IPAddress, 10) = "192.168.1.") Then Exit Sub
I also try to add and configure this but some false positive IP's are being blocked.
There's a lot of stuff going on there. You're going to have to be more specific. :D

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-09-26 16:14

palinka wrote:
2021-09-26 12:22

Sorry, I forgot to mention you should exclude local and LAN like this (at the very top of OnHELO or OnAcceptMessage or both):

Code: Select all

   '   Exclude Backup-MX & local LAN from test
   '
   If (oClient.IPAddress = "127.0.0.1") Then Exit Sub
   If (Left(oClient.IPAddress, 10) = "192.168.1.") Then Exit Sub
Tried again and added these codes as suggested but one user IP address trying to send email located outside LAN for some reason has been banned by HMS autoban feature.

Here's my full code under sub OnHELO:

Code: Select all

Sub OnHELO(oClient)
   If (oClient.Port = 25) Then Wait(20)
	Dim PTR_Record, strRegEx
    If (Left(oClient.IPAddress, 8) = "192.168.1.") Then Exit Sub
      If (Left(oClient.IPAddress, 9) = "127.0.0.1") Then Exit Sub

      strRegEx = "(127\.0\.0\.1|mail\.mydomain\.com|\*\.\*|User|127\.0\.0\.1|ylmf-pc|info-api\.ru)" 'my ip xxx.xxx.xxx.xxx
      If (Lookup(strRegEx, oClient.HELO) = True) Then

          Result.Value = 2
          Result.Message = "5.7.1 Your access to this mail system has been rejected due to the sending\n" &_
                           "      MTA's poor reputation. If you believe that this failure is in error,\n" &_
                           "      please contact the intended recipient via alternate means."
          EventLog.Write(HELO & vbTab & oClient.IPAddress & vbTab & oClient.Port)
          Call AutoBan(oClient.IPAddress, "HELO " & oClient.HELO, 2, "d")
      End If
   	'	Grab PTR-Record
	PTR_Record = PTRLookup(oClient.IPAddress)
	'	Reject on No-PTR
	If (oClient.Port = 25) Then
		If PTR_Record = "No.PTR.Record" Then
			Result.Value = 2
			Result.Message = ". 03 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means."
			Exit Sub
		End If
	End If

	'   Filter bots using residential FQDN as HELO
	Dim a, i 
	a = Split(oClient.IPAddress, ".")
	For i = 0 to 3
	Next
	'   Exclude certain false positives
	strRegEx = "sendgrid|facebook.com"
	If Lookup(strRegEx, oClient.HELO) Then Exit Sub
	'   Search for residential looking HELO
	strRegEx = 	"(.*(((?:[0]{0,2})" & a(0) & "|(?:[0]{0,2})" & a(1) & "|(?:[0]{0,2})" & a(2) & "|(?:[0]{0,2})" & a(3) & ")(?:.+)){3}" &_
				"((?:[0]{0,2})" & a(0) & "|(?:[0]{0,2})" & a(1) & "|(?:[0]{0,2})" & a(2) & "|(?:[0]{0,2})" & a(3) & ").+)$"
	If (oClient.Port = 25) Then
		If Lookup(strRegEx, oClient.HELO) Then
			Result.Value = 2
			Result.Message = ". 05 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means."
			Exit Sub
		End If
	End If

	'	Filter dynamic-looking PTR-Record
	If PTR_Record <> "" Then
		'   Exclude certain false positives
		strRegEx = "sendgrid|facebook.com"
		If Lookup(strRegEx, PTR_Record) Then Exit Sub
		'   Search for residential looking HELO
		strRegEx = 	"(.*(((?:[0]{0,2})" & a(0) & "|(?:[0]{0,2})" & a(1) & "|(?:[0]{0,2})" & a(2) & "|(?:[0]{0,2})" & a(3) & ")(?:.+)){3}" &_
					"((?:[0]{0,2})" & a(0) & "|(?:[0]{0,2})" & a(1) & "|(?:[0]{0,2})" & a(2) & "|(?:[0]{0,2})" & a(3) & ").+)$"
		If (oClient.Port = 25) Then
			If Lookup(strRegEx, PTR_Record) Then
				Result.Value = 2
				Result.Message = ". 18 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means."
			End If
		End If
	End If
       '
   '   Validate HELO/EHLO greeting
   '
   Const strFQDN = "^(?=^.{1,254}$)(^(?:(?!\.|-)([a-z0-9\-\*]{1,63}|([a-z0-9\-]{1,62}[a-z0-9]))\.)+(?:[a-z]{2,})$)$"
   Const strIPv4 = "^\[(?:[0-9]{1,3}\.){3}[0-9]{1,3}\]$"
   Const strIPv6 = "^\[(IPv6)((?:[0-9A-Fa-f]{0,4}:){1,7}(?:(?:(>25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)|[0-9A-Fa-f]{1,4}))\]$"
   strRegEx = strFQDN & "|" & strIPv4 & "|" & strIPv6
    If (Lookup(strRegEx, oClient.HELO) = False) Then
      Result.Value = 2
      Result.Message = "5.7.1 CODE03 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means."
      Call AutoBan(oClient.IPAddress, "Bad HELO - " & oClient.HELO, 7, "d")
      Exit Sub
   End If
End Sub

User avatar
SorenR
Senior user
Senior user
Posts: 4902
Joined: 2006-08-21 15:38
Location: Denmark

Re: Issue related to hMailServer_CatchSpam

Post by SorenR » 2021-09-26 23:54

Code: Select all

If (Left(oClient.IPAddress, 8) = "192.168.1.") Then Exit Sub
Left(x, 8 ) means 8 CHARACTERS ! Comparing an 8 character string to a 10 character string will NEVER match ! It should be

Code: Select all

If (Left(oClient.IPAddress, 8 ) = "192.168.") Then Exit Sub


1: You rule out 127.0.0.1 with "Exit Sub", WHY include it in your RegEx ??

Btw.

Code: Select all

If (Left(oClient.IPAddress, 9) = "127.0.0.1") Then Exit Sub
is the same as

Code: Select all

If oClient.IPAddress = "127.0.0.1" Then Exit Sub
2: WTF is this:

Code: Select all

\*\.\*

Code: Select all

ORIGINAL: strRegEx = "(127\.0\.0\.1|mail\.mydomain\.com|\*\.\*|User|127\.0\.0\.1|ylmf-pc|info-api\.ru)" 'my ip xxx.xxx.xxx.xxx

EDITED:   strRegEx = "(mail\.mydomain\.com|User|ylmf-pc|info-api\.ru)" 'my ip xxx.xxx.xxx.xxx
SørenR.

Engineer (noun)
- I'm Not Arguing, I'm Just Explaining Why I'm Right

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-09-27 02:13

SorenR wrote:
2021-09-26 23:54
2: WTF is this:

Code: Select all

\*\.\*
Anything you want. It's the Genie RegEx™ !!! :D

User avatar
SorenR
Senior user
Senior user
Posts: 4902
Joined: 2006-08-21 15:38
Location: Denmark

Re: Issue related to hMailServer_CatchSpam

Post by SorenR » 2021-09-27 02:27

palinka wrote:
2021-09-27 02:13
SorenR wrote:
2021-09-26 23:54
2: WTF is this:

Code: Select all

\*\.\*
Anything you want. It's the Genie RegEx™ !!! :D
It's actually the string *.* since both the stars and the dot are escaped. Why would you need that?
SørenR.

Engineer (noun)
- I'm Not Arguing, I'm Just Explaining Why I'm Right

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-09-27 02:34

SorenR wrote:
2021-09-27 02:27
palinka wrote:
2021-09-27 02:13
SorenR wrote:
2021-09-26 23:54
2: WTF is this:

Code: Select all

\*\.\*
Anything you want. It's the Genie RegEx™ !!! :D
It's actually the string *.* since both the stars and the dot are escaped. Why would you need that?
That's what I said - the Genie RegEx™ :D

Its in a matching test on a HELO string. Maybe some fool thought that could cause a failure or something to gain entry, and it showed up on the OP's logs???

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-09-27 16:10

Hello,

I adjusted the script based on recommendation above. Will continue to monitor and provide updates if issue (CatchSpam) occur again. Thank you for the help.

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-09-27 17:24

Don't forget you can whitelist domains in the catchspam php admin thing.

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-09-28 08:27

palinka wrote:
2021-09-27 17:24
Don't forget you can whitelist domains in the catchspam php admin thing.
Yes, I will. Thanks a lot.

I have one more question: This script and admin thing are all the same or consolidated from this? Because if not, I'll try to implement it as well on my settings but I'm still reading the thread and see the outcome.

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-09-28 11:48

ashtec014 wrote:
2021-09-28 08:27
palinka wrote:
2021-09-27 17:24
Don't forget you can whitelist domains in the catchspam php admin thing.
Yes, I will. Thanks a lot.

I have one more question: This script and admin thing are all the same or consolidated from this? Because if not, I'll try to implement it as well on my settings but I'm still reading the thread and see the outcome.
Catchspam has it own php admin. I forgot you already used it to mark one domain safe (according to your screenshot above).

I abandoned the firewall ban script. Its grew to be too complicated and was not effective enough to justify the constant maintenance. One thing its REALLY good at is statistics because it records so much info. I found that about 60% of banned IPs never ever returned. I figure they're part of very large bot nets. What Soren calls "snowshoe spam". Mostly they are just password guessers. Anyway, one day my server crashed and I had to rebuild it so I decided not to re-implement the firewall ban. Since then I've noticed that I get more spam from gmail - which can't be blocked - than any other source, so I'm not really missing anything without the firewall ban. Most of the bans were geoip related anyway. It was a good experiment and I gained a lot of knowledge from it, but its really not necessary.

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-09-28 18:53

palinka wrote:
2021-09-28 11:48
Catchspam has it own php admin. I forgot you already used it to mark one domain safe (according to your screenshot above).

I abandoned the firewall ban script. Its grew to be too complicated and was not effective enough to justify the constant maintenance. One thing its REALLY good at is statistics because it records so much info. I found that about 60% of banned IPs never ever returned. I figure they're part of very large bot nets. What Soren calls "snowshoe spam". Mostly they are just password guessers. Anyway, one day my server crashed and I had to rebuild it so I decided not to re-implement the firewall ban. Since then I've noticed that I get more spam from gmail - which can't be blocked - than any other source, so I'm not really missing anything without the firewall ban. Most of the bans were geoip related anyway. It was a good experiment and I gained a lot of knowledge from it, but its really not necessary.
I see, so I won't implement it then.
Since then I've noticed that I get more spam from gmail - which can't be blocked - than any other source, so I'm not really missing anything without the firewall ban.
- True, I've seen a lot of spammer as well using gmail and thankfully, I have your consolidated scripts taken from this forum and it helps a lot, as in a lot and I can't thank enough for sharing the scripts and all your help, so much appreciated.

BTW, I'm still struggling with this code:

Code: Select all

 '   Validate HELO/EHLO greeting
   '
   Const strFQDN = "^(?=^.{1,254}$)(^(?:(?!\.|-)([a-z0-9\-\*]{1,63}|([a-z0-9\-]{1,62}[a-z0-9]))\.)+(?:[a-z]{2,})$)$"
   Const strIPv4 = "^\[(?:[0-9]{1,3}\.){3}[0-9]{1,3}\]$"
   Const strIPv6 = "^\[(IPv6)((?:[0-9A-Fa-f]{0,4}:){1,7}(?:(?:(>25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)|[0-9A-Fa-f]{1,4}))\]$"
   strRegEx = strFQDN & "|" & strIPv4 & "|" & strIPv6
    If (Lookup(strRegEx, oClient.HELO) = False) Then
      Result.Value = 2
      Result.Message = "5.7.1 CODE03 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means."
      Call AutoBan(oClient.IPAddress, "Bad HELO - " & oClient.HELO, 7, "d")
      Exit Sub
   End If
When this code is enabled under Sub OnHELO(oClient) along with this one:

Code: Select all

 If (Left(oClient.IPAddress, 8 ) = "200.0.") Then Exit Sub 'This is local LAN
    If oClient.IPAddress = "127.0.0.1" Then Exit Sub ' This is localhost/Server
Dim PTR_Record, strRegEx, sIPAddress
       
      strRegEx = "(mail\.mydomain\.com|User|ylmf-pc|info-api\.ru)" 'my ip xxx.xxx.xxx.xxx
      If (Lookup(strRegEx, oClient.HELO) = True) Then
          Result.Value = 2
          Result.Message = "5.7.1 Your access to this mail system has been rejected due to the sending\n" &_
                           "      MTA's poor reputation. If you believe that this failure is in error,\n" &_
                           "      please contact the intended recipient via alternate means."
          EventLog.Write(HELO & vbTab & oClient.IPAddress & vbTab & oClient.Port)
          EventLog.Write("No PTR Record, IP Blocked: " & oClient.IpAddress & "")
          Call AutoBan(oClient.IPAddress, "HELO " & oClient.HELO, 2, "d")
      End If
They are both conflict. I don't know why. I tried to test, login on my webmail (deployed on the same server) it got banned/blocked. However, when I tried to send email on my webmail using my mobile internet connection the email was sent successfully. I can't figure out why the LAN was blocked by the server as it is already excluded local and LAN like at the very top of OnHELO and OnAcceptMessage.

Here's sample logs for email testing (this was sent using outlook on my desktop), 200.0.0.103 is the LAN IP like gateway, 200.0.0.8 is the local IP of my server:

Code: Select all

"TCPIP"	11672	"2021-09-28 19:18:19.540"	"TCP - 200.0.0.103 connected to 200.0.0.8:587."
"DEBUG"	11672	"2021-09-28 19:18:19.556"	"Executing event OnClientConnect"
"DEBUG"	11672	"2021-09-28 19:18:20.016"	"Event completed"
"DEBUG"	11672	"2021-09-28 19:18:20.016"	"TCP connection started for session 3134"
"SMTPD"	11672	3134	"2021-09-28 19:18:20.016"	"200.0.0.103"	"SENT: 220 mail.mydomain.com ESMTP"
"SMTPD"	9228	3134	"2021-09-28 19:18:20.016"	"200.0.0.103"	"RECEIVED: EHLO DESKTOPV645LQC"
"DEBUG"	9228	"2021-09-28 19:18:20.016"	"Executing event OnHELO"
"DEBUG"	13600	"2021-09-28 19:18:20.078"	"The read operation failed. Bytes transferred: 0 Remote IP: 200.0.0.103, Session: 3131, Code: 1236, Message: The network connection was aborted by the local system"
"DEBUG"	13600	"2021-09-28 19:18:20.078"	"Ending session 3131"
"DEBUG"	9228	"2021-09-28 19:18:20.078"	"Event completed"
"SMTPD"	9228	3134	"2021-09-28 19:18:20.078"	"200.0.0.103"	"SENT: 554 5.7.1 CODE03 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means."
"DEBUG"	13600	"2021-09-28 19:18:20.078"	"The write operation failed. Bytes transferred: 0 Remote IP: 200.0.0.103, Session: 3134, Code: 10054, Message: An existing connection was forcibly closed by the remote host"
"DEBUG"	9228	"2021-09-28 19:18:20.078"	"Ending session 3134"

User avatar
johang
Senior user
Senior user
Posts: 584
Joined: 2008-09-01 09:20

Re: Issue related to hMailServer_CatchSpam

Post by johang » 2021-09-28 19:32

palinka wrote:
2021-09-28 11:48

I abandoned the firewall ban script. Its grew to be too complicated and was not effective enough to justify the constant maintenance. One thing its REALLY good at is statistics because it records so much info. I found that about 60% of banned IPs never ever returned. I figure they're part of very large bot nets. What Soren calls "snowshoe spam". Mostly they are just password guessers. Anyway, one day my server crashed and I had to rebuild it so I decided not to re-implement the firewall ban. Since then I've noticed that I get more spam from gmail - which can't be blocked - than any other source, so I'm not really missing anything without the firewall ban. Most of the bans were geoip related anyway. It was a good experiment and I gained a lot of knowledge from it, but its really not necessary.



eeeeehhh ...

password guessers do not do their game so they can spam you...... they do it to get an legitimate account and log in and use your plattform to spam others ,
and that is where autoban and your firewall ban script does its work.. stopping them from hammering .. and possibly getting access..

You getting spam ( from gmail or others ) is another game... and your firewall ban would not ever fix that ( that is where you get spamassasin and the likes in play .... )

i think you have missed/forgotten the purpose of your own tool :shock:


and if 60% never came back... you should have "up:d" the ban time so the other 40% would have gotten the finger more often/longer .. :twisted: just my 2 cents :wink:
lets cheat darwin out of his legacy, find a cure for cancer...

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-09-28 20:36

johang wrote:
2021-09-28 19:32
You getting spam ( from gmail or others ) is another game... and your firewall ban would not ever fix that ( that is where you get spamassasin and the likes in play .... )

i think you have missed/forgotten the purpose of your own tool :shock:
IPs don't ban themselves. Tests must be performed with pass/fail in order to determine whether an IP should be banned or not (among other things).

Before (with firewall ban):

Code: Select all

If test fail Then
	1) disconnect spammy client
	2) call autoban on IP
	3) report to custom log
	4) add IP to firewall ban
End If
After (no firewall ban):

Code: Select all

If test fail Then
	1) disconnect spammy client
	2) call autoban on IP
	3) report to custom log
End If
The tests (scripts) before and after are the same. All I did in my eventhandlers.vbs was comment out the "Call FWBan(oClient.IPAddress)" in each of the tests where it was present.

I never firewall banned gmail or other well known domains, plus if there was a false positive I could easily reverse it or whitelist it. I actually have a function that looks up various whitelists. If IP is whitelisted for any on the list, then some tests are skipped. :D

Tests I used to determine whether to firewall ban:
* geoip
* invalid helo
* tor exit node
* catchspam (see this thread above :wink: )
* no PTR
* "dynamic looking" helo (signifies residential IP)
* "dynamic looking" PTR (signifies residential IP)
* reported as abusive on AbuseIPDB
* spamhaus zen
* helo appears in spamhaus dbl
* PTR appears in spamhaus dbl
* from address domain appears in spamhaus dbl
* envelope from address domain appears in spamhaus dbl
* UCE Protect (but disabled a long time ago due to false positives)
* specific helos (from database I created)
* specific PTRs (from database I created)
* specific envelope from addresses (from database I created)

I figured its safe to perma ban any IP that fails any of those tests (after having passed through the whitelist function in certain cases). I still do believe that those tests are bona fide bannable offenses. However, I no longer believe the firewall ban meets cost/benefit when dealing with them because of the maintenance involved. Autoban is fine. If they come back - and only 40% do - then they'll be autobanned before getting the opportunity to password guess or send spam. :D :D :D

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-09-28 20:41

johang wrote:
2021-09-28 19:32
palinka wrote:
2021-09-28 11:48
and if 60% never came back... you should have "up:d" the ban time so the other 40% would have gotten the finger more often/longer .. :twisted: just my 2 cents :wink:
I never released/expired any ban except false positives. After almost 2 years of record keeping, that 60% which never came back really never did come back. I know that because I also recorded dropped connections from the firewall log. 60% of the IPs I banned were never seen again in the almost 2 years that I was running it.

A small percentage came back A LOT. A very few tried almost daily for the entire time. You can still see the data on my firewall ban demo site: https://firewallban.dynu.net/

Code: Select all

Top 5 Repeat Spammers:
Parsed from the firewall log dropped connections: IPs that knocked on the door but couldn't get in.

58,261 knocks by 45.82.153.131 from Russia
50,074 knocks by 45.82.153.132 from Russia
46,148 knocks by 212.70.149.4 from Bulgaria
42,687 knocks by 136.147.183.133 from United States
39,524 knocks by 136.147.183.131 from United States

21,454 IPs [of 53,967] attempted to connect but were dropped at the firewall a total of 2,624,594 times since July 17th, 2019

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-09-28 21:00

ashtec014 wrote:
2021-09-28 18:53
When this code is enabled under Sub OnHELO(oClient) along with this one:

Code: Select all

 If (Left(oClient.IPAddress, 8 ) = "200.0.") Then Exit Sub 'This is local LAN
    If oClient.IPAddress = "127.0.0.1" Then Exit Sub ' This is localhost/Server
Dim PTR_Record, strRegEx, sIPAddress
       
      strRegEx = "(mail\.mydomain\.com|User|ylmf-pc|info-api\.ru)" 'my ip xxx.xxx.xxx.xxx
      If (Lookup(strRegEx, oClient.HELO) = True) Then
          Result.Value = 2
          Result.Message = "5.7.1 Your access to this mail system has been rejected due to the sending\n" &_
                           "      MTA's poor reputation. If you believe that this failure is in error,\n" &_
                           "      please contact the intended recipient via alternate means."
          EventLog.Write(HELO & vbTab & oClient.IPAddress & vbTab & oClient.Port)
          EventLog.Write("No PTR Record, IP Blocked: " & oClient.IpAddress & "")
          Call AutoBan(oClient.IPAddress, "HELO " & oClient.HELO, 2, "d")
      End If
strRegEx = "(mail\.mydomain\.com|User|ylmf-pc|info-api\.ru)" 'my ip xxx.xxx.xxx.xxx

I presume mail.mydomain.com doesn't exist, correct?

Try wrapping both of these tests in this:

If oClient.Port = 25 Then
'
' code
'
End If

Here is exactly what I have copied verbatim from my eventhandlers.vbs:

Code: Select all

Sub OnHELO(oClient)

	Dim [lots of vars]

	REM	- Exclude local LAN & Backup from test after recording connection
	If oClient.IPAddress = WANIP Then
		Call AccRejDB(0, oClient.Port, "OnHELO", "Accepted", "Local", oClient.IPAddress, oClient.HELO)
		Exit Sub
	End If
	If (Left(oClient.IPAddress, 10) = "192.168.0.") Then 
		Call AccRejDB(0, oClient.Port, "OnHELO", "Accepted", "Local", oClient.IPAddress, oClient.HELO)
		Exit Sub
	End If
	If oClient.IPAddress = "127.0.0.1" Then
		Call AccRejDB(0, oClient.Port, "OnHELO", "Accepted", "Local", oClient.IPAddress, oClient.HELO)
		Exit Sub
	End If
	If (Left(oClient.IPAddress, 12) = "184.105.182.") Then
		Call AccRejDB(0, oClient.Port, "OnHELO", "Accepted", "BackupMX", oClient.IPAddress, oClient.HELO)
		Exit Sub
	End If
	If (Left(oClient.IPAddress, 11) = "130.158.75.") Then
		Call AccRejDB(0, oClient.Port, "OnHELO", "Accepted", "VPN-ddns", oClient.IPAddress, oClient.HELO)
		Exit Sub
	End If

	[lots of other code]

	REM	- Validate HELO/EHLO greeting
	Const strFQDN = "^(?=^.{1,254}$)(^(?:(?!\.|-)([a-z0-9\-\*]{1,63}|([a-z0-9\-]{1,62}[a-z0-9]))\.)+(?:[a-z]{2,})$)$"
	Const strIPv4 = "^\[(?:[0-9]{1,3}\.){3}[0-9]{1,3}\]$"
	Const strIPv6 = "^\[(IPv6)((?:[0-9A-Fa-f]{0,4}:){1,7}(?:(?:(>25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)|[0-9A-Fa-f]{1,4}))\]$"
	strRegEx = strFQDN & "|" & strIPv4 & "|" & strIPv6
	If (Lookup(strRegEx, oClient.HELO) = False) Then
		Result.Value = 2
		Result.Message = ". 04 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means."
		Call Disconnect(oClient.IPAddress)
		'Call FWBan(oClient.IPAddress, "HELO-Inv", oClient.HELO, PTR_Record)
		Call AutoBan(oClient.IPAddress, "Invalid HELO - " & oClient.HELO, 1, "h")
		Call AccRejDB(0, oClient.Port, "OnHELO", "REJECTED", "HELO-Inv", oClient.IPAddress, oClient.HELO)
		Call ReportToAbuseIPDB(oClient.IPAddress, "11", "Mail Rejected for Invalid HELO on port " & oClient.Port & ", EHLO: " & oClient.HELO)
		Exit Sub
	End If

	[lots of other code]

End Sub
I don't get false positives from that function. I also don't have it limited to port 25. But I do reject local/LAN/backup mx from those tests.

User avatar
SorenR
Senior user
Senior user
Posts: 4902
Joined: 2006-08-21 15:38
Location: Denmark

Re: Issue related to hMailServer_CatchSpam

Post by SorenR » 2021-09-29 00:06

ashtec014 wrote:
2021-09-28 18:53
palinka wrote:
2021-09-28 11:48
Catchspam has it own php admin. I forgot you already used it to mark one domain safe (according to your screenshot above).

I abandoned the firewall ban script. Its grew to be too complicated and was not effective enough to justify the constant maintenance. One thing its REALLY good at is statistics because it records so much info. I found that about 60% of banned IPs never ever returned. I figure they're part of very large bot nets. What Soren calls "snowshoe spam". Mostly they are just password guessers. Anyway, one day my server crashed and I had to rebuild it so I decided not to re-implement the firewall ban. Since then I've noticed that I get more spam from gmail - which can't be blocked - than any other source, so I'm not really missing anything without the firewall ban. Most of the bans were geoip related anyway. It was a good experiment and I gained a lot of knowledge from it, but its really not necessary.
I see, so I won't implement it then.
Since then I've noticed that I get more spam from gmail - which can't be blocked - than any other source, so I'm not really missing anything without the firewall ban.
- True, I've seen a lot of spammer as well using gmail and thankfully, I have your consolidated scripts taken from this forum and it helps a lot, as in a lot and I can't thank enough for sharing the scripts and all your help, so much appreciated.

BTW, I'm still struggling with this code:

Code: Select all

 '   Validate HELO/EHLO greeting
   '
   Const strFQDN = "^(?=^.{1,254}$)(^(?:(?!\.|-)([a-z0-9\-\*]{1,63}|([a-z0-9\-]{1,62}[a-z0-9]))\.)+(?:[a-z]{2,})$)$"
   Const strIPv4 = "^\[(?:[0-9]{1,3}\.){3}[0-9]{1,3}\]$"
   Const strIPv6 = "^\[(IPv6)((?:[0-9A-Fa-f]{0,4}:){1,7}(?:(?:(>25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)|[0-9A-Fa-f]{1,4}))\]$"
   strRegEx = strFQDN & "|" & strIPv4 & "|" & strIPv6
    If (Lookup(strRegEx, oClient.HELO) = False) Then
      Result.Value = 2
      Result.Message = "5.7.1 CODE03 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means."
      Call AutoBan(oClient.IPAddress, "Bad HELO - " & oClient.HELO, 7, "d")
      Exit Sub
   End If
When this code is enabled under Sub OnHELO(oClient) along with this one:

Code: Select all

 If (Left(oClient.IPAddress, 8 ) = "200.0.") Then Exit Sub 'This is local LAN
    If oClient.IPAddress = "127.0.0.1" Then Exit Sub ' This is localhost/Server
Dim PTR_Record, strRegEx, sIPAddress
       
      strRegEx = "(mail\.mydomain\.com|User|ylmf-pc|info-api\.ru)" 'my ip xxx.xxx.xxx.xxx
      If (Lookup(strRegEx, oClient.HELO) = True) Then
          Result.Value = 2
          Result.Message = "5.7.1 Your access to this mail system has been rejected due to the sending\n" &_
                           "      MTA's poor reputation. If you believe that this failure is in error,\n" &_
                           "      please contact the intended recipient via alternate means."
          EventLog.Write(HELO & vbTab & oClient.IPAddress & vbTab & oClient.Port)
          EventLog.Write("No PTR Record, IP Blocked: " & oClient.IpAddress & "")
          Call AutoBan(oClient.IPAddress, "HELO " & oClient.HELO, 2, "d")
      End If
They are both conflict. I don't know why. I tried to test, login on my webmail (deployed on the same server) it got banned/blocked. However, when I tried to send email on my webmail using my mobile internet connection the email was sent successfully. I can't figure out why the LAN was blocked by the server as it is already excluded local and LAN like at the very top of OnHELO and OnAcceptMessage.

Here's sample logs for email testing (this was sent using outlook on my desktop), 200.0.0.103 is the LAN IP like gateway, 200.0.0.8 is the local IP of my server:

Code: Select all

"TCPIP"	11672	"2021-09-28 19:18:19.540"	"TCP - 200.0.0.103 connected to 200.0.0.8:587."
"DEBUG"	11672	"2021-09-28 19:18:19.556"	"Executing event OnClientConnect"
"DEBUG"	11672	"2021-09-28 19:18:20.016"	"Event completed"
"DEBUG"	11672	"2021-09-28 19:18:20.016"	"TCP connection started for session 3134"
"SMTPD"	11672	3134	"2021-09-28 19:18:20.016"	"200.0.0.103"	"SENT: 220 mail.mydomain.com ESMTP"
"SMTPD"	9228	3134	"2021-09-28 19:18:20.016"	"200.0.0.103"	"RECEIVED: EHLO DESKTOPV645LQC"
"DEBUG"	9228	"2021-09-28 19:18:20.016"	"Executing event OnHELO"
"DEBUG"	13600	"2021-09-28 19:18:20.078"	"The read operation failed. Bytes transferred: 0 Remote IP: 200.0.0.103, Session: 3131, Code: 1236, Message: The network connection was aborted by the local system"
"DEBUG"	13600	"2021-09-28 19:18:20.078"	"Ending session 3131"
"DEBUG"	9228	"2021-09-28 19:18:20.078"	"Event completed"
"SMTPD"	9228	3134	"2021-09-28 19:18:20.078"	"200.0.0.103"	"SENT: 554 5.7.1 CODE03 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means."
"DEBUG"	13600	"2021-09-28 19:18:20.078"	"The write operation failed. Bytes transferred: 0 Remote IP: 200.0.0.103, Session: 3134, Code: 10054, Message: An existing connection was forcibly closed by the remote host"
"DEBUG"	9228	"2021-09-28 19:18:20.078"	"Ending session 3134"
First thing... "200.0.0.103 is the LAN IP like gateway" and it identifies as "DESKTOPV645LQC" :?: :!: :?: :!:

Yes, the first codeblock smacks the hell out of "DESKTOPV645LQC"! It is NOT a FQDN as specified by the RFC's.
That would also explain this:

Code: Select all

"DEBUG"	13600	"2021-09-28 19:18:20.078"	"The read operation failed. Bytes transferred: 0 Remote IP: 200.0.0.103, Session: 3131, Code: 1236, Message: The network connection was aborted by the local system"
and this:

Code: Select all

"SMTPD"	9228	3134	"2021-09-28 19:18:20.078"	"200.0.0.103"	"SENT: 554 5.7.1 CODE03 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means."
"DEBUG"	13600	"2021-09-28 19:18:20.078"	"The write operation failed. Bytes transferred: 0 Remote IP: 200.0.0.103, Session: 3134, Code: 10054, Message: An existing connection was forcibly closed by the remote host"
since your server virtually hung up the conversation.
SørenR.

Engineer (noun)
- I'm Not Arguing, I'm Just Explaining Why I'm Right

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-09-29 09:16

palinka wrote:
2021-09-28 21:00
I presume mail.mydomain.com doesn't exist, correct?

Try wrapping both of these tests in this:

If oClient.Port = 25 Then
'
' code
'
End If

I don't get false positives from that function. I also don't have it limited to port 25. But I do reject local/LAN/backup mx from those tests.
This one works for me. Thank you so much!
SorenR wrote:
2021-09-29 00:06

First thing... "200.0.0.103 is the LAN IP like gateway" and it identifies as "DESKTOPV645LQC" :?: :!: :?: :!:

Yes, the first codeblock smacks the hell out of "DESKTOPV645LQC"! It is NOT a FQDN as specified by the RFC's.
DESKTOPV645LQC is the name of my PC and 200.0.0.103 is an IP of my Firewall/Gateway. But it is working now after wrapping it up with the code above as suggested by Palinka. Thank you Soren, I appreciate your inputs.

User avatar
SorenR
Senior user
Senior user
Posts: 4902
Joined: 2006-08-21 15:38
Location: Denmark

Re: Issue related to hMailServer_CatchSpam

Post by SorenR » 2021-09-29 11:37

ashtec014 wrote:
2021-09-29 09:16
SorenR wrote:
2021-09-29 00:06

First thing... "200.0.0.103 is the LAN IP like gateway" and it identifies as "DESKTOPV645LQC" :?: :!: :?: :!:

Yes, the first codeblock smacks the hell out of "DESKTOPV645LQC"! It is NOT a FQDN as specified by the RFC's.
DESKTOPV645LQC is the name of my PC and 200.0.0.103 is an IP of my Firewall/Gateway. But it is working now after wrapping it up with the code above as suggested by Palinka. Thank you Soren, I appreciate your inputs.
Well...

Code: Select all

"SMTPD"	11672	3134	"2021-09-28 19:18:20.016"	"200.0.0.103"	"SENT: 220 mail.mydomain.com ESMTP"
"SMTPD"	9228	3134	"2021-09-28 19:18:20.016"	"200.0.0.103"	"RECEIVED: EHLO DESKTOPV645LQC"
Which is it? Your PC or your Firewall/Gateway?
SørenR.

Engineer (noun)
- I'm Not Arguing, I'm Just Explaining Why I'm Right

palinka
Senior user
Senior user
Posts: 2984
Joined: 2017-09-12 17:57

Re: Issue related to hMailServer_CatchSpam

Post by palinka » 2021-09-29 12:14

SorenR wrote:
2021-09-29 11:37
Well...

Code: Select all

"SMTPD"	11672	3134	"2021-09-28 19:18:20.016"	"200.0.0.103"	"SENT: 220 mail.mydomain.com ESMTP"
"SMTPD"	9228	3134	"2021-09-28 19:18:20.016"	"200.0.0.103"	"RECEIVED: EHLO DESKTOPV645LQC"
Which is it? Your PC or your Firewall/Gateway?
ashtec014 wrote:
2021-09-29 09:16
DESKTOPV645LQC is the name of my PC and 200.0.0.103 is an IP of my Firewall/Gateway.
Are you using a software gateway? 200.0.0.103 is not a private address.

What do you have in settings > protocols > smtp > advanced > bind to local IP address? Is it blank or is it 200.0.0.103?

Please describe your network setup.

ashtec014
Normal user
Normal user
Posts: 197
Joined: 2019-09-05 11:56

Re: Issue related to hMailServer_CatchSpam

Post by ashtec014 » 2021-09-29 17:30

Network setup is like this:

All local IP's (private IP's) setup in an internal Network Switch (CISCO) starts 200.0.0.1/255 then the router/firewall has an IP of 200.0.0.103.

Example:
PC local client (200.0.0.34) connect to mail server (200.0.0.20) >> This IP ((200.0.0.34) will be forwarded to router/firewall gateway (200.0.0.103) then the router/firewall will forward the traffic to mail server (200.0.0.20) and vice versa.

If the traffic is within the local network IP being logged is "200.0.0.103". However, when the traffic is outside the network the actual public IP of the client will be logged as it is. According to our network admin, they design the network infra like this as an added security. Therefore, the IP being logged under webmail is also the same IP as the firewall. So, basically all traffic are being taken care of by the gateway/firewall/router whether internal or external.

User avatar
SorenR
Senior user
Senior user
Posts: 4902
Joined: 2006-08-21 15:38
Location: Denmark

Re: Issue related to hMailServer_CatchSpam

Post by SorenR » 2021-09-29 18:40

ashtec014 wrote:
2021-09-29 17:30
Network setup is like this:

All local IP's (private IP's) setup in an internal Network Switch (CISCO) starts 200.0.0.1/255 then the router/firewall has an IP of 200.0.0.103.

Example:
PC local client (200.0.0.34) connect to mail server (200.0.0.20) >> This IP ((200.0.0.34) will be forwarded to router/firewall gateway (200.0.0.103) then the router/firewall will forward the traffic to mail server (200.0.0.20) and vice versa.

If the traffic is within the local network IP being logged is "200.0.0.103". However, when the traffic is outside the network the actual public IP of the client will be logged as it is. According to our network admin, they design the network infra like this as an added security. Therefore, the IP being logged under webmail is also the same IP as the firewall. So, basically all traffic are being taken care of by the gateway/firewall/router whether internal or external.
It's not "safer" - it's just a PITA to work with. Fire the network admin and spend the money on a PROPER firewall and a solid Anti-Virus!
The main security risk is not someone breaking your network from the outside in... It's someone sending you an email with a few bytes of code. No matter how secure you make your routing and filters - if you invite them in, you are doomed!
Just look at Maersk. One (1, uno, Eine) idiot employee responded to an email with some bad code and Maersk suddenly lost $300 million.
Not to mention they had to REBUILD 49,000 laptops!

https://www.i-cio.com/management/insigh ... ber-attack
SørenR.

Engineer (noun)
- I'm Not Arguing, I'm Just Explaining Why I'm Right

Post Reply