Some email bypass spamasssassin

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
johnyu2012
Normal user
Normal user
Posts: 135
Joined: 2012-09-11 06:33

Some email bypass spamasssassin

Post by johnyu2012 » 2022-08-06 10:19

I don't understand why some email can bypass spamassassin. Usually after the return-path, it should go to spamasssassin checker but it doesn't.

Return-Path: prs.squ@ibillcentre.com
Received: from kphaykwf.ibillcentre.com (kphaykwf.ibillcentre.com [85.217.145.172]) by mail.<mydomain>.com with ESMTP ; Sat, 6 Aug 2022 15:57:56 +0800
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ibillcentre.com; s=mail; h=Message-ID:References:In-Reply-To:From:Date: Content-Type:MIME-Version:Subject:To:Sender:Reply-To:Cc: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=CIyjhCf8b5uOKOC/O8SykJzpNrTKKnCj6mpx1PCBa6o=; b=jWCqgfJHpXjDBTDm9yryqqP2F 8BCxoi+0kBmfe8T9wNaHdlI6lgbVLbR64IyrocVuE3J1YouzCwCOhtCmqMhyZ8AjZYxUoR+lfHohf b7M0eJ3uK5E7ClPs4H61lcE9dBoa9/ul/G87vPpKPlQ0gCV/uRRwHz7vf12Q+QRbPLXHM=;
Received: from admin by kphaykwf.ibillcentre.com with local (Exim 4.92.3) (envelope-from <prs.squ@ibillcentre.com>) id 1oKECo-0001Cd-S2; Sat, 06 Aug 2022 07:26:42 +0000
To: undisclosed-recipients:;
Subject: (PRIORITY) DHL Shipment Notification : Custom Release Form (E) Error AWB: 82****51
X-PHP-Originating-Script: 0:rcube.php
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=_c0c47f6d75c7977cb7ad98a325d66bf9"
Date: Sat, 06 Aug 2022 10:26:42 +0300
From: DHL Custom Clearance <prs.squ@ibillcentre.com>
In-Reply-To: <CAFfRH8CYe_128PCxLzUafzbt4auLOD0jkpS=W-B0p9g_V51R_A@mail.gmail.com>
References: <CAFfRH8CYe_128PCxLzUafzbt4auLOD0jkpS=W-B0p9g_V51R_A@mail.gmail.com>
Message-ID: <24e7c3e9c78a4c76e72f7dc2d3399504@ibillcentre.com>
X-Sender: prs.squ@ibillcentre.com
User-Agent: Roundcube Webmail/1.0.12

User avatar
jim.bus
Senior user
Senior user
Posts: 1571
Joined: 2011-05-28 11:49
Location: US

Re: Some email bypass spamasssassin

Post by jim.bus » 2022-08-06 13:06

How do you know it isn't going to Spamassassin?

Before I would say it didn't go to Spamassassin, I would want to see if it did not attempt to connect to Spamassassin in the hMailServer Log Entries. I would also want to see in your configuration if you Whitelisted anything from SPAM Checking (Anti-spam in hMailAdmin). I wouldn't rely on the Internet Message Headers which incidentally can be spoofed if you didn't know.
If you think you understand quantum mechanics, you don't understand quantum mechanics.

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-06 13:18

I have this occasionally, eg: The WinSock error code is 2. Maybe 1, 2 times a days, but there are also days it won't happen at all

If you have any WinSock error code is 2 errors in your hmailserver_error.log you might compare date/time with a message that supposedly passed thru without being checked by SA.
Up till now there is no failsafe solution for the 'WinSock error code is 2' issue, some people claim to have less when they set the amount of command threads higher, but for me that had no effect at all
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
jim.bus
Senior user
Senior user
Posts: 1571
Joined: 2011-05-28 11:49
Location: US

Re: Some email bypass spamasssassin

Post by jim.bus » 2022-08-06 13:28

RvdH wrote:
2022-08-06 13:18
I have this occasionally, eg: The WinSock error code is 2. Maybe 1, 2 times a days, but there are also days it won't happen at all

If you have any WinSock error code is 2 errors in your hmailserver_error.log you might compare date/time with a message that supposedly passed thru without being checked by SA.
Up till now there is no failsafe solution for the 'WinSock error code is 2' issue, some people claim to have less when they set the amount of command threads higher, but for me that had no effect at all
That was me and I have to agree with you now about the 'Asynchronous task threads'. I was probably just lucky on having the 4 month long period without any of the Winsock Errors. And then again, my message volume is very low compared to many others' message volume.
If you think you understand quantum mechanics, you don't understand quantum mechanics.

johnyu2012
Normal user
Normal user
Posts: 135
Joined: 2012-09-11 06:33

Re: Some email bypass spamasssassin

Post by johnyu2012 » 2022-08-06 14:53

jim.bus wrote:
2022-08-06 13:06
How do you know it isn't going to Spamassassin?

Before I would say it didn't go to Spamassassin, I would want to see if it did not attempt to connect to Spamassassin in the hMailServer Log Entries. I would also want to see in your configuration if you Whitelisted anything from SPAM Checking (Anti-spam in hMailAdmin). I wouldn't rely on the Internet Message Headers which incidentally can be spoofed if you didn't know.
True. It does go through spam test and the score is 0.

"SMTPD" 7008 2547 "2022-08-06 15:57:53.667" "85.217.145.172" "SENT: 220 mail.<mydomain>.com ESMTP"
"SMTPD" 6960 2547 "2022-08-06 15:57:53.839" "85.217.145.172" "RECEIVED: EHLO kphaykwf.ibillcentre.com"
"SMTPD" 6960 2547 "2022-08-06 15:57:53.839" "85.217.145.172" "SENT: 250-mail.<mydomain>.com[nl]250-SIZE 100480000[nl]250-AUTH LOGIN[nl]250 HELP"
"SMTPD" 6960 2547 "2022-08-06 15:57:54.026" "85.217.145.172" "RECEIVED: MAIL FROM:<prs.squ@ibillcentre.com> SIZE=1711156"
"TCPIP" 6960 "2022-08-06 15:57:54.042" "DNS lookup: 172.145.217.85.sbl-xbl.spamhaus.org, 0 addresses found: (none), Match: False"
"TCPIP" 6960 "2022-08-06 15:57:54.245" "DNS lookup: 172.145.217.85.bl.spamcop.net, 0 addresses found: (none), Match: False"
"TCPIP" 6960 "2022-08-06 15:57:54.261" "DNS query failure. Query: 172.145.217.85.combined.njabl.org, Type: A/AAAA, DnsQuery return value: 11002. Message: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server"
"TCPIP" 6960 "2022-08-06 15:57:54.261" "DNS lookup: 172.145.217.85.combined.njabl.org, 0 addresses found: (none), Match: False"
"TCPIP" 6960 "2022-08-06 15:57:54.604" "DNS lookup: 172.145.217.85.cbl.abuseat.org, 0 addresses found: (none), Match: False"
"TCPIP" 6960 "2022-08-06 15:57:55.042" "DNS lookup: 172.145.217.85.dsbl.dnsbl.net.au, 0 addresses found: (none), Match: False"
"TCPIP" 6960 "2022-08-06 15:57:55.042" "DNS query failure. Query: 172.145.217.85.list.dsbl.org, Type: A/AAAA, DnsQuery return value: 11002. Message: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server"
"TCPIP" 6960 "2022-08-06 15:57:55.042" "DNS lookup: 172.145.217.85.list.dsbl.org, 0 addresses found: (none), Match: False"
"TCPIP" 6960 "2022-08-06 15:57:55.057" "DNS lookup: 172.145.217.85.zen.spamhaus.org, 0 addresses found: (none), Match: False"
"DEBUG" 6960 "2022-08-06 15:57:55.057" "Spam test: SpamTestDNSBlackLists, Score: 0"
"DEBUG" 6960 "2022-08-06 15:57:55.057" "Spam test: SpamTestHeloHost, Score: 0"
"DEBUG" 6960 "2022-08-06 15:57:55.073" "Spam test: SpamTestMXRecords, Score: 0"
"DEBUG" 6960 "2022-08-06 15:57:55.073" "Spam test: SpamTestSPF, Score: 0"
"DEBUG" 6960 "2022-08-06 15:57:55.073" "Total spam score: 0

How can I deal with them though?

User avatar
SorenR
Senior user
Senior user
Posts: 6315
Joined: 2006-08-21 15:38
Location: Denmark

Re: Some email bypass spamasssassin

Post by SorenR » 2022-08-06 15:47

RvdH wrote:
2022-08-06 13:18
I have this occasionally, eg: The WinSock error code is 2. Maybe 1, 2 times a days, but there are also days it won't happen at all

If you have any WinSock error code is 2 errors in your hmailserver_error.log you might compare date/time with a message that supposedly passed thru without being checked by SA.
Up till now there is no failsafe solution for the 'WinSock error code is 2' issue, some people claim to have less when they set the amount of command threads higher, but for me that had no effect at all
I run hMailServer on a Windows 2003 R2 and my SpamAssassin on a Windows 2019 Essential. There are no physical shares between them.

If error 2 means "file or directory not found" it can only be hMailServer at fault since hMailServer communicate with SpamAssassin via TCP/IP.

I have an idea that hMailServer "SpamAssassinClient" has not released the received report before another thread is trying to access it OR BOOT is fooked OR Windows got some access rights f-up.

Now on my systems both hMailServer AND SpamAssassin run as Administrator (They are in my attic so NO access unless I say GO) so I don't expect access right problems...

Which leaves hMailServer and/or BOOST. I have previously compiled much newer versions of BOOST with hMailServer and I don't expect such an error to survive 5 years worth of new BOOST versions.

So we are back to hMailServer ... Maybe add an extra layer of "are you really sure the file is not there" code ??

Currently I have no idea exactly where to look for this but probably somewhere in the SpamAssassinClient.code ... I presume ...
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
jim.bus
Senior user
Senior user
Posts: 1571
Joined: 2011-05-28 11:49
Location: US

Re: Some email bypass spamasssassin

Post by jim.bus » 2022-08-06 20:02

You didn't check your Error Log to see if there were any errors there. The Winsock Error will show there if that error occurred. it should also show in one of your Log Entries as well if you have enabled all your Logs. You also didn't say whether or not any Antispam whitelisting was set up in hMailAdmin.

I believe Spamassassin is checked after the other Spamtests are performed. At least it looked that way to me in my own log entries.
If you think you understand quantum mechanics, you don't understand quantum mechanics.

User avatar
johang
Senior user
Senior user
Posts: 1128
Joined: 2008-09-01 09:20

Re: Some email bypass spamasssassin

Post by johang » 2022-08-06 21:24

johnyu2012 wrote:
2022-08-06 14:53

"TCPIP" 6960 "2022-08-06 15:57:54.261" "DNS query failure. Query: 172.145.217.85.combined.njabl.org, Type: A/AAAA, DnsQuery return value: 11002. Message: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server"

"TCPIP" 6960 "2022-08-06 15:57:55.042" "DNS query failure. Query: 172.145.217.85.list.dsbl.org, Type: A/AAAA, DnsQuery return value: 11002. Message: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server"
this has nothing to do with your problem.. however..
you should disable combined.njabl.org in your DNS blacklist in your hmailserver setup because -> https://www.dnsbl.info/dnsbl-njabl-org.php
... and you should disable list.dsbl.org in your DNS blacklist in your hmailserver setup because -> https://www.dnsbl.info/list-dsbl-org.php
you only waste your own resources and possibly make accepting emails slower..

the IP is now blacklisted with zen.spamhaus.org not that it helped at the time of your problem..
lets cheat darwin out of his legacy, find a cure for cancer...

johnyu2012
Normal user
Normal user
Posts: 135
Joined: 2012-09-11 06:33

Re: Some email bypass spamasssassin

Post by johnyu2012 » 2022-08-08 05:55

jim.bus wrote:
2022-08-06 20:02
You didn't check your Error Log to see if there were any errors there. The Winsock Error will show there if that error occurred. it should also show in one of your Log Entries as well if you have enabled all your Logs. You also didn't say whether or not any Antispam whitelisting was set up in hMailAdmin.

I believe Spamassassin is checked after the other Spamtests are performed. At least it looked that way to me in my own log entries.
Yes, I do have winsock error but it does not happen very often (every other few days).

"ERROR" 6964 "2022-08-07 08:20:13.019" "Severity: 3 (Medium), Code: HM5157, Source: SpamAssassinClient::OnReadError, Description: There was a communication error with SpamAssassin. hMailServer tried to retrieve data from SpamAssassin but the connection to SpamAssassin was lost. The WinSock error code is 10054. Enable debug logging to retrieve more information regarding this problem. The problem could be that SpamAssassin is malfunctioning."
"ERROR" 6728 "2022-08-07 08:20:13.035" "Severity: 2 (High), Code: HM5508, Source: SpamAssassinTestConnect::TestConnect, Description: The SpamAssassin tests did not complete. Please confirm that the configuration (host name and port) is valid and that SpamAssassin is running."

I also setup some whitelisting before.

User avatar
jim.bus
Senior user
Senior user
Posts: 1571
Joined: 2011-05-28 11:49
Location: US

Re: Some email bypass spamasssassin

Post by jim.bus » 2022-08-08 06:10

johnyu2012 wrote:
2022-08-08 05:55
jim.bus wrote:
2022-08-06 20:02
You didn't check your Error Log to see if there were any errors there. The Winsock Error will show there if that error occurred. it should also show in one of your Log Entries as well if you have enabled all your Logs. You also didn't say whether or not any Antispam whitelisting was set up in hMailAdmin.

I believe Spamassassin is checked after the other Spamtests are performed. At least it looked that way to me in my own log entries.
Yes, I do have winsock error but it does not happen very often (every other few days).

"ERROR" 6964 "2022-08-07 08:20:13.019" "Severity: 3 (Medium), Code: HM5157, Source: SpamAssassinClient::OnReadError, Description: There was a communication error with SpamAssassin. hMailServer tried to retrieve data from SpamAssassin but the connection to SpamAssassin was lost. The WinSock error code is 10054. Enable debug logging to retrieve more information regarding this problem. The problem could be that SpamAssassin is malfunctioning."
"ERROR" 6728 "2022-08-07 08:20:13.035" "Severity: 2 (High), Code: HM5508, Source: SpamAssassinTestConnect::TestConnect, Description: The SpamAssassin tests did not complete. Please confirm that the configuration (host name and port) is valid and that SpamAssassin is running."

I also setup some whitelisting before.
This would mean for those email messages where the Winsock Error occurred that Spamassassin did not analyze the email message and where Whitelisting occurred all SPAM Checking did not get performed. With the Spamassassin Error you did not loose that email message. You just didn't get any Spamassassin checking but the other SPAM checking activities did occur even though you received the Winsock Error. Whitelisting should also bypass using Spamassassin as well as well as the other Antispam checks of hMailServer. With Whitelisting all SPAM Checks are supposed to be bypassed is my understanding.
If you think you understand quantum mechanics, you don't understand quantum mechanics.

User avatar
jim.bus
Senior user
Senior user
Posts: 1571
Joined: 2011-05-28 11:49
Location: US

Re: Some email bypass spamasssassin

Post by jim.bus » 2022-08-08 08:32

I have also just now gone back and looked at your posted Log Entries.

You appear to not have listed all your Log Entries. As I've already stated, the Winsock Error should be showing if it appeared in the Error Log. However, the timestamps don't seem to match the Log Entries you are showing and complaining that Spamasssassin seems to be bypassed. Therefore the Winsock error doesn't appear to be a cause for these the email message these Log Entries are for.

But it sill looks like you have not provided all the Log Entries for the given email message you are saying that had Spamassassin bypassed. If that is so then I cannot confirm whether Spamassassin has actually been bypassed or not. You seem to also have edited the Log Entries to some extent.

I have compared your Log Entries to one of my own Log Entries which uses Spamassassin. I see differences in your Log Entires compared to mine. My Log Entries have two Entries which state 'Total spam score'. One Total SPAM Score for the Built in Antispam and one Total SPAM Score for Spamassassin. It is possible you have not found all the Log Entries for your particular Email Message you claim bypassed Spamassassin. You should look again and see if you have missed any Log Entries. And I would PREFER SEEING ALL LOG ENTRIES ASSOCIATED WITH THIS ONE PARTICULAR EMAIL MESSAGE. You have not included all the Log Entries that would have been produced by this email message if it was successfully delivered to you.
If you think you understand quantum mechanics, you don't understand quantum mechanics.

User avatar
SorenR
Senior user
Senior user
Posts: 6315
Joined: 2006-08-21 15:38
Location: Denmark

Re: Some email bypass spamasssassin

Post by SorenR » 2022-08-08 09:32

jim.bus wrote:
2022-08-08 06:10
This would mean for those email messages where the Winsock Error occurred that Spamassassin did not analyze the email message and where Whitelisting occurred all SPAM Checking did not get performed. With the Spamassassin Error you did not loose that email message. You just didn't get any Spamassassin checking but the other SPAM checking activities did occur even though you received the Winsock Error. Whitelisting should also bypass using Spamassassin as well as well as the other Antispam checks of hMailServer. With Whitelisting all SPAM Checks are supposed to be bypassed is my understanding.
I get these errors too and when I compare logs I see that SpamAssassin DID check the email and DID deliver a report ... hMailServer (BOOST ??) just made a f*up of it.

I'm just curious why noone pursued this f*up.

Only thing I have not done is throwing WireShark at it as I have SpamAss on another windows server from hMailServer.
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Some email bypass spamasssassin

Post by mattg » 2022-08-08 10:43

i actually think it is a time out, in that Spamassassin takes too long for hMailserver to wait

I do have these in my hmailserver.ini (Bill's ini settings), but they don't seem to do much

Code: Select all

SAMinTimeout=1500
; 30 seconds is default

SAMaxTimeout=2400
; 90 seconds is default
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jim.bus
Senior user
Senior user
Posts: 1571
Joined: 2011-05-28 11:49
Location: US

Re: Some email bypass spamasssassin

Post by jim.bus » 2022-08-08 10:53

SorenR wrote:
2022-08-08 09:32
jim.bus wrote:
2022-08-08 06:10
This would mean for those email messages where the Winsock Error occurred that Spamassassin did not analyze the email message and where Whitelisting occurred all SPAM Checking did not get performed. With the Spamassassin Error you did not loose that email message. You just didn't get any Spamassassin checking but the other SPAM checking activities did occur even though you received the Winsock Error. Whitelisting should also bypass using Spamassassin as well as well as the other Antispam checks of hMailServer. With Whitelisting all SPAM Checks are supposed to be bypassed is my understanding.
I get these errors too and when I compare logs I see that SpamAssassin DID check the email and DID deliver a report ... hMailServer (BOOST ??) just made a f*up of it.

I'm just curious why noone pursued this f*up.

Only thing I have not done is throwing WireShark at it as I have SpamAss on another windows server from hMailServer.
All I ever noted was that I could confirm the Email Message was delivered to my hMailServer but with hMailServer reporting that Spamassassin had an error. To me, therefore, it looked like Spamassasin just failed to do any processing and the Email Delivery just completed. While I don't monitor everybody as to whether or not they had any suggestions as to the cause, the only person to seemingly have any suspicion as to specifically where the cause actually was is you. But I think I remember mattg commenting about this error but I don't think he had any ideas as to how to correct the cause but I can't be sure. A possible reason for no one doing anything about it is that the frequency of this error was not seemingly a lot and nothing very deleterious happened. I once went 4 months without any Winsock Error. I believe it was thought it was a Spamassassin problem but again I can't remember for sure.

However, if you saw my earlier Postings to johnu2012, you should see I'm suspecting that he didn't show all his Log Entries and if he does find more Log Entries, we may very well see that Spamassassin wasn't bypassed. Other places where he might think Spamassassin was bypassed might be because he Whitelisted the Email ID that is sending the email message to his hMailServer. The Log Entries he shows doesn't show enough of what transpired in that particular Email Message being received. Also, based on the Log Entries he shows it would appear and as my own Log Entries for received email messages shows, it looks like hMailServer's Help Documentation is incorrect in that it states that anti-spam processing takes place after the RCPT TO command but I can see where it is taking place after the MAIL FROM command.
If you think you understand quantum mechanics, you don't understand quantum mechanics.

User avatar
SorenR
Senior user
Senior user
Posts: 6315
Joined: 2006-08-21 15:38
Location: Denmark

Re: Some email bypass spamasssassin

Post by SorenR » 2022-08-08 10:54

mattg wrote:
2022-08-08 10:43
i actually think it is a time out, in that Spamassassin takes too long for hMailserver to wait

I do have these in my hmailserver.ini (Bill's ini settings), but they don't seem to do much

Code: Select all

SAMinTimeout=1500
; 30 seconds is default

SAMaxTimeout=2400
; 90 seconds is default
If the errorcode 2 is "file or folder not found" should hMailServer not .... eh ... Never mind :roll:

Code: Select all

   void
   SpamAssassinClient::OnConnectionTimeout()
   {
      // do nothing
   }
https://github.com/hmailserver/hmailser ... #L113-L117
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
SorenR
Senior user
Senior user
Posts: 6315
Joined: 2006-08-21 15:38
Location: Denmark

Re: Some email bypass spamasssassin

Post by SorenR » 2022-08-08 11:02

SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-08 11:16

SorenR wrote:
2022-08-08 11:02
Problem is here...

https://github.com/hmailserver/hmailser ... #L257-L281
Nah, that just the error message, the problem is elsewhere...and not even related to timeout, i get winsock 2 errors on messages that are successfully processed by SA in a few seconds

Code: Select all

Mon Aug  8 10:17:33 2022 [-11924] info: spamd: connection from Server [127.0.0.1]:62433 to port 783, fd 6
Mon Aug  8 10:17:33 2022 [-11924] info: spamd: processing message <1b71f10012c0606c14c56e3dd.e770b041cc.20220808081719.748f9efe22.8b3fa1b6@mail121.sea91.rsgsv.net> for (unknown):0
Mon Aug  8 10:17:37 2022 [-11924] info: spamd: clean message (-2.1/4.0) for (unknown):0 in 4.0 seconds, 129936 bytes.
Mon Aug  8 10:17:37 2022 [-11924] info: spamd: result: . -2 - BAYES_00,DKIM_INVALID,DKIM_SIGNED,FMBLA_HELO_OUTMX,FMBLA_RDNS_OUTMX,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,KAM_DMARC_STATUS,LOCAL_SCAM_15,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE,RCVD_IN_UCEPROTECT3,SPF_HELO_PASS,SPF_PASS,TXREP,T_KAM_HTML_FONT_INVALID,T_SCC_BODY_TEXT_LINE,URIBL_AMI_WHITE,URIBL_GREY scantime=4.0,size=129936,user=(unknown),uid=0,required_score=4.0,rhost=Server,raddr=127.0.0.1,rport=62433,mid=<1b71f10012c0606c14c56e3dd.e770b041cc.20220808081719.748f9efe22.8b3fa1b6@mail121.sea91.rsgsv.net>,bayes=0.000000,autolearn=no autolearn_force=no

Code: Select all

"ERROR"	11864	"2022-08-08 10:17:37.220"	"Severity: 3 (Medium), Code: HM5157, Source: SpamAssassinClient::OnReadError, Description: There was a communication error with SpamAssassin. hMailServer tried to retrieve data from SpamAssassin but the connection to SpamAssassin was lost. The WinSock error code is 2. Enable debug logging to retrieve more information regarding this problem. The problem could be that SpamAssassin is malfunctioning."
"ERROR"	10640	"2022-08-08 10:17:37.220"	"Severity: 2 (High), Code: HM5508, Source: SpamAssassinTestConnect::TestConnect, Description: The SpamAssassin tests did not complete. Please confirm that the configuration (host name and port) is valid and that SpamAssassin is running."
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6315
Joined: 2006-08-21 15:38
Location: Denmark

Re: Some email bypass spamasssassin

Post by SorenR » 2022-08-08 11:30

I've added some LOG_DEBUG() to the code at various places... Now I just need to trigger the error :roll:
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-08 11:35

The error code 2 is the boost EOF (End Of File) error, according to this post: https://www.hmailserver.com/forum/viewt ... 25#p184625
But after this post, https://www.hmailserver.com/forum/viewt ... 15#p185215 the person never reported back if the problem went away
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6315
Joined: 2006-08-21 15:38
Location: Denmark

Re: Some email bypass spamasssassin

Post by SorenR » 2022-08-08 11:44

RvdH wrote:
2022-08-08 11:16

Code: Select all

"ERROR"	10640	"2022-08-08 10:17:37.220"	"Severity: 2 (High), Code: HM5508, Source: SpamAssassinTestConnect::TestConnect, Description: The SpamAssassin tests did not complete. Please confirm that the configuration (host name and port) is valid and that SpamAssassin is running."
That error (5508) comes from SpamTestSpamAssassin.cpp and not from SpamAssassinTestConnect !

https://github.com/hmailserver/hmailser ... p#L97-L103
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-08 12:36

SorenR wrote:
2022-08-08 11:44
RvdH wrote:
2022-08-08 11:16

Code: Select all

"ERROR"	10640	"2022-08-08 10:17:37.220"	"Severity: 2 (High), Code: HM5508, Source: SpamAssassinTestConnect::TestConnect, Description: The SpamAssassin tests did not complete. Please confirm that the configuration (host name and port) is valid and that SpamAssassin is running."
That error (5508) comes from SpamTestSpamAssassin.cpp and not from SpamAssassinTestConnect !

https://github.com/hmailserver/hmailser ... p#L97-L103
So, you found a typo? (sloppy copy & paste work i guess) :lol:

Both instances should read SpamTestSpamAssassin::RunTest
https://github.com/hmailserver/hmailser ... in.cpp#L83
https://github.com/hmailserver/hmailser ... in.cpp#L99
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-08 19:11

My best bet it is caused anywhere between https://github.com/hmailserver/hmailser ... #L445-L588

Google: 'boost async_read eof' and you notice quite a few result similar to what we are experiencing with SA
Not sure if we could simply ignore error of type "boost::asio::error::eof" (as if understand right, it is not really a error)
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: Some email bypass spamasssassin

Post by jimimaseye » 2022-08-08 19:57

A nice reminder of this being discussed years ago when 'Binkle' (of Jam Spamassassin) also had a look and suggested Boost as the culprit (read on the next few posts): https://www.hmailserver.com/forum/viewt ... 82#p181582

And my example highlighted here shows that SA had received the email only for HMS give up before it successfully received the message back.

Then Martin got involved (and 'Superman20' did some good analysis)........
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-08 20:36

jimimaseye wrote:
2022-08-08 19:57
A nice reminder of this being discussed years ago when 'Binkle' (of Jam Spamassassin) also had a look and suggested Boost as the culprit (read on the next few posts): https://www.hmailserver.com/forum/viewt ... 82#p181582

And my example highlighted here shows that SA had received the email only for HMS give up before it successfully received the message back.

Then Martin got involved (and 'Superman20' did some good analysis)........
Nothing new there, all mentioned/referenced ^
Even your posted example is exactly the same as what i posted when i initially posted that topic
Superman20's analysis is good, but incomplete
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-08 21:24

Code: Select all

   void 
   TCPConnection::AsyncReadCompleted(const boost::system::error_code& error,  size_t bytes_transferred)
   {
      UpdateAutoLogoutTimer();

      if (error.value() != 0 && error.value() != boost::asio::error::eof)
      {
         if (connection_state_ != StateConnected)
         {
         ....
         
Running this for about an hour now, one remarkable observation i have no more EOF errors in my IMAP logs, eg: The read operation failed. Bytes transferred: 0 Remote IP: xxx.xxx.xxx.xxx, Session: 9204, Code: 2, Message: End of file, coincidence?
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-09 10:13

Ignore the above code, this will introduce other issues, but i think i have pinpointed down the culprit.

TCPConnection::AsyncRead uses transfer_at_least(1), which expect at least 1 byte to be transferred but if no more bytes are transferred as there no more bytes in the (file)stream, this results in the mentioned boost::asio::error::eof (boost error code 2, which is not a winsock error as reported by HMS)

using this example
"DEBUG" 3424 "2015-05-15 16:22:21.717" "Parsing response from SpamAssassin. Session 30202"
"DEBUG" 3424 "2015-05-15 16:22:21.811" "The read operation failed. Bytes transferred: 0 Remote IP: 127.0.0.1, Session: 30202, Code: 2, Message: End of file"
"DEBUG" 3424 "2015-05-15 16:22:21.811" "Ending session 30202"
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6315
Joined: 2006-08-21 15:38
Location: Denmark

Re: Some email bypass spamasssassin

Post by SorenR » 2022-08-09 17:21

So, change boost::asio::transfer_at_least(1) to boost::asio::transfer_all()

boost::asio::transfer_at_least(1) is only used in TCPConnection.cpp and HTTPClient.cpp.
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
SorenR
Senior user
Senior user
Posts: 6315
Joined: 2006-08-21 15:38
Location: Denmark

Re: Some email bypass spamasssassin

Post by SorenR » 2022-08-09 18:12

SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-09 18:14

no, no boost::asio::transfer_all() is :evil: and will give you undesired results (tried it :lol: )
remember besides the internal communication with SA TCPConnection.cpp is used for IMAP, POP3, SMTP protocols as well

We simply use transfer_at_least(1) and then need to ignore boost::asio::error::eof, so it doesn't throw the error and continues reading the stream, but i am not completely sure what would be the best approach

Most straight forward method would be

Code: Select all

         if (connection_state_ != StateConnected)
         {
            // The read failed, but we've already started the disconnection. So we should not log the failure
            // or enqueue a new disconnect.
            return;
         }

         // Ignore end_of_stream error, there may still be data in the receive buffer we can read.
         if (error.value() == boost::asio::error::eof)
         {
            return;
         }

Another approach...if EnqueueDisconnect(); would be mandatory
not sure about the EnqueueDisconnect(); placement, but i havent seen a winsock error 2 error whole day although that might be luck, and needs additional testing

Code: Select all

         if (connection_state_ != StateConnected)
         {
            // The read failed, but we've already started the disconnection. So we should not log the failure
            // or enqueue a new disconnect.
            return;
         }

         // Ignore end_of_stream error, there may still be data in the receive buffer we can read.
         if (error.value() != boost::asio::error::eof)
         {

            OnReadError(error.value());

            String message;
            message.Format(_T("The read operation failed. Bytes transferred: %d"), bytes_transferred);
            ReportDebugMessage(message, error);

            if (error.value() == boost::asio::error::not_found)
            {
               // read buffer is full...
               OnExcessiveDataReceived();
            }

            //EnqueueDisconnect();
         }

         EnqueueDisconnect();
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6315
Joined: 2006-08-21 15:38
Location: Denmark

Re: Some email bypass spamasssassin

Post by SorenR » 2022-08-09 19:37

https://www.boost.org/doc/libs/1_35_0/d ... n/eof.html

An EOF error may be used to distinguish the end of a stream from a successful read of size 0.

Well.... OK then ... :roll:
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-09 19:45

SorenR wrote:
2022-08-09 19:37
https://www.boost.org/doc/libs/1_35_0/d ... n/eof.html

An EOF error may be used to distinguish the end of a stream from a successful read of size 0.

Well.... OK then ... :roll:
Exactly, therefor i stated it is not really a error and could/should be ignored

But in HMS this was treated as error, eg:

Code: Select all

   void 
   TCPConnection::AsyncReadCompleted(const boost::system::error_code& error,  size_t bytes_transferred)
   {
      UpdateAutoLogoutTimer();

      if (error.value() != 0)
Have to have patience with this one though, hard to test with the error only occurring occasional, normally maybe 1, 2 times a days, but there are also days it won't happen at all
Guess if i ran this for a week without errors it fairly safe to say it works

Or.... martin must come up with something else, https://github.com/hmailserver/hmailserver/issues/167
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6315
Joined: 2006-08-21 15:38
Location: Denmark

Re: Some email bypass spamasssassin

Post by SorenR » 2022-08-10 00:13

Mmmm.....

https://www.boost.org/doc/libs/1_42_0/d ... client.cpp

Code sample:

Code: Select all

bla
bla
bla

      // Read until EOF, writing data to output as we go.
      while (boost::asio::read(socket, response,
         boost::asio::transfer_at_least(1), error))
         std::cout << &response;
      if (error != boost::asio::error::eof)
         throw boost::system::system_error(error);

bla
bla
bla
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-10 00:17

I used this example code as reference, scroll down to the bottom, https://www.boost.org/doc/libs/1_70_0/d ... client.cpp

Code: Select all

    // Read until EOF, writing data to output as we go.
    boost::system::error_code error;
    while (boost::asio::read(socket, response,
          boost::asio::transfer_at_least(1), error))
      std::cout << &response;
    if (error != boost::asio::error::eof)
      throw boost::system::system_error(error);
As you can see, they also ignore the boost::asio::error::eof in that example code, so i don't understand your 'Mmmm.....'

The only difference we read is async, using async_read or async_read_until in void TCPConnection::AsyncRead(const AnsiString &delimitor)
And the response (or error) is handled in void TCPConnection::AsyncReadCompleted(const boost::system::error_code& error, size_t bytes_transferred)
https://github.com/hmailserver/hmailser ... #L445-L588
Last edited by RvdH on 2022-08-10 00:27, edited 1 time in total.
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6315
Joined: 2006-08-21 15:38
Location: Denmark

Re: Some email bypass spamasssassin

Post by SorenR » 2022-08-10 00:25

Ah, missed that. Sorry :oops:
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-10 23:35

Looks promising, still without winsock error 2 for 48h now🤞🏻

FyI, I am using the second variant of the proposed fixes
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

palinka
Senior user
Senior user
Posts: 4461
Joined: 2017-09-12 17:57

Re: Some email bypass spamasssassin

Post by palinka » 2022-08-11 09:25

RvdH wrote:
2022-08-10 23:35
Looks promising, still without winsock error 2 for 48h now🤞🏻

FyI, I am using the second variant of the proposed fixes
Good job. This will be the biggest development breakthrough since the addition of OnHELO. 👍

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-11 09:33

palinka wrote:
2022-08-11 09:25
RvdH wrote:
2022-08-10 23:35
Looks promising, still without winsock error 2 for 48h now🤞🏻

FyI, I am using the second variant of the proposed fixes
Good job. This will be the biggest development breakthrough since the addition of OnHELO. 👍
Not sure if this the biggest development breakthrough, but with this the biggest annoyance i had with HMS seems to be history :mrgreen:
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

palinka
Senior user
Senior user
Posts: 4461
Joined: 2017-09-12 17:57

Re: Some email bypass spamasssassin

Post by palinka » 2022-08-11 09:54

RvdH wrote:
2022-08-11 09:33
palinka wrote:
2022-08-11 09:25
RvdH wrote:
2022-08-10 23:35
Looks promising, still without winsock error 2 for 48h now🤞🏻

FyI, I am using the second variant of the proposed fixes
Good job. This will be the biggest development breakthrough since the addition of OnHELO. 👍
Not sure if this the biggest development breakthrough, but with this the biggest annoyance i had with HMS seems to be history :mrgreen:
Either way, its big.

I use sub OnError to send SMS with error details. This one constitutes the largest category except for "Delivery Failure - To: user@domain, Error: Message delivery cancelled during global rules". Actually, that's one I'd like to suppress. I think I'll do that now that I'm thinking about it. :D

Edit - OOPS... that's not even part of OnError. Its something else I created at OnDeliveryFailed. Duh. Anyway, even easier to suppress. :lol:

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: Some email bypass spamasssassin

Post by jimimaseye » 2022-08-11 10:04

This sounds good. Presumably the lack of errors has also meant that every message has been proved to be passed to SA and back again (in other words, has every message been returned with SA scoring)?

Probably worth a check.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-11 10:22

jimimaseye wrote:
2022-08-11 10:04
This sounds good. Presumably the lack of erros has also meant that every message has been proved to be passed to SA and back again (in other words, has every message been returned with SA scoring)?
That was never a issue, both HMS and SA did send/receive the request/response successfully all this time, boost::asio::error::eof faulty triggered the Winsock 2 error that made HMS stop processing the SA response (although it was there and ready to be processed)
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: Some email bypass spamasssassin

Post by jimimaseye » 2022-08-11 10:28

Oh really? My experince was that after winsock error the message was processed by SA but failed to get passed back with the additional SA headers (like this: https://www.hmailserver.com/forum/viewt ... 64#p182064). (In other words SA logs showed the processing and scoring as you expect but the SA headers never appeared in the resultant message in HMS)
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-11 10:41

jimimaseye wrote:
2022-08-11 10:28
Oh really? My experince was that after winsock error the message was processed by SA but failed to get passed back with the additional SA headers (like this: https://www.hmailserver.com/forum/viewt ... 64#p182064). (In other words SA logs showed the processing and scoring as you expect but the SA headers never appeared in the resultant message in HMS)
Exactly, that is what i am saying, not? Due to the boost::asio::error::eof HMS (faulty) triggered the Winsock 2 error and therefor the SA response was never read/processed by HMS, by ignoring boost::asio::error::eof HMS can read/process the response up till the moment the EOF (error.value() == 2) occurred (which equals the whole response send by SA)
if (error.value() != 0) <-- this included boost::asio::error::eof, eg: boost error enum 2, which triggered the HMS OnReadError Event (winsock error 2)
{
OnReadError(error.value());
}
Boost docs
An EOF error may be used to distinguish the end of a stream from a successful read of size 0.
may is the secret word there, we do not need to and therefor can ignore boost::asio::error::eof
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
katip
Senior user
Senior user
Posts: 1161
Joined: 2006-12-22 07:58
Location: Istanbul

Re: Some email bypass spamasssassin

Post by katip » 2022-08-11 11:28

RvdH wrote:
2022-08-11 09:33
Not sure if this the biggest development breakthrough, but with this the biggest annoyance i had with HMS seems to be history :mrgreen:
great news. in case we're sure that this annoyance is definitely addressed, i hope 5.7 will take it's share too?? :wink:
Katip
--
HMS 5.7, MariaDB 10.4.10, SA 4.0.0, ClamAV 0.103.8

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: Some email bypass spamasssassin

Post by jimimaseye » 2022-08-11 12:54

RvdH wrote:
2022-08-11 10:41
jimimaseye wrote:
2022-08-11 10:28
Oh really? My experince was that after winsock error the message was processed by SA but failed to get passed back with the additional SA headers (like this: https://www.hmailserver.com/forum/viewt ... 64#p182064). (In other words SA logs showed the processing and scoring as you expect but the SA headers never appeared in the resultant message in HMS)
Exactly, that is what i am saying, not? Due to the boost::asio::error::eof HMS (faulty) triggered the Winsock 2 error and therefor the SA response was never read/processed by HMS,...
Yep, all good, understood.

So previously the issue was that the message, when being passed back to HMS (with SA headers), there was:

a, a reported (logged) error and
b, the modified message was not being received (because of HMS erroring).

Therefore, in these occasions, the headers were not apparent.

What I was asking was, as part of full testing of your solution to ensure everything is covered, that we now have full guarantee that:

a, there is no longer a reported error AND
b, all messages are now being received with the SA scores (as well as the absence of a reported error).

You have confirmed that for a couple of days now you have not had a reported error - just need to sheck that functionality is still as required. (Belts and braces an' all that.)
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-11 13:52

jimimaseye wrote:
2022-08-11 12:54
RvdH wrote:
2022-08-11 10:41
jimimaseye wrote:
2022-08-11 10:28
Oh really? My experince was that after winsock error the message was processed by SA but failed to get passed back with the additional SA headers (like this: https://www.hmailserver.com/forum/viewt ... 64#p182064). (In other words SA logs showed the processing and scoring as you expect but the SA headers never appeared in the resultant message in HMS)
Exactly, that is what i am saying, not? Due to the boost::asio::error::eof HMS (faulty) triggered the Winsock 2 error and therefor the SA response was never read/processed by HMS,...
Yep, all good, understood.

So previously the issue was that the message, when being passed back to HMS (with SA headers), there was:

a, a reported (logged) error and
b, the modified message was not being received (because of HMS erroring).

Therefore, in these occasions, the headers were not apparent.

What I was asking was, as part of full testing of your solution to ensure everything is covered, that we now have full guarantee that:

a, there is no longer a reported error AND
b, all messages are now being received with the SA scores (as well as the absence of a reported error).

You have confirmed that for a couple of days now you have not had a reported error - just need to sheck that functionality is still as required. (Belts and braces an' all that.)
a & b both are true as far as ik can tell, although it is hard to simulate a error that only pops up once in a while

But this not only is helpful with SA, this also gets rid of other weird (unwanted) errors in your logs where boost::asio::error::eof was triggered, for example IMAP, POP and SMTP
eg: The read operation failed. Bytes transferred: 0 Remote IP: xxx.xxx.xxx.xxx, Session: xxx, Code: 2, Message: End of file, eg: https://github.com/hmailserver/hmailserver/issues/195

Like to test? Usual location, *.49
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: Some email bypass spamasssassin

Post by jimimaseye » 2022-08-11 15:39

RvdH wrote:
2022-08-11 13:52
a & b both are true as far as ik can tell, although it is hard to simulate a error that only pops up once in a while
Im sure you are right but it was worth mentioning for double security.

Perhaps with the use of notepad++, a drag and drop of a days worth of emails (.EML) , and a search for all emails where "X-Spam-Status:" doesnt exist, that might show/highlight any.

Just a suggestion.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-11 16:03

"ERROR" 6548 "2022-08-11 15:28:39.544" "Severity: 2 (High), Code: HM5508, Source: SpamTestSpamAssassin::RunTest, Description: The SpamAssassin tests did not complete. Please confirm that the configuration (host name and port) is valid and that SpamAssassin is running."
God damn :twisted:

Maybe i should not call EnqueueDisconnect(); after if (error.value() != boost::asio::error::eof){}
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-12 18:01

RvdH wrote:
2022-08-11 13:52
Like to test? Usual location, *.49
made another build, without call to EnqueueDisconnect();
*.49 (old one renamed to OLD)
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-13 15:40

"ERROR" 1224 "2022-08-13 13:22:10.218" "Severity: 2 (High), Code: HM5508, Source: SpamTestSpamAssassin::RunTest, Description: The SpamAssassin tests did not complete. Please confirm that the configuration (host name and port) is valid and that SpamAssassin is running."
Damn, this is not really going anywhere, i'm quite positive i am looking @ the right code...it has to be fixed within TCPConnection::AsyncReadCompleted :!:
https://github.com/hmailserver/hmailser ... #L476-L569

Have gone back to my original code, eg:

Code: Select all

      if (error.value() != 0 && error.value() != boost::asio::error::eof)
      {
            ...
      }
All though that change alone, give unexpected/undesired command in your logs , example:

Code: Select all

"IMAPD"	4472	48	"2022-08-13 14:40:40.909"	"::1"	"SENT:  BAD NULL COMMAND"
Now trying:

Code: Select all

            if (s.size() > 0)
            {
               try
               {
                  ParseData(s);
               }
               catch (DisconnectedException&)
               {
                  throw;
               }
               catch (...)
               {
                  String message;
                  message.Format(_T("An error occured while parsing data. Data length: %d, Data: %s."), s.size(), String(s).c_str());

                  ReportError(ErrorManager::Medium, 5136, "TCPConnection::AsyncReadCompleted", message);

                  throw;
               }
            }
This seems to get rid of the BAD NULL COMMAND's, now i need to wait again, until SA craps out, if it craps out :)

Anyone, has other suggestions?
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6315
Joined: 2006-08-21 15:38
Location: Denmark

Re: Some email bypass spamasssassin

Post by SorenR » 2022-08-13 16:54

He talk about replacing async_read with async_read_until but also getting the result by using boost::asio::transfer_at_least(1) inside async_read ...

https://stackoverflow.com/questions/174 ... file-error

Just browsing the 'net for an easy solution :mrgreen:
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-13 17:01

SorenR wrote:
2022-08-13 16:54
He talk about replacing async_read with async_read_until but also getting the result by using boost::asio::transfer_at_least(1) inside async_read ...

https://stackoverflow.com/questions/174 ... file-error

Just browsing the 'net for an easy solution :mrgreen:
Read until what? async_read_until expects a fixed buffer size, as this is a variable buffer size, eg: commands send bij client(s) it is kinda hard to use async_read_until (i think)
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 6315
Joined: 2006-08-21 15:38
Location: Denmark

Re: Some email bypass spamasssassin

Post by SorenR » 2022-08-13 17:17

RvdH wrote:
2022-08-13 17:01
SorenR wrote:
2022-08-13 16:54
He talk about replacing async_read with async_read_until but also getting the result by using boost::asio::transfer_at_least(1) inside async_read ...

https://stackoverflow.com/questions/174 ... file-error

Just browsing the 'net for an easy solution :mrgreen:
Read until what? async_read_until expects a fixed buffer size, as this is a variable buffer size, eg: commands send bij client(s) it is kinda hard to use async_read_until (i think)
Just tossing ideas out on the table. Some go over and end up on the floor, and some stay :mrgreen:
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
SorenR
Senior user
Senior user
Posts: 6315
Joined: 2006-08-21 15:38
Location: Denmark

Re: Some email bypass spamasssassin

Post by SorenR » 2022-08-13 17:24

Do I understand this correctly that all events (SA, SMTP, IMAP and POP3) reporting the "Winsock 2 error" really is about the sender is done and disconnecting from hMailServer and hMailServer is not "getting the message"? Even if hMailServer thinks it is an error the data is still in the buffer and hMailServer just need to read it and deal with it ... or?
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-13 17:37

SorenR wrote:
2022-08-13 17:24
Do I understand this correctly that all events (SA, SMTP, IMAP and POP3) reporting the "Winsock 2 error" really is about the sender is done and disconnecting from hMailServer and hMailServer is not "getting the message"? Even if hMailServer thinks it is an error the data is still in the buffer and hMailServer just need to read it and deal with it ... or?
if the error = 2 then yes, then it's EOF and the file/stream exists and is read, deleted or whatever to command was... not sure if HMS is not "getting the message", as it is read, deleted or whatever successfully, might not be a issue there and only have the annoying winsock error 2 effect on SA

maybe HMS should differentiatie between EOF and 0 size read, but that is a whole other story (if (s.size() > 0) is a start :wink: )

the receive_buffer_ is read line by line, eg:

std::string s;
std::istream is(&receive_buffer_);
std::getline(is, s, '\r');

// consume trailing \n on line.
receive_buffer_.consume(1);

if the last read line of a variable buffer, eg: s.size() == 0 it now won't try to parse the data, as this resulted in: BAD NULL COMMAND
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
jim.bus
Senior user
Senior user
Posts: 1571
Joined: 2011-05-28 11:49
Location: US

Re: Some email bypass spamasssassin

Post by jim.bus » 2022-08-14 20:51

I believe I found an example of when Spamassassin was not executed in my logs from today.

There were no errors in my logs for any connections.

Antispam was enabled and processed.

DNS Blaclists Total SPAM Score = 0.

SURBL Total SPAM Score = 6.

Total Size of Message as reported in Outlook 247 KB.

SPAM Mark Threshold = 5.

SPAM Delete Threshold = 0. Nothing in Help Documentation regarding a SPAM Delete Threshold of 0 such as the Delete process not being enabled.

Maximum message size to scan = 1024 KB.

Message was marked as SPAM in the Subject saved and processed for Delivery.

I can find nothing in the Help Documentation regarding this situation where this situation occurs but the only peculiar thing I can see in this example is that the SPAM Delete Threshold was 0. So this message should have been deleted but instead it was Marked as SPAM and passed on to be processed for Delivery.

Another peculiarity is I believe I originally had the SPAM Delete Threshold set to 50 and now it is set at 0. A few days ago, I upgraded to Beta Version hMailServer 5.6.9-B2602. If my memory as to this value being set at 50 is correct then it may be possible the installation process defaulted this SPAM Delete Threshold to 0. I had looked at my SPAM Delete Threshold value very recently probably when I first looked at this Forum Topic.
If you think you understand quantum mechanics, you don't understand quantum mechanics.

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: Some email bypass spamasssassin

Post by jimimaseye » 2022-08-14 21:27

Anything in spamd.log?

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
jim.bus
Senior user
Senior user
Posts: 1571
Joined: 2011-05-28 11:49
Location: US

Re: Some email bypass spamasssassin

Post by jim.bus » 2022-08-14 21:46

jimimaseye wrote:
2022-08-14 21:27
Anything in spamd.log?

[Entered by mobile. Excuse my spelling.]
There would be nothing in spamd.log because Spamassassin was never called in the first place. hMailServer just skipped connecting to Spamassassin all together. No references to Spamassassin were in the Log Entries for this email message.
If you think you understand quantum mechanics, you don't understand quantum mechanics.

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-14 23:40

Sincerely doubt that, SA always reports errors, see hmailserver_error log

If SA didn't report error we would not have this discussion, would we?
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
jim.bus
Senior user
Senior user
Posts: 1571
Joined: 2011-05-28 11:49
Location: US

Re: Some email bypass spamasssassin

Post by jim.bus » 2022-08-15 01:18

RvdH wrote:
2022-08-14 23:40
Sincerely doubt that, SA always reports errors, see hmailserver_error log

If SA didn't report error we would not have this discussion, would we?
See what I said in my original statement.
There were no errors in my logs for any connections.
And what I responded to jimimaseye.
There would be nothing in spamd.log because Spamassassin was never called in the first place. hMailServer just skipped connecting to Spamassassin all together. No references to Spamassassin were in the Log Entries for this email message.
There were no Error Log Entries in hMailServer period. And in respect to this particular Email Message, hMailServer did not make any connection to Spamassassin. Therefore, Spamassassin would not report any errors either because the email message was never passed to Spamassassin to report on it.

In short, for this email message Spamassassin was not called by hMailServer. There were no errors but hMailServer marked message as SPAM due to the SURBL checks. After the SURBL checks were performed hMailServer proceeded to delivering the email message.

Here are my Log Entries:

"TCPIP" 128780 "2022-08-14 08:58:34.763" "TCP - 156.70.63.119 connected to 192.168.x.x:25."
"DEBUG" 128780 "2022-08-14 08:58:34.763" "TCP connection started for session 100"
"SMTPD" 128780 100 "2022-08-14 08:58:34.763" "156.70.63.119" "SENT: 220 Pleased To Meet You"
"SMTPD" 102996 100 "2022-08-14 08:58:35.029" "156.70.63.119" "RECEIVED: EHLO mta-70-63-119.sparkpostmail.com"
"SMTPD" 102996 100 "2022-08-14 08:58:35.029" "156.70.63.119" "SENT: 250-mail.xxxx.com[nl]250-SIZE 25600000[nl]250-STARTTLS[nl]250 HELP"
"SMTPD" 128780 100 "2022-08-14 08:58:35.076" "156.70.63.119" "RECEIVED: STARTTLS"
"SMTPD" 128780 100 "2022-08-14 08:58:35.076" "156.70.63.119" "SENT: 220 Ready to start TLS"
"DEBUG" 127452 "2022-08-14 08:58:35.076" "Performing SSL/TLS handshake for session 100. Verify certificate: False"
"TCPIP" 127452 "2022-08-14 08:58:35.200" "TCPConnection - TLS/SSL handshake completed. Session Id: 100, Remote IP: 156.70.63.119, Version: TLSv1.2, Cipher: ECDHE-ECDSA-AES128-GCM-SHA256, Bits: 128"
"SMTPD" 127452 100 "2022-08-14 08:58:35.263" "156.70.63.119" "RECEIVED: EHLO mta-70-63-119.sparkpostmail.com"
"SMTPD" 127452 100 "2022-08-14 08:58:35.263" "156.70.63.119" "SENT: 250-mail.xxxx.com[nl]250-SIZE 25600000[nl]250 HELP"
"SMTPD" 128780 100 "2022-08-14 08:58:35.310" "156.70.63.119" "RECEIVED: MAIL FROM:<msprvs1=192254hLQYfb5=bounces-298270@bounce.classmates.com>"
"TCPIP" 128780 "2022-08-14 08:58:35.379" "DNS lookup: 119.63.70.156.zen.spamhaus.org, 0 addresses found: (none), Match: False"
"TCPIP" 128780 "2022-08-14 08:58:35.408" "DNS lookup: 119.63.70.156.bl.spamcop.net, 0 addresses found: (none), Match: False"
"DEBUG" 128780 "2022-08-14 08:58:35.408" "Spam test: SpamTestDNSBlackLists, Score: 0"
"DEBUG" 128780 "2022-08-14 08:58:35.440" "Spam test: SpamTestHeloHost, Score: 0"
"DEBUG" 128780 "2022-08-14 08:58:35.519" "Spam test: SpamTestSPF, Score: 0"
"DEBUG" 128780 "2022-08-14 08:58:35.519" "Total spam score: 0"
"SMTPD" 128780 100 "2022-08-14 08:58:35.535" "156.70.63.119" "SENT: 250 OK"
"SMTPD" 127452 100 "2022-08-14 08:58:35.581" "156.70.63.119" "RECEIVED: RCPT TO:<userl@domain.Net>"
"SMTPD" 127452 100 "2022-08-14 08:58:35.613" "156.70.63.119" "SENT: 250 OK"
"SMTPD" 128780 100 "2022-08-14 08:58:35.660" "156.70.63.119" "RECEIVED: DATA"
"SMTPD" 128780 100 "2022-08-14 08:58:35.660" "156.70.63.119" "SENT: 354 OK, send."
"DEBUG" 102996 "2022-08-14 08:58:35.894" "Adding task AsynchronousTask to work queue Asynchronous task queue"
"DEBUG" 128372 "2022-08-14 08:58:35.894" "Executing task AsynchronousTask in work queue Asynchronous task queue"
"DEBUG" 128372 "2022-08-14 08:58:35.894" "SURBL: Execute"
"DEBUG" 128372 "2022-08-14 08:58:35.894" "SURBL: Found URL: w3.org"
"DEBUG" 128372 "2022-08-14 08:58:35.909" "SURBL: Found URL: classmates.com"
"DEBUG" 128372 "2022-08-14 08:58:35.909" "SURBL: Found URL: pplcnhld.com"
"DEBUG" 128372 "2022-08-14 08:58:35.925" "SURBL: 3 unique addresses found."
"DEBUG" 128372 "2022-08-14 08:58:35.925" "SURBL: Lookup: classmates.com.multi.surbl.org"
"DEBUG" 128372 "2022-08-14 08:58:36.050" "SURBL: Lookup: pplcnhld.com.multi.surbl.org"
"DEBUG" 128372 "2022-08-14 08:58:36.238" "SURBL: Match found"
"DEBUG" 128372 "2022-08-14 08:58:36.253" "Spam test: SpamTestSURBL, Score: 6"
"DEBUG" 128372 "2022-08-14 08:58:36.253" "Total spam score: 6"
"DEBUG" 128372 "2022-08-14 08:58:36.253" "Saving message: {A378BB3E-B6F7-491F-B6CF-36A0D116B509}.eml"
"DEBUG" 128372 "2022-08-14 08:58:36.848" "Requesting SMTPDeliveryManager to start message delivery"
"SMTPD" 128372 100 "2022-08-14 08:58:36.848" "156.70.63.119" "SENT: 250 Queued (0.608 seconds)"
"DEBUG" 92472 "2022-08-14 08:58:37.004" "Adding task DeliveryTask to work queue SMTP delivery queue"
"DEBUG" 103568 "2022-08-14 08:58:37.004" "Executing task DeliveryTask in work queue SMTP delivery queue"
"DEBUG" 103568 "2022-08-14 08:58:37.004" "Delivering message..."
"APPLICATION" 103568 "2022-08-14 08:58:37.004" "SMTPDeliverer - Message 53674: Delivering message from msprvs1=192254hLQYfb5=bounces-298270@bounce.classmates.com to user@domain.Net. File: D:\hMailServer\Data\{A378BB3E-B6F7-491F-B6CF-36A0D116B509}.eml"
"DEBUG" 103568 "2022-08-14 08:58:37.004" "Applying rules"
"DEBUG" 103568 "2022-08-14 08:58:37.004" "Performing local delivery"
"DEBUG" 103568 "2022-08-14 08:58:37.129" "Applying rules"
"DEBUG" 103568 "2022-08-14 08:58:37.145" "Forwarding message"
"DEBUG" 103568 "2022-08-14 08:58:37.145" "Copying mail contents"
"DEBUG" 103568 "2022-08-14 08:58:37.145" "Saving message: {FF01710E-6AB6-4B22-AC7D-EBFFC0184125}.eml"
"DEBUG" 103568 "2022-08-14 08:58:37.566" "Local delivery completed"
"DEBUG" 103568 "2022-08-14 08:58:37.566" "Deleting message"
"DEBUG" 103568 "2022-08-14 08:58:37.675" "Deleting message file."
"APPLICATION" 103568 "2022-08-14 08:58:37.675" "SMTPDeliverer - Message 53674: Message delivery thread completed."
"SMTPD" 102996 100 "2022-08-14 08:58:41.909" "156.70.63.119" "RECEIVED: QUIT"
"SMTPD" 102996 100 "2022-08-14 08:58:41.909" "156.70.63.119" "SENT: 221 goodbye"
"DEBUG" 127452 "2022-08-14 08:58:41.909" "Ending session 100"
If you think you understand quantum mechanics, you don't understand quantum mechanics.

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: Some email bypass spamasssassin

Post by RvdH » 2022-08-15 01:33

Jim, it is simply impossible, period :!:
Probably you f*cked up somehow, but your claim is pure nonsense

if SA is enabled your logs ALWAYS should read:

Code: Select all

"TCPIP"	3200	"2022-08-15 01:27:18.206"	"Connecting to 127.0.0.1:783..."
"DEBUG"	7564	"2022-08-15 01:27:20.269"	"Failed to connect to SpamAssassin. Session 45"
or

Code: Select all

"TCPIP"	3200	"2022-08-15 01:27:21.284"	"Connecting to 127.0.0.1:783..."
"DEBUG"	7564	"2022-08-15 01:27:22.816"	"Sending message to SpamAssassin. Session 46, File: C:\Program Files\hMailServer\Data\{759FD7F0-66EA-4DAA-84C2-C1634E7A9311}.eml"
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

Post Reply