Clean up X-Spam-Report

[palinka]

I'm pretty sure everyone that looks at X-Spam-Report is frustrated by the failure to properly format. I have not seen any fix for this so I decided to do it myself.

Code: Select all

Sub OnDeliverMessage(oMessage)
	
	If oMessage.HeaderValue("X-Spam-Report") <> "" Then
		Dim ReportParts, a, i, iHeaderName, iFormattedNumber
		ReportParts = Split(oMessage.HeaderValue("X-Spam-Report"), "*")
		a = UBound(ReportParts)
		oMessage.HeaderValue("X-SR-00") = "SpamAssassin Report"
		oMessage.Save
		For i = 1 to a - 1
			If i < 10 Then
				iFormattedNumber = "0" & i
			Else
				iFormattedNumber = i
			End If
			iHeaderName = "X-SR-" & iFormattedNumber
			oMessage.HeaderValue(iHeaderName) = " *" & ReportParts(i)
			oMessage.Save
		Next
		oMessage.HeaderValue("X-Spam-Report").Delete
		oMessage.Save
	End If
	
End Sub

I just confirmed this works. I spits out a report that looks like this:

Code: Select all

X-SR-00: SpamAssassin Report
X-SR-01:  *  3.0 SUBJ_YOUR_FAMILY Subject contains "Your Family" 
X-SR-02:  *  2.5 URIBL_DBL_SPAM Contains a spam URL listed in the Spamhaus DBL 
X-SR-03:  *      blocklist 
X-SR-04:  *      [URIs: theonegiftus.com] 
X-SR-05:  * -0.0 SPF_PASS SPF: sender matches SPF record 
X-SR-06:  *  0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3) 
X-SR-07:  *    [209.85.218.65 listed in wl.mailspike.net] 
X-SR-08:  *  0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail 
X-SR-09:  *      provider (nguyenthigai2003[at]gmail.com) 
X-SR-10:  *  0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 
X-SR-11:  *  0.2 FREEMAIL_ENVFROM_END_DIGIT Envelope-from freemail username ends 
X-SR-12:  *       in digit (nguyenthigai2003[at]gmail.com) 
X-SR-13:  *  0.0 HTML_MESSAGE BODY: HTML included in message 
X-SR-14:  *  0.0 T_KAM_HTML_FONT_INVALID BODY: Test for Invalidly Named or 
X-SR-15:  *      Formatted Colors in HTML 
X-SR-16:  * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 
X-SR-17:  * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from 
X-SR-18:  *      author's domain 
X-SR-19:  *  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily 
X-SR-20:  *      valid 
X-SR-21:  * -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from 
X-SR-22:  * envelope-from domain 
X-SR-23:  *  2.4 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50% 
X-SR-24:  *      [cf: 100] 
X-SR-25:  *  1.7 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/) 
X-SR-26:  *  0.0 RCVD_IN_MSPIKE_WL Mailspike good senders 
X-SR-27:  *  0.0 UNPARSEABLE_RELAY Informational: message has unparseable relay 
X-SR-28:  *      lines 
X-SR-29:  *  0.0 FSL_BULK_SIG Bulk signature with no Unsubscribe 

Its not perfect. There are a couple of lines that are missing the spaces required for alignment. I'm not sure yet why that is. But its much more readable now, for sure!

[palinka]

All kinds of goodness from this one. High score, verified clamav plugin with phishing heuristics is working properly and the re-printed spam report headers formatted properly.

HOWEVER , the original X-Spam-Report header was not deleted. Why not?

That's not a big deal, but obviously I did something wrong and I'd like to know why.

Code: Select all

X-Spam-Flag: YES
X-Spam-Level: *************************
X-Spam-Status: Yes, score=25.6 required=5.0 tests=CLAMAV,CLAMAV_SANE, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FORGED_GMAIL_RCVD,
 FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,FREEMAIL_REPLYTO_END_DIGIT,
 GOOGLE_DOC_SUSP,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE, KAM_FAKEFORM,KAM_GOOGLEFORM,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS,
 SUBJ_ALL_CAPS autolearn=disabled version=3.4.2
X-Spam-Virus: Yes (Sanesecurity.Scam4.2366.UNOFFICIAL)
X-Spam-Report: *  0.2 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level *     
 mail domains are different *  0.0 SPF_HELO_NONE SPF: HELO does not publish
 an SPF Record *  0.0 FREEMAIL_FROM Sender email is commonly abused enduser
 mail *      provider (unitednation.un0011[at]gmail.com) *  1.0
 FORGED_GMAIL_RCVD 'From' gmail.com does not match 'Received' *     
 headers *  0.5 SUBJ_ALL_CAPS Subject is all capitals * -0.0 SPF_PASS SPF:
 sender matches SPF record *  0.2 FREEMAIL_REPLYTO_END_DIGIT Reply-To
 freemail username ends in *      digit (unitednation.un0011[at]gmail.com) *
 -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) *      [209.85.219.72 listed
 in wl.mailspike.net] *  0.0 HTML_MESSAGE BODY: HTML included in message *
 -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature *  0.1
 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily *      
 valid *  0.0 CLAMAV Clam AntiVirus detected something... *      [Sanesecurity.Scam4.2366.UNOFFICIAL]
 * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from *     
 author's domain *  0.2 FREEMAIL_FORGED_FROMDOMAIN 2nd level domains in
 From and *      EnvelopeFrom freemail headers are different *   15
 CLAMAV_SANE SPAM found by ClamAV SaneSecurity signatures *  2.0
 KAM_GOOGLEFORM Untitled Google Form *  4.0 KAM_FAKEFORM Fake Form for
 Scams *  2.5 GOOGLE_DOC_SUSP Suspicious use of Google DocsX-Spam-Report-00: SpamAssassin Report
X-Spam-Report-00: SpamAssassin Report
X-Spam-Report-01:  *  0.2 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level 
X-Spam-Report-02:  *      mail domains are different 
X-Spam-Report-03:  *  0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 
X-Spam-Report-04:  *  0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail 
X-Spam-Report-05:  *      provider (unitednation.un0011[at]gmail.com) 
X-Spam-Report-06:  *  1.0 FORGED_GMAIL_RCVD 'From' gmail.com does not match 'Received' 
X-Spam-Report-07:  *      headers 
X-Spam-Report-08:  *  0.5 SUBJ_ALL_CAPS Subject is all capitals 
X-Spam-Report-09:  * -0.0 SPF_PASS SPF: sender matches SPF record 
X-Spam-Report-10:  *  0.2 FREEMAIL_REPLYTO_END_DIGIT Reply-To freemail username ends in 
X-Spam-Report-11:  *      digit (unitednation.un0011[at]gmail.com) 
X-Spam-Report-12:  * -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) 
X-Spam-Report-13:  *      [209.85.219.72 listed in wl.mailspike.net] 
X-Spam-Report-14:  *  0.0 HTML_MESSAGE BODY: HTML included in message 
X-Spam-Report-15:  * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 
X-Spam-Report-16:  *  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily 
X-Spam-Report-17:  *       valid 
X-Spam-Report-18:  *  0.0 CLAMAV Clam AntiVirus detected something... 
X-Spam-Report-19:  *      [Sanesecurity.Scam4.2366.UNOFFICIAL] 
X-Spam-Report-20:  * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from 
X-Spam-Report-21:  *      author's domain 
X-Spam-Report-22:  *  0.2 FREEMAIL_FORGED_FROMDOMAIN 2nd level domains in From and 
X-Spam-Report-23:  *      EnvelopeFrom freemail headers are different 
X-Spam-Report-24:  *   15 CLAMAV_SANE SPAM found by ClamAV SaneSecurity signatures 
X-Spam-Report-25:  *  2.0 KAM_GOOGLEFORM Untitled Google Form 
X-Spam-Report-26:  *  4.0 KAM_FAKEFORM Fake Form for Scams 

[jimimaseye]

Tried removing the 1st message.save and having the single one at the end if?

[Entered by mobile. Excuse my spelling.]

[palinka]

Tried removing the 1st message.save and having the single one at the end if?

[Entered by mobile. Excuse my spelling.]

I'll give it a try. Thanks.

Could be a while before I get a hit. You know, my system is laser focused on rejecting spammers before they can get a message through.

[palinka]

Well, I didn't even think to look for an error log, which I just did.

Code: Select all

"ERROR"	6392	"2020-12-10 11:17:43.770"	"Script Error: Source: Microsoft VBScript runtime error - Error: 800A01A8 - Description: Object required: 'oMessage.HeaderValue(...)' - Line: 1613 Column: 2 - Code: (null)"

Line 1613:

Code: Select all

oMessage.HeaderValue("X-Spam-Report").Delete

I think I figured it out by looking at one of Soren's scripts as an example. I think it should be:

Code: Select all

oMessage.Headers.ItemByName("X-Spam-Report").Delete

Going to give that a try.

[SorenR]

Took me a couple of try's to figure that one out back in the day

[palinka]

Took me a couple of try's to figure that one out back in the day

When in doubt - just ask Soren. That's my motto.

[SorenR]

Sorry, could not help myself...

Code: Select all

    If oMessage.HeaderValue("X-Spam-Report") <> "" Then
        Dim i, XSpamReport : XSpamReport = oMessage.HeaderValue("X-Spam-Report")
        XSpamReport = Split(XSpamReport, "*")
        For i = LBound(XSpamReport)+1 To UBound(XSpamReport) 
            oMessage.HeaderValue("X-Spam-Report-" & Right("0" & i, 2)) = "*" & RTrim(XSpamReport(i))
        Next
        oMessage.Save
    End If

[palinka]

Sorry, could not help myself...

Code: Select all

    If oMessage.HeaderValue("X-Spam-Report") <> "" Then
        Dim i, XSpamReport : XSpamReport = oMessage.HeaderValue("X-Spam-Report")
        XSpamReport = Split(XSpamReport, "*")
        For i = LBound(XSpamReport)+1 To UBound(XSpamReport) 
            oMessage.HeaderValue("X-Spam-Report-" & Right("0" & i, 2)) = "*" & RTrim(XSpamReport(i))
        Next
        oMessage.Save
    End If

Improvements are always welcome.

[palinka]

Huh... it turns out that this is even simpler than I thought.

Code: Select all

If oMessage.HeaderValue("X-Spam-Report") <> "" Then
	oMessage.HeaderValue("X-Spam-Report-Test") = Replace(oMessage.HeaderValue("X-Spam-Report"), "*", vbCrLf & " *") 
	oMessage.Save
End If

This produces this:

Code: Select all

X-Spam-Report-Test: 
 * -1.5 RCVD_IN_SPFBL_WL_W RBL: Sender listed in SPFBL whitelist 
 *      [209.85.167.50 listed in dnswl.spfbl.net] 
 *  1.5 BAYES_60 BODY: Bayes spam probability is 60 to 80% 
 *      [score: 0.6709] 
 *  3.0 SUBJ_YOUR_FAMILY Subject contains "Your Family" 
 *  3.0 RCVD_THRU_GMAILAPI Message originating through GmailAPI 
 * -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) 
 *      [209.85.167.50 listed in wl.mailspike.net] 
 * -1.5 RCVD_IN_NSZONES_WL_W RBL: Sender listed in NSZones whitelist 
 *      [209.85.167.50 listed in wl.nszones.com] 
 *  1.3 PDS_OTHER_BAD_TLD Untrustworthy TLDs 
 *      [URI: 1.thadieusom.xyz (xyz)] 
 * -0.0 SPF_PASS SPF: sender matches SPF record 
 *  0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 
 *  0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail 
 *      provider (levantuan4568[at]gmail.com) 
 *  0.2 FREEMAIL_ENVFROM_END_DIGIT Envelope-from freemail username ends 
 *       in digit (levantuan4568[at]gmail.com) 
 *  0.0 HTML_MESSAGE BODY: HTML included in message 
 *  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily 
 *       valid 
 * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from 
 *      author's domain 
 * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 
 * -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from 
 *      envelope-from domain 
 *  0.0 UNPARSEABLE_RELAY Informational: message has unparseable relay 
 *      lines 
 *  0.0 T_KAM_HTML_FONT_INVALID Test for Invalidly Named or Formatted 
 *      Colors in HTML 
 *  0.0 T_REMOTE_IMAGE Message contains an external image

The formatting was retained when viewed in roundcube > view source as well as outlook > properties > headers.

Trying this out. Haven't received a spam yet, but I think it should work.

Code: Select all

Sub OnDeliverMessage(oMessage)
	REM - Make X-Spam-Report Readable Again
	If oMessage.HeaderValue("X-Spam-Report") <> "" Then
		oMessage.HeaderValue("X-Spam-Report") = Replace(oMessage.HeaderValue("X-Spam-Report"), "*", vbCrLf & " *")
		oMessage.Save
	End If
End Sub

[mattg]

Huh... it turns out that this is even simpler than I thought.

Code: Select all

If oMessage.HeaderValue("X-Spam-Report") <> "" Then
	oMessage.HeaderValue("X-Spam-Report-Test") = Replace(oMessage.HeaderValue("X-Spam-Report"), "*", vbCrLf & " *") 
	oMessage.Save
End If

I also replace "* [" with "[" to stop the square brackets going to a new line, and replace multiple consecutive spaces with a single space using regex

I also use the single lines to write to my inline warning message where the test gives more than 0.5

[palinka]

Huh... it turns out that this is even simpler than I thought.

Code: Select all

If oMessage.HeaderValue("X-Spam-Report") <> "" Then
	oMessage.HeaderValue("X-Spam-Report-Test") = Replace(oMessage.HeaderValue("X-Spam-Report"), "*", vbCrLf & " *") 
	oMessage.Save
End If

I also replace "* [" with "[" to stop the square brackets going to a new line, and replace multiple consecutive spaces with a single space using regex

I also use the single lines to write to my inline warning message where the test gives more than 0.5

Yeah, sure, but can you do that in COLOR?

[mattg]

Huh... it turns out that this is even simpler than I thought.

Code: Select all

If oMessage.HeaderValue("X-Spam-Report") <> "" Then
	oMessage.HeaderValue("X-Spam-Report-Test") = Replace(oMessage.HeaderValue("X-Spam-Report"), "*", vbCrLf & " *") 
	oMessage.Save
End If

I also replace "* [" with "[" to stop the square brackets going to a new line, and replace multiple consecutive spaces with a single space using regex

I also use the single lines to write to my inline warning message where the test gives more than 0.5

Yeah, sure, but can you do that in COLOR?

How about Colour

we use English spelling here in Australia


And yes, I do use colour in my inline spam warinings

[palinka]

We use English spelling too. You know.. en_US...

[SorenR]

We use English spelling too. You know.. en_US...

This issue dates back to the early settlers... Apparently the lexicographer Noah Webster was a dyslexic.

https://www.boredpanda.com/british-amer ... gn=organic

[jimimaseye]

Why is dyslexic so difficult to spell?

[SorenR]

Why is dyslexic so difficult to spell?

Well, if you get it wrong you can always use your rubber to fix it.

[jimimaseye]

If your dyslexic, how would you know its wrong?

[RvdH]

Huh... it turns out that this is even simpler than I thought.

Code: Select all

If oMessage.HeaderValue("X-Spam-Report") <> "" Then
	oMessage.HeaderValue("X-Spam-Report-Test") = Replace(oMessage.HeaderValue("X-Spam-Report"), "*", vbCrLf & " *") 
	oMessage.Save
End If

I also replace "* [" with "[" to stop the square brackets going to a new line, and replace multiple consecutive spaces with a single space using regex

I also use the single lines to write to my inline warning message where the test gives more than 0.5

Do you have a simple onliner for that? I tried palinka's script but the resulting report isn't completely formatted properly, eg:

Code: Select all

X-Spam-Report: 
 *  2.0 BAYES_80 BODY: Bayes spam probability is 80 to 95% 
 *      [score: 0.8536] 
 * -0.0 SPF_PASS SPF: sender matches SPF record 
 * -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 
 *  0.0 RCVD_IN_UCEPROTECT3 RBL: Network listed in 
 *      dnsbl-3.uceprotect.net 
 *      [Your ISP OVH, FR/AS16276 is UCEPROTECT-Level3] [listed because of a
 spamscore of 128.9. See:] [<http://www.uceprotect.net/rblcheck.php?ipr=5.135.24.172>]
 
 *  0.0 HTML_IMAGE_RATIO_08 BODY: HTML has a low ratio of text to image 
 *   area 
 *  0.0 HTML_MESSAGE BODY: HTML included in message 
 *  0.0 HTML_FONT_LOW_CONTRAST BODY: HTML font color similar or 
 *      identical to background 
 *  0.3 LOCAL_SCAM_15 RAW: Keyword used of scams 
 *  2.0 LOCAL_OTHER_BAD_TLD Other untrustworthy TLDs

[katip]

Do you have a simple onliner for that? I tried palinka's script but the resulting report isn't completely formatted properly, eg:

with this:

Code: Select all

If oMessage.HeaderValue("X-Spam-Report") <> "" Then
	oMessage.HeaderValue("X-Spam-Report-Test") = Replace(oMessage.HeaderValue("X-Spam-Report"), "*", vbCrLf & " *") 
	oMessage.Save
End If

my results are not bad (longer lines only):

Code: Select all

X-Spam-Report: 
 *  1.0 RCVD_IN_UCEPROTECT1 RBL: Listed in dnsbl-1.uceprotect.net (open 
 *       relay/proxy/dialup) 
 *      [IP 114.88.193.216 is UCEPROTECT-Level 1 listed.] [See <http://www.uceprotect.net/rblcheck.php?ipr=114.88.193.216>]
 
 *  1.3 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in 
 *      bl.spamcop.net 
 *      [Blocked - see <https://www.spamcop.net/bl.shtml?114.88.193.216>] 
 * 3.3 RCVD_IN_PBL RBL: Received via a relay in Spamhaus PBL 
 *      [114.88.193.216 listed in zen.spamhaus.org] 
 *  2.5 RCVD_IN_MSPIKE_L5 RBL: Very bad reputation (-5) 
 *      [114.88.193.216 listed in bl.mailspike.net] 
 *  1.3 RCVD_IN_RP_RNBL RBL: Relay in RNBL, 
 *      https://senderscore.org/blacklistlookup/ 
 *      [114.88.193.216 listed in bl.score.senderscore.com] 
 *  1.5 RCVD_IN_HOSTKARMA_BL RBL: Sender listed in HOSTKARMA-BLACK 
 *      [114.88.193.216 listed in hostkarma.junkemailfilter.com] 
 *  3.0 RCVD_IN_NERDS_CN RBL: Received from China (People's Republic of 
 *       China) 
 *      [114.88.193.216 listed in zz.countries.nerd.dk] 
 *  0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail 
 *      provider (keanvip01[at]163.com) 
 *  0.0 SPF_HELO_FAIL SPF: HELO does not match SPF record (fail) 
 *      [SPF failed: Please see http://www.openspf.org/Why?s=helo;id=163.com;ip=114.88.193.216;r=xxxx.xxxx.local]
 
 *  0.2 FREEMAIL_REPLYTO_END_DIGIT Reply-To freemail username ends in 
 *     digit (keanvip01[at]163.com) 

[RvdH]

Do you have a simple onliner for that? I tried palinka's script but the resulting report isn't completely formatted properly, eg:

with this:

Code: Select all

If oMessage.HeaderValue("X-Spam-Report") <> "" Then
	oMessage.HeaderValue("X-Spam-Report-Test") = Replace(oMessage.HeaderValue("X-Spam-Report"), "*", vbCrLf & " *") 
	oMessage.Save
End If

my results are not bad (longer lines only):

Code: Select all

X-Spam-Report: 
 *  1.0 RCVD_IN_UCEPROTECT1 RBL: Listed in dnsbl-1.uceprotect.net (open 
 *       relay/proxy/dialup) 
 *      [IP 114.88.193.216 is UCEPROTECT-Level 1 listed.] [See <http://www.uceprotect.net/rblcheck.php?ipr=114.88.193.216>]
 
 *  1.3 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in 
 *      bl.spamcop.net 
 *      [Blocked - see <https://www.spamcop.net/bl.shtml?114.88.193.216>] 
 * 3.3 RCVD_IN_PBL RBL: Received via a relay in Spamhaus PBL 
 *      [114.88.193.216 listed in zen.spamhaus.org] 
 *  2.5 RCVD_IN_MSPIKE_L5 RBL: Very bad reputation (-5) 
 *      [114.88.193.216 listed in bl.mailspike.net] 
 *  1.3 RCVD_IN_RP_RNBL RBL: Relay in RNBL, 
 *      https://senderscore.org/blacklistlookup/ 
 *      [114.88.193.216 listed in bl.score.senderscore.com] 
 *  1.5 RCVD_IN_HOSTKARMA_BL RBL: Sender listed in HOSTKARMA-BLACK 
 *      [114.88.193.216 listed in hostkarma.junkemailfilter.com] 
 *  3.0 RCVD_IN_NERDS_CN RBL: Received from China (People's Republic of 
 *       China) 
 *      [114.88.193.216 listed in zz.countries.nerd.dk] 
 *  0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail 
 *      provider (keanvip01[at]163.com) 
 *  0.0 SPF_HELO_FAIL SPF: HELO does not match SPF record (fail) 
 *      [SPF failed: Please see http://www.openspf.org/Why?s=helo;id=163.com;ip=114.88.193.216;r=xxxx.xxxx.local]
 
 *  0.2 FREEMAIL_REPLYTO_END_DIGIT Reply-To freemail username ends in 
 *     digit (keanvip01[at]163.com) 

You got the same extra linebreaks i have....i don't like that

[palinka]

You got the same extra linebreaks i have....i don't like that

I think it must have something to do with the way hmailserver wraps the lines. Hmailserver seems to remove the original line breaks from spamassassin, which wrecks the formatting. Then inserts new ones so the lines wrap. Maybe it's possible to remove the hmailserver created line breaks, then insert new ones at the asterisks?

[RvdH]

You got the same extra linebreaks i have....i don't like that

I think it must have something to do with the way hmailserver wraps the lines. Hmailserver seems to remove the original line breaks from spamassassin, which wrecks the formatting.

Really? You are a freaking genius

[palinka]

We use English spelling too. You know.. en_US...

This issue dates back to the early settlers... Apparently the lexicographer Noah Webster was a dyslexic.

https://www.boredpanda.com/british-amer ... gn=organic

Is the ground floor/first floor thing a european thing? I've noticed that in France and other places. I had a meeting last week where we were going over construction plans and someone asked if we were looking at the ground floor or the first floor. I answered yes and looked at him funny.

[palinka]

You got the same extra linebreaks i have....i don't like that

I think it must have something to do with the way hmailserver wraps the lines. Hmailserver seems to remove the original line breaks from spamassassin, which wrecks the formatting.

Really? You are a freaking genius

My mom always told me that.

Now run along and find the hmailserver inserted line breaks. Unfortunately I don't have any spam with intact X-Spam-Report headers to look since I've been using the script in the OP longer than my delete-old-trash/spam script (30 days). The original script deletes the X-Spam-Report header.

Where/how does hmailserver insert line breaks?

[RvdH]

Thats the question.... i have been trying to pinpoint the location where the result is added, my best guess is the Spam-Report is simply taken and added from the response data taken from SpamAssassinClient.cpp request
As far as i can tell hmailserver doesn't do anything to it, so most likely it is simply a incompatibility Unix Linebreaks vs Windows Linebreaks

[jimimaseye]

We use English spelling too. You know.. en_US...

This issue dates back to the early settlers... Apparently the lexicographer Noah Webster was a dyslexic.

https://www.boredpanda.com/british-amer ... gn=organic

Is the ground floor/first floor thing a european thing? I've noticed that in France and other places. I had a meeting last week where we were going over construction plans and someone asked if we were looking at the ground floor or the first floor. I answered yes and looked at him funny.

Its an ENGLISH (from England - where the language comes from) thing. Every building has at least a floor as a starting point (usually at the ground level) - no point calling it the 1st floor. You start counting the subsequent floor as you go up. (I mean.... why wouldnt you?)

[SorenR]

Thats the question.... i have been trying to pinpoint the location where the result is added, my best guess is the Spam-Report is simply taken and added from the response data taken from SpamAssassinClient.cpp request
As far as i can tell hmailserver doesn't do anything to it, so most likely it is simply a incompatibility Unix Linebreaks vs Windows Linebreaks

I'm playing with this at the moment..

Code: Select all

    If oMessage.HeaderValue("X-Spam-Report") <> "" Then
        Dim strTemp
        With CreateObject("VBScript.RegExp")
            .Global = True
            .MultiLine = True
            .IgnoreCase = True
            .Pattern = "(\x0A)"
            strTemp = .Replace(oMessage.HeaderValue("X-Spam-Report"), "")
            .Pattern = "(\*)"
            oMessage.HeaderValue("X-Spam-Report-Test") = .Replace(strTemp, "\x0A *")
        End With
    End If

    oMessage.Save

[palinka]

This issue dates back to the early settlers... Apparently the lexicographer Noah Webster was a dyslexic.

https://www.boredpanda.com/british-amer ... gn=organic

Is the ground floor/first floor thing a european thing? I've noticed that in France and other places. I had a meeting last week where we were going over construction plans and someone asked if we were looking at the ground floor or the first floor. I answered yes and looked at him funny.

Its an ENGLISH (from England - where the language comes from) thing. Every building has at least a floor as a starting point (usually at the ground level) - no point calling it the 1st floor. You start counting the subsequent floor as you go up. (I mean.... why wouldnt you?)

In France the prémier étage is also the 2nd floor. Same deal in Hungary. I just assumed all of Europe does it the same.

If you really want to get technical, the cellar should be the first floor. Or in programming terms, the 0th floor. That actually makes a lot more sense as to why the ground floor is the first floor.

[SorenR]

Thats the question.... i have been trying to pinpoint the location where the result is added, my best guess is the Spam-Report is simply taken and added from the response data taken from SpamAssassinClient.cpp request
As far as i can tell hmailserver doesn't do anything to it, so most likely it is simply a incompatibility Unix Linebreaks vs Windows Linebreaks

I'm playing with this at the moment..

Code: Select all

    If oMessage.HeaderValue("X-Spam-Report") <> "" Then
        Dim strTemp
        With CreateObject("VBScript.RegExp")
            .Global = True
            .MultiLine = True
            .IgnoreCase = True
            .Pattern = "(\x0A)"
            strTemp = .Replace(oMessage.HeaderValue("X-Spam-Report"), "")
            .Pattern = "(\*)"
            oMessage.HeaderValue("X-Spam-Report-Test") = .Replace(strTemp, "\x0A *")
        End With
    End If

    oMessage.Save

Mmmm.... Bugger.... Regex101.com usually works for me but something is weird with VBScript RegEx...

This is "view source" from Roundcube...

Code: Select all

X-Spam-Report-Test: \x0A *  3.5 BAYES_99 BODY: Bayes spam probability is 99 to 100% \x0A *      [score:
 1.0000] \x0A *  0.2 BAYES_999 BODY: Bayes spam probability is 99.9 to 100%
 \x0A *      [score: 1.0000] \x0A *  1.0 BOTNET_BADDNS Relay doesn't have
 full circle DNS \x0A *      [botnet_baddns,ip=79.135.58.88,rdns=79-135-58-88.ip.welcomeitalia.it]
 \x0A *  1.0 BOTNET_CLIENTWORDS Hostname contains client-like substrings \x0A
 *      [botnet_clientwords,ip=79.135.58.88,rdns=79-135-58-88.ip.welcomeitalia.it]
 \x0A *  1.0 BOTNET Relay might be a spambot or virusbot \x0A *      [botnet0.9,ip=79.135.58.88,rdns=79-135-58-88.ip.welcomeitalia.it,baddns,client,ipinhostname,clientwords]
 \x0A *  1.0 BOTNET_IPINHOSTNAME Hostname contains its own IP address \x0A *
      [botnet_ipinhosntame,ip=79.135.58.88,rdns=79-135-58-88.ip.welcomeitalia.it]
 \x0A *  0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record \x0A * 
 1.0 BOTNET_CLIENT Relay has a client-like hostname \x0A *      [botnet_client,ip=79.135.58.88,rdns=79-135-58-88.ip.welcomeitalia.it,ipinhostname,clientwords]
 \x0A *  0.1 MIME_HTML_MOSTLY BODY: Multipart message mostly text/html MIME
 \x0A *  0.0 HTML_MESSAGE BODY: HTML included in message \x0A *  0.8
 MPART_ALT_DIFF BODY: HTML and text parts are different \x0A *  3.6
 HELO_DYNAMIC_IPADDR2 Relay HELO'd using suspicious hostname (IP addr \x0A *
       2) \x0A *  1.0 RDNS_DYNAMIC Delivered to internal network by host
 with \x0A *      dynamic-looking rDNS \x0A *  0.0 RCVD_IN_MSPIKE_BL
 Mailspike blacklisted \x0A *  0.0 LOTS_OF_MONEY Huge... sums of money \x0A
 *  2.7 RCVD_IN_MSPIKE_ZBI No description available. \x0A *  0.6
 XFER_LOTSA_MONEY Transfer a lot of money

[RvdH]

\u000A maybe?

[jimimaseye]

Not every building has a cellar. The usual 'ground' floor is usually at ground level (as in an opening from outside) - every building has one of these.

[palinka]

Not every building has a cellar.

Only in Floriduh...

The usual 'ground' floor is usually at ground level (as in an opening from outside) - every building has one of these.

Windows are openings from outside as well.

The next building I build (hopefully a medical building in Queens - still waiting on the lease to go through) will have a 0th floor. The one I'm currently working on would have been perfect for inverted floor numbers. Its a quirky rental in Brooklyn. Roof would be "1R", Top floor 2, next down 3, etc, until you get to the ground floor, which would be floor "8GF" and the cellar being "9C". I doubt the building dept would allow it. The fire dept would throw an absolute fit over that.

Also, for what its worth, I opened a random eml from my data folder in notepad++ and line endings are shown simply as [CR][LF].

[jimimaseye]

Not every building has a cellar.

Only in Floriduh...

When I say 'not every building has a cellar' - I mean MOST buildings do not have cellars. (I guess this is going to lead to a straw poll for those that dont agree).

[palinka]

Not every building has a cellar.

Only in Floriduh...

When I say 'not every building has a cellar' - I mean MOST buildings do not have cellars. (I guess this is going to lead to a straw poll for those that dont agree).

I would disagree, but it seems once again to be a language barrier. From wackypedia:

In British English, the word basement is used for underground floors of, for example, department stores, but the word is used only with houses when the space below the ground floor is habitable, with windows and (usually) its own access. The word cellar applies to the whole underground level or to any large underground room. A subcellar is a cellar that lies further underneath.[2]

Basement and cellar are generally interchangeable in the US. In colder areas all houses have cellars. In the south, not so much. Try and find one in Floriduh and Texas. But up here every house has one. They don't have them in Europe? I thought every house had a wine cellar at least. Maybe not in Germany. Nobody builds beer cellars.

[SorenR]

Only in Floriduh...

When I say 'not every building has a cellar' - I mean MOST buildings do not have cellars. (I guess this is going to lead to a straw poll for those that dont agree).

I would disagree, but it seems once again to be a language barrier. From wackypedia:

In British English, the word basement is used for underground floors of, for example, department stores, but the word is used only with houses when the space below the ground floor is habitable, with windows and (usually) its own access. The word cellar applies to the whole underground level or to any large underground room. A subcellar is a cellar that lies further underneath.[2]

Basement and cellar are generally interchangeable in the US. In colder areas all houses have cellars. In the south, not so much. Try and find one in Floriduh and Texas. But up here every house has one. They don't have them in Europe? I thought every house had a wine cellar at least. Maybe not in Germany. Nobody builds beer cellars.

https://vinorage.dk/en/

Anyone in Denmark with a *proper* wine cellar usually have a 1200 or 1600 style castle built on top of it ... Next to the torture chamber and the dungeons.

[katip]

As far as i can tell hmailserver doesn't do anything to it, so most likely it is simply a incompatibility Unix Linebreaks vs Windows Linebreaks

there is a .cf option in SA : fold_headers which is 1 by default.
https://spamassassin.apache.org/full/3. ... _Conf.html

i turned it off about 2 hours ago. since then no more blank lines (35 spam).
it's strange, appearantly only X-Spam-Test-Scores and X-Spam-Status are affected but not X-Spam-Report, yet blanks disappeared.

//EDIT: forget about it. i got one now

Code: Select all

 *  1.0 RCVD_IN_UCEPROTECT1 RBL: Listed in dnsbl-1.uceprotect.net (open 
 *       relay/proxy/dialup) 
 *      [IP 188.124.18.58 is UCEPROTECT-Level 1 listed.] [See <http://www.uceprotect.net/rblcheck.php?ipr=188.124.18.58>]
 
 *  0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 

[palinka]

When I say 'not every building has a cellar' - I mean MOST buildings do not have cellars. (I guess this is going to lead to a straw poll for those that dont agree).

I would disagree, but it seems once again to be a language barrier. From wackypedia:

In British English, the word basement is used for underground floors of, for example, department stores, but the word is used only with houses when the space below the ground floor is habitable, with windows and (usually) its own access. The word cellar applies to the whole underground level or to any large underground room. A subcellar is a cellar that lies further underneath.[2]

Basement and cellar are generally interchangeable in the US. In colder areas all houses have cellars. In the south, not so much. Try and find one in Floriduh and Texas. But up here every house has one. They don't have them in Europe? I thought every house had a wine cellar at least. Maybe not in Germany. Nobody builds beer cellars.

https://vinorage.dk/en/

Anyone in Denmark with a *proper* wine cellar usually have a 1200 or 1600 style castle built on top of it ... Next to the torture chamber and the dungeons.

Every farmhouse in Hungary has a wine cellar. Fully equipped for making estate wine from the 30 or 40 vines in the back yard. But pàlinka is usually outsourced to distilleries. Grapes for wine; plums, apples and peaches for pàlinka. That's a sweet life. Wine cellar included.

[mattg]

We don't have cellars or basements very often in Australia (Although some houses are completely built underground in very hot regions)
Normally only have basements in tall buildings for car parking.These are normally called 'Lower Ground' if only one, or B1, B2 B3 etc (where B = Basement) for multiple underground levels.

Our ground floor is 'on the ground', and we may have an upper ground floor if the building is on a hill, and has two levels that come out level with 'the ground'. The floor above is the 'first floor' because the when counting you always start at ZERO, even if you don't say it. When you reach the first floor, you've gone up one level. When you reach the 1 mile marker, you've already traveled a mile.

Do you have a simple onliner for that?

Nope

multiple lines sorry

Currently looks like this

Code: Select all

Function SplitSpamAssassinHeaders(oMessage)
    If oMessage.HeaderValue("X-Spam-Report") <> "" Then
        Dim i, XSpamReport : XSpamReport = oMessage.HeaderValue("X-Spam-Report")
        eventlog.write(XSpamReport)
        XSpamReport = ReplaceText(XSpamReport," {2,}"," ")
        XSpamReport = Replace(XSpamReport,"* [","[")
        XSpamReport = Split(XSpamReport, "*")
        For i = LBound(XSpamReport)+1 To UBound(XSpamReport) 
            oMessage.HeaderValue("X-Spam-Report-" & Right("0" & i, 2)) = "*" & RTrim(XSpamReport(i))
        Next
        oMessage.Save
    End If
End Function

Function ReplaceText(str1, patrn, replStr)
	'custom event
	'uses functions: 
	'uses globals: 
	'called from:
	eventlog.write("Replace text - original text:- " & str1)
	Dim regEx
	Set regEx = New RegExp
	With regEx
		.Pattern = patrn
		.IgnoreCase = True
		.Global = True
	End With
	eventlog.write("Replace text - After text:- " & regEx.Replace(str1, replStr))
	ReplaceText = regEx.Replace(str1, replStr)
End Function

In my matt.cf Spamassassin main configuration file, i have
report_wrap_width 150
and
fold_headers 1

[katip]

In my matt.cf Spamassassin main configuration file, i have
report_wrap_width 150
and
fold_headers 1

2 culprits:
one is UCEPROTECT, the other is SPF_HELO_FAIL.
my current workaround:

Code: Select all

Dim xSpamReport
	If oMessage.HeaderValue("X-Spam-Report") <> "" Then
		xSpamReport = oMessage.HeaderValue("X-Spam-Report")
		xSpamReport = Replace(xSpamReport, "] [", "] *      [") ' -> UCEPROTECT
		xSpamReport = Replace(xSpamReport, "/Why?s=", "/Why? *      s=") ' -> SPF HELO
		oMessage.HeaderValue("X-Spam-Report") = Replace(xSpamReport, "*", vbCrLf & " *")
		oMessage.Save
	End If

result:

Code: Select all

 *  2.0 BAYES_80 BODY: Bayes spam probability is 80 to 95% 
 *      [score: 0.8038] 
 *  1.0 RCVD_IN_UCEPROTECT1 RBL: Listed in dnsbl-1.uceprotect.net (open 
 *   relay/proxy/dialup) 
 *      [IP 45.154.4.193 is UCEPROTECT-Level 1 listed.] 
 *      [See <http://www.uceprotect.net/rblcheck.php?ipr=45.154.4.193>] 
 * 0.0 RCVD_IN_MSPIKE_L5 RBL: Very bad reputation (-5) 
 *      [45.154.4.193 listed in bl.mailspike.net] 
 *  1.3 RCVD_IN_RP_RNBL RBL: Relay in RNBL, 
 *      https://senderscore.org/blacklistlookup/ 
 *      [45.154.4.193 listed in bl.score.senderscore.com] 
 *  0.0 SPF_HELO_FAIL SPF: HELO does not match SPF record (fail) 
 *      [SPF failed: Please see http://www.openspf.org/Why? 
 *      s=helo;id=sistemamodaitalia.it;ip=45.154.4.193;r=Server] 
 *  5.0 HS_BODY_68 BODY: Heinlein Support Spamschutz Body-68 
 *   10 CLAMAV Clam AntiVirus detected a virus 
 *      [Sanesecurity.Foxhole.Rar_fs1345.UNOFFICIAL] 
 *  0.0 RCVD_IN_MSPIKE_BL Mailspike blacklisted 
 *  0.8 RDNS_NONE Delivered to internal network by a host with no rDNS 
 *   10 MIME_EXT_XX MIMEHeader high risk attachment 
 *  3.5 MALW_ATTACH Attachment filename suspicious, probable malware 
 *      exploit 
 *  0.4 TO_NO_BRKTS_FROM_MSSP Multiple header formatting problems 
 *  0.0 FROM_MISSPACED From: missing whitespace

fold_headers doesn't apply to X-Spam-Report.
didn't know about report_wrap_width. perhaps this was it..

[RvdH]

Arrggg, i simply give up as it is 'more' readable then before
there is always some rule that breaks the formatting, makes no sense to create workarounds for lines that are dynamic in length

This is with fold_headers 1 and report_wrap_width 150

Code: Select all

X-Spam-Report: 
 * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% 
 *      [score: 0.0009] 
 * -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) 
 *      [40.92.16.38 listed in wl.mailspike.net] 
 * -0.0 SPF_PASS SPF: sender matches SPF record 
 *  1.0 MISSING_HEADERS Missing To: header 
 *  0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (robert.bayer[at]outlook.fr)
 
 * -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 
 *  0.0 HTML_MESSAGE BODY: HTML included in message 
 *  0.3 LOCAL_SCAM_15 RAW: Keyword used of scams 
 * -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no
 trust 
 *      [40.92.16.38 listed in list.dnswl.org] 
 *  1.0 MALFORMED_FREEMAIL Bad headers on message from free email service 
 *  1.0 FREEMAIL_REPLY From and body contain different freemails 
 *  3.0 OUTLOOK_NOT_SIGNED_BUT_EXPECTED Message should have been signed by
 DKIM but wasn't

fold_headers 1 already was present in my local.cf, although i doubt it makes a difference... i have been running with and without it

[jimimaseye]

Normally only have basements in tall buildings for car parking.These are normally called 'Lower Ground' if only one, or B1, B2 B3 etc (where B = Basement) for multiple underground levels.

Our ground floor is 'on the ground', and we may have an upper ground floor if the building is on a hill, and has two levels that come out level with 'the ground'. The floor above is the 'first floor' because the when counting you always start at ZERO, even if you don't say it. When you reach the first floor, you've gone up one level. When you reach the 1 mile marker, you've already traveled a mile.

Exactly. Perfect sense and applies to most. I would guess that 95% (or more) of homes do not have a room 'below ground'. At best there will be a cupboard under the stairs (if they have stairs).

[SorenR]

Arrggg, i simply give up as it is 'more' readable then before
there is always some rule that breaks the formatting, makes no sense to create workarounds for lines that are dynamic in length

This is with fold_headers 1 and report_wrap_width 150

Code: Select all

X-Spam-Report: 
 * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% 
 *      [score: 0.0009] 
 * -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) 
 *      [40.92.16.38 listed in wl.mailspike.net] 
 * -0.0 SPF_PASS SPF: sender matches SPF record 
 *  1.0 MISSING_HEADERS Missing To: header 
 *  0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (robert.bayer[at]outlook.fr)
 
 * -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 
 *  0.0 HTML_MESSAGE BODY: HTML included in message 
 *  0.3 LOCAL_SCAM_15 RAW: Keyword used of scams 
 * -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no
 trust 
 *      [40.92.16.38 listed in list.dnswl.org] 
 *  1.0 MALFORMED_FREEMAIL Bad headers on message from free email service 
 *  1.0 FREEMAIL_REPLY From and body contain different freemails 
 *  3.0 OUTLOOK_NOT_SIGNED_BUT_EXPECTED Message should have been signed by
 DKIM but wasn't

fold_headers 1 already was present in my local.cf, although i doubt it makes a difference... i have been running with and without it

RFC 5322 section 2.1.1.

There are two limits that this specification places on the number of
characters in a line. Each line of characters MUST be no more than
998 characters, and SHOULD be no more than 78 characters, excluding
the CRLF.

I see blank lines following a line of 77+ characters.

[katip]

RFC 5322 section 2.1.1.

There are two limits that this specification places on the number of
characters in a line. Each line of characters MUST be no more than
998 characters, and SHOULD be no more than 78 characters, excluding
the CRLF.

I see blank lines following a line of 77+ characters.

hence report_wrap_width default 70 is safe enough i think.
with 2 manipulations i mentioned above (UCEPROTECT & SPF HELO) and default report_wrap_width, up to now i have decent output (+200 spam)

[RvdH]

I see blank lines following a line of 77+ characters.

SHOULD != MUST

report_wrap_width 998

[RvdH]

This is what i am currently testing, VbLf instead VbCrLf seems to filter out the empty lines

Code: Select all

Dim strSpamReport
If oMessage.HeaderValue("X-Spam-Report") <> "" Then
	strSpamReport = oMessage.HeaderValue("X-Spam-Report")
	strSpamReport = ereg_replace(strSpamReport, "([\x0d]?[\x0a]{1,})", VbLf, False)
	oMessage.HeaderValue("X-Spam-Report-Test") = ereg_replace(strSpamReport, "([\x20]?[\x2a])", VbLf & chr(32) & chr(42), False)
	oMessage.Save
End If
		
' Function replaces pattern with replacement
' varIgnoreCase must be TRUE (match is case insensitive) or FALSE (match is case sensitive)
' from http://www.addedbytes.com/asp/vbscript-regular-expressions/
function ereg_replace(strOriginalString, strPattern, strReplacement, varIgnoreCase)
	dim objRegExp : set objRegExp = new RegExp
	With objRegExp
		.Pattern = strPattern
		.IgnoreCase = varIgnoreCase
		.Global = True
	End With
	ereg_replace = objRegExp.replace(strOriginalString, strReplacement)
	set objRegExp = nothing
end Function	

[SorenR]

I see blank lines following a line of 77+ characters.

SHOULD != MUST

report_wrap_width 998

998 is total length, the other number is "folding"

You should read it... Section "2.2.3. Long Header Fields" seems relevant.

[RvdH]

I see blank lines following a line of 77+ characters.

SHOULD != MUST

report_wrap_width 998

998 is total length, the other number is "folding"

You should read it... Section "2.2.3. Long Header Fields" seems relevant.

I really don't think that is a problem (and havent encountered a problem with it yet) as we apply our own linebreaks now...

Besides that, if it is marked as spam...it is not going to be send anywhere

[RvdH]

This is what i am currently testing, VbLf instead VbCrLf seems to filter out the empty lines

Code: Select all

Dim strSpamReport
If oMessage.HeaderValue("X-Spam-Report") <> "" Then
	strSpamReport = oMessage.HeaderValue("X-Spam-Report")
	strSpamReport = ereg_replace(strSpamReport, "([\x0d]?[\x0a]{1,})", VbLf, False)
	oMessage.HeaderValue("X-Spam-Report-Test") = ereg_replace(strSpamReport, "([\x20]?[\x2a])", VbLf & chr(32) & chr(42), False)
	oMessage.Save
End If
		
' Function replaces pattern with replacement
' varIgnoreCase must be TRUE (match is case insensitive) or FALSE (match is case sensitive)
' from http://www.addedbytes.com/asp/vbscript-regular-expressions/
function ereg_replace(strOriginalString, strPattern, strReplacement, varIgnoreCase)
	dim objRegExp : set objRegExp = new RegExp
	With objRegExp
		.Pattern = strPattern
		.IgnoreCase = varIgnoreCase
		.Global = True
	End With
	ereg_replace = objRegExp.replace(strOriginalString, strReplacement)
	set objRegExp = nothing
end Function	

Code: Select all

X-Spam-Report-Test: 
 *  0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60%
 *      [score: 0.5000]
 *  0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
 *  2.5 RCVD_IN_GBUDB RBL: Listed in truncate.gbudb.net
 *      [217.182.72.226 listed in truncate.gbudb.net]
 *  0.0 HTML_MESSAGE BODY: HTML included in message
 *  0.3 LOCAL_SCAM_15 RAW: Keyword used of scams
 *  0.1 LOCAL_SCAM_6 RAW: Keyword used of scams
 *  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily
 valid
 * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
 * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's
 domain
 *  0.9 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/)
 *  1.9 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50%
 *      [cf: 100]
 * -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list
 manager
 *  0.6 TXREP TXREP: Score normalizing based on sender's reputation

[SorenR]

This is what i am currently testing, VbLf instead VbCrLf seems to filter out the empty lines

Code: Select all

Dim strSpamReport
If oMessage.HeaderValue("X-Spam-Report") <> "" Then
	strSpamReport = oMessage.HeaderValue("X-Spam-Report")
	strSpamReport = ereg_replace(strSpamReport, "([\x0d]?[\x0a]{1,})", VbLf, False)
	oMessage.HeaderValue("X-Spam-Report-Test") = ereg_replace(strSpamReport, "([\x20]?[\x2a])", VbLf & chr(32) & chr(42), False)
	oMessage.Save
End If
		
' Function replaces pattern with replacement
' varIgnoreCase must be TRUE (match is case insensitive) or FALSE (match is case sensitive)
' from http://www.addedbytes.com/asp/vbscript-regular-expressions/
function ereg_replace(strOriginalString, strPattern, strReplacement, varIgnoreCase)
	dim objRegExp : set objRegExp = new RegExp
	With objRegExp
		.Pattern = strPattern
		.IgnoreCase = varIgnoreCase
		.Global = True
	End With
	ereg_replace = objRegExp.replace(strOriginalString, strReplacement)
	set objRegExp = nothing
end Function	

Code: Select all

X-Spam-Report-Test: 
 *  0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60%
 *      [score: 0.5000]
 *  0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
 *  2.5 RCVD_IN_GBUDB RBL: Listed in truncate.gbudb.net
 *      [217.182.72.226 listed in truncate.gbudb.net]
 *  0.0 HTML_MESSAGE BODY: HTML included in message
 *  0.3 LOCAL_SCAM_15 RAW: Keyword used of scams
 *  0.1 LOCAL_SCAM_6 RAW: Keyword used of scams
 *  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily
 valid
 * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
 * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's
 domain
 *  0.9 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/)
 *  1.9 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50%
 *      [cf: 100]
 * -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list
 manager
 *  0.6 TXREP TXREP: Score normalizing based on sender's reputation

MimeCode.cpp ... ??
Line 236.

Code: Select all

	// fold the line if it's longer than 76
	if (nLineLen >= MAX_MIME_LINE_LEN && lastSpacePos != -1)
	{
		int charsSinceSpace = (int) output.size() - lastSpacePos;
		output.Insert(lastSpacePos, "\r\n");

		lastSpacePos = -1;
		nLineLen = charsSinceSpace + 1;
	}

[RvdH]

That 76 chars contradicts the piece of rfc you quoted earlier, not?
...and SHOULD be no more than 78 characters, excluding the CRLF.

Maybe setting report_wrap_width 76 is the answer

[SorenR]

That 76 chars contradicts the piece of rfc you quoted earlier, not?
...and SHOULD be no more than 78 characters, excluding the CRLF.

Maybe setting report_wrap_width 76 is the answer

Hey... I'm not writing the RFC's

[SorenR]

Added text formatting... Everything should line up nicely now.. Disabling encoding should make it ASCII. Not sure if the "\x.."/Chr() is overkill on pure ASCII?

Code: Select all

    If oMessage.HeaderValue("X-Spam-Report") <> "" Then
        Dim strReport, ECFlag : ECFlag = oMessage.EncodeFields
        oMessage.EncodeFields = False
        With CreateObject("VBScript.RegExp")
            .Global = True
            .Pattern = "([\x0D]?[\x0A]{1,})"
            strReport = .Replace(oMessage.HeaderValue("X-Spam-Report"), vbCrLf)
            .Pattern = "([\x20]?[\x2A])"
            strReport = .Replace(strReport, vbLf & Chr(32) & Chr(42))
            .Pattern = "(\*\s{1,3})(?=-)"
            strReport = .Replace(strReport, "* ")  ' 1x space
            .Pattern = "(\*\s{1,3})(?=\d\.\d)"
            strReport = .Replace(strReport, "*  ")  ' 2x space
            .Pattern = "(\*\s{1,})(?=[a-z])|(\*\s{1,})(?=\[)"
            oMessage.HeaderValue("X-Spam-Report") = .Replace(strReport, "*      ")  ' 6x space
        End With
        oMessage.EncodeFields = ECFlag
        oMessage.Save
    End If

[RvdH]

Added text formatting... Everything should line up nicely now.. Disabling encoding should make it ASCII. Not sure if the "\x.."/Chr() is overkill on pure ASCII?

Code: Select all

    If oMessage.HeaderValue("X-Spam-Report") <> "" Then
        Dim strReport, ECFlag : ECFlag = oMessage.EncodeFields
        oMessage.EncodeFields = False
        With CreateObject("VBScript.RegExp")
            .Global = True
            .Pattern = "([\x0D]?[\x0A]{1,})"
            strReport = .Replace(oMessage.HeaderValue("X-Spam-Report"), vbCrLf)
            .Pattern = "([\x20]?[\x2A])"
            strReport = .Replace(strReport, vbLf & Chr(32) & Chr(42))
            .Pattern = "(\*\s{1,3})(?=-)"
            strReport = .Replace(strReport, "* ")  ' 1x space
            .Pattern = "(\*\s{1,3})(?=\d\.\d)"
            strReport = .Replace(strReport, "*  ")  ' 2x space
            .Pattern = "(\*\s{1,})(?=[a-z])|(\*\s{1,})(?=\[)"
            oMessage.HeaderValue("X-Spam-Report") = .Replace(strReport, "*      ")  ' 6x space
        End With
        oMessage.EncodeFields = ECFlag
        oMessage.Save
    End If

overkill, on processing time you mean? I think it is negligible if any...
I am running this script now to test, but still no spam received.. where is the spam when you need it?:)

Are you using report_wrap_width 70 (default) with this or an alternative width?

Few alterations/suggestions, vbLf vs vbCrLf made no difference after all in my test (with 1 )

(1) 1st Pattern
.Pattern = "([\x0D]?[\x0A]){1,}"

2nd Replace
strReport = .Replace(strReport, vbCrLf & Chr(32) & Chr(42))

In pattern 3, 4 & 5 you use /s (is this to match as space only?, in that case maybe better use [\x20] as \s also matches linebreaks [\x0D]?[\x0A] and tabs [\x09])

[RvdH]

@SorenR
That looks pretty clean

UCEPROTECT still seems to be a tricky one to tackle and a SHORTCIRCUIT rule (score without decimal)
Few results:

Code: Select all

X-Spam-Report-Test: 
 *  0.0 USER_IN_BLOCKLIST From: address is in the user's block-list
 *  0.0 SHORTCIRCUIT Not all rules were run, due to a shortcircuited
 *      rule
 * 100 USER_IN_BLACKLIST DEPRECATED: See USER_IN_BLOCKLIST
 
 
 X-Spam-Report-Test: 
 *  0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60%
 *      [score: 0.5037]
 *  2.5 RCVD_IN_UCEPROTECT1 RBL: Listed in dnsbl-1.uceprotect.net
 *      [IP 206.189.224.63 is UCEPROTECT-Level 1 listed.] [See <http://www.uceprotect.net/rblcheck.php?ipr=206.189.224.63>]
 *  0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
 * -0.0 SPF_PASS SPF: sender matches SPF record
 *  0.0 RCVD_IN_UCEPROTECT3 RBL: Network listed in
 *      dnsbl-3.uceprotect.net
 *      [Your ISP DIGITALOCEAN-ASN, US/AS14061 is] [UCEPROTECT-Level3 listed
 because of a spamscore] [of 531.7. See: <http://www.uceprotect.net/rblcheck.php?ipr=206.189.224.63>]
 *  1.4 RCVD_IN_BRBL_LASTEXT RBL: The last external relay in the
 *      Received chain was listed in the DNSBL Barracuda Reputation
 *      Block List (BRBL)
 *      [206.189.224.63 listed in bb.barracudacentral.org]
 *  1.0 RCVD_IN_SPAMRATS_SPAM RBL: RATS-Spam: sender is a spam source
 *      [206.189.224.63 listed in all.spamrats.com]
 *  2.5 RCVD_IN_GBUDB RBL: Listed in truncate.gbudb.net
 *      [206.189.224.63 listed in truncate.gbudb.net]
 *  5.0 SCHAALIT_URI_888 URI: No description available.
 *  0.0 HTML_MESSAGE BODY: HTML included in message
 *  0.3 LOCAL_SCAM_15 RAW: Keyword used of scams
 *  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily
 *      valid
 * -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from
 *      envelope-from domain
 * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
 *      author's domain
 * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
 *  1.9 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50%
 *      [cf: 100]
 *  0.9 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/)
 *  6.0 CUSTOM_MANY_BL Message received in more than 3 RBLs
 *  0.0 FSL_BULK_SIG Bulk signature with no Unsubscribe
 *  0.0 T_KAM_HTML_FONT_INVALID Test for Invalidly Named or Formatted
 *      Colors in HTML
 *  0.0 LOTS_OF_MONEY Huge... sums of money
 
 X-Spam-Report-Test: 
 *  0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60%
 *      [score: 0.5002]
 *  0.1 RCVD_IN_UCEPROTECT2 RBL: Network listed in
 *      dnsbl-2.uceprotect.net
 *      [Net 46.101.128.0/17 is UCEPROTECT-Level2 listed] [because 67
 impacts are seen from] [DIGITALOCEAN-ASN, US/AS14061 there. See:] [<http://www.uceprotect.net/rblcheck.php?ipr=46.101.244.37>]
 *  0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
 *  0.0 RCVD_IN_UCEPROTECT3 RBL: Network listed in
 *      dnsbl-3.uceprotect.net
 *      [Your ISP DIGITALOCEAN-ASN, US/AS14061 is] [UCEPROTECT-Level3 listed
 because of a spamscore] [of 530.7. See: <http://www.uceprotect.net/rblcheck.php?ipr=46.101.244.37>]
 * -0.0 SPF_PASS SPF: sender matches SPF record
 *  2.5 RCVD_IN_UCEPROTECT1 RBL: Listed in dnsbl-1.uceprotect.net
 *      [IP 46.101.244.37 is UCEPROTECT-Level 1 listed.] [See <http://www.uceprotect.net/rblcheck.php?ipr=46.101.244.37>]
 *  1.4 RCVD_IN_BRBL_LASTEXT RBL: The last external relay in the
 *      Received chain was listed in the DNSBL Barracuda Reputation
 *      Block List (BRBL)
 *      [46.101.244.37 listed in bb.barracudacentral.org]
 *  1.0 RCVD_IN_SPAMRATS_SPAM RBL: RATS-Spam: sender is a spam source
 *      [46.101.244.37 listed in all.spamrats.com]
 *  5.0 SCHAALIT_URI_888 URI: No description available.
 *  0.0 HTML_MESSAGE BODY: HTML included in message
 *  0.3 LOCAL_SCAM_15 RAW: Keyword used of scams
 *  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily
 *      valid
 * -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from
 *      envelope-from domain
 * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
 *      author's domain
 * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
 *  2.5 RCVD_IN_GBUDB RBL: Listed in truncate.gbudb.net
 *      [46.101.244.37 listed in truncate.gbudb.net]
 *  6.0 CUSTOM_MANY_BL Message received in more than 3 RBLs
 *  0.0 T_KAM_HTML_FONT_INVALID Test for Invalidly Named or Formatted
 *      Colors in HTML

[SorenR]

@SorenR
That looks pretty clean

UCEPROTECT still seems to be a tricky one to tackle and a SHORTCIRCUIT rule (score without decimal)
Few results:

Code: Select all

X-Spam-Report-Test: 
 *  0.0 USER_IN_BLOCKLIST From: address is in the user's block-list
 *  0.0 SHORTCIRCUIT Not all rules were run, due to a shortcircuited
 *      rule
 * 100 USER_IN_BLACKLIST DEPRECATED: See USER_IN_BLOCKLIST
 
 
 X-Spam-Report-Test: 
 *  0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60%
 *      [score: 0.5037]
 *  2.5 RCVD_IN_UCEPROTECT1 RBL: Listed in dnsbl-1.uceprotect.net
 *      [IP 206.189.224.63 is UCEPROTECT-Level 1 listed.] [See <http://www.uceprotect.net/rblcheck.php?ipr=206.189.224.63>]
 *  0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
 * -0.0 SPF_PASS SPF: sender matches SPF record
 *  0.0 RCVD_IN_UCEPROTECT3 RBL: Network listed in
 *      dnsbl-3.uceprotect.net
 *      [Your ISP DIGITALOCEAN-ASN, US/AS14061 is] [UCEPROTECT-Level3 listed
 because of a spamscore] [of 531.7. See: <http://www.uceprotect.net/rblcheck.php?ipr=206.189.224.63>]
 *  1.4 RCVD_IN_BRBL_LASTEXT RBL: The last external relay in the
 *      Received chain was listed in the DNSBL Barracuda Reputation
 *      Block List (BRBL)
 *      [206.189.224.63 listed in bb.barracudacentral.org]
 *  1.0 RCVD_IN_SPAMRATS_SPAM RBL: RATS-Spam: sender is a spam source
 *      [206.189.224.63 listed in all.spamrats.com]
 *  2.5 RCVD_IN_GBUDB RBL: Listed in truncate.gbudb.net
 *      [206.189.224.63 listed in truncate.gbudb.net]
 *  5.0 SCHAALIT_URI_888 URI: No description available.
 *  0.0 HTML_MESSAGE BODY: HTML included in message
 *  0.3 LOCAL_SCAM_15 RAW: Keyword used of scams
 *  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily
 *      valid
 * -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from
 *      envelope-from domain
 * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
 *      author's domain
 * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
 *  1.9 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50%
 *      [cf: 100]
 *  0.9 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/)
 *  6.0 CUSTOM_MANY_BL Message received in more than 3 RBLs
 *  0.0 FSL_BULK_SIG Bulk signature with no Unsubscribe
 *  0.0 T_KAM_HTML_FONT_INVALID Test for Invalidly Named or Formatted
 *      Colors in HTML
 *  0.0 LOTS_OF_MONEY Huge... sums of money
 
 X-Spam-Report-Test: 
 *  0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60%
 *      [score: 0.5002]
 *  0.1 RCVD_IN_UCEPROTECT2 RBL: Network listed in
 *      dnsbl-2.uceprotect.net
 *      [Net 46.101.128.0/17 is UCEPROTECT-Level2 listed] [because 67
 impacts are seen from] [DIGITALOCEAN-ASN, US/AS14061 there. See:] [<http://www.uceprotect.net/rblcheck.php?ipr=46.101.244.37>]
 *  0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
 *  0.0 RCVD_IN_UCEPROTECT3 RBL: Network listed in
 *      dnsbl-3.uceprotect.net
 *      [Your ISP DIGITALOCEAN-ASN, US/AS14061 is] [UCEPROTECT-Level3 listed
 because of a spamscore] [of 530.7. See: <http://www.uceprotect.net/rblcheck.php?ipr=46.101.244.37>]
 * -0.0 SPF_PASS SPF: sender matches SPF record
 *  2.5 RCVD_IN_UCEPROTECT1 RBL: Listed in dnsbl-1.uceprotect.net
 *      [IP 46.101.244.37 is UCEPROTECT-Level 1 listed.] [See <http://www.uceprotect.net/rblcheck.php?ipr=46.101.244.37>]
 *  1.4 RCVD_IN_BRBL_LASTEXT RBL: The last external relay in the
 *      Received chain was listed in the DNSBL Barracuda Reputation
 *      Block List (BRBL)
 *      [46.101.244.37 listed in bb.barracudacentral.org]
 *  1.0 RCVD_IN_SPAMRATS_SPAM RBL: RATS-Spam: sender is a spam source
 *      [46.101.244.37 listed in all.spamrats.com]
 *  5.0 SCHAALIT_URI_888 URI: No description available.
 *  0.0 HTML_MESSAGE BODY: HTML included in message
 *  0.3 LOCAL_SCAM_15 RAW: Keyword used of scams
 *  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily
 *      valid
 * -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from
 *      envelope-from domain
 * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from
 *      author's domain
 * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature
 *  2.5 RCVD_IN_GBUDB RBL: Listed in truncate.gbudb.net
 *      [46.101.244.37 listed in truncate.gbudb.net]
 *  6.0 CUSTOM_MANY_BL Message received in more than 3 RBLs
 *  0.0 T_KAM_HTML_FONT_INVALID Test for Invalidly Named or Formatted
 *      Colors in HTML

oh.. a score of 100! oops

first two "replacements" should be same as yours. last three are purely formatting.

still using spamass 3.4.0 so missing some settings in local.cf.

[RvdH]

UCEPROTECT is a custom dnsbl i am using,

Code: Select all

ifplugin Mail::SpamAssassin::Plugin::DNSEval

	# UCEPROTECT1 (open relays/proxys/dialups) http://uceprotect.net
	header		RCVD_IN_UCEPROTECT1	eval:check_rbl_txt('uceprotect1-lastexternal', 'dnsbl-1.uceprotect.net.')
	describe	RCVD_IN_UCEPROTECT1	Listed in dnsbl-1.uceprotect.net
	tflags		RCVD_IN_UCEPROTECT1	net
	reuse		RCVD_IN_UCEPROTECT1
	score		RCVD_IN_UCEPROTECT1	2.5 # please adjust the score value
	
	# UCEPROTECT2 (open relays/proxys/dialups networks) http://uceprotect.net
	header		RCVD_IN_UCEPROTECT2	eval:check_rbl_txt('uceprotect2-lastexternal', 'dnsbl-2.uceprotect.net.')
	describe	RCVD_IN_UCEPROTECT2	Network listed in dnsbl-2.uceprotect.net
	tflags		RCVD_IN_UCEPROTECT2	net
	reuse		RCVD_IN_UCEPROTECT2
	score		RCVD_IN_UCEPROTECT2	0.5 # please adjust the score value

	# UCEPROTECT3 (bad networks) http://uceprotect.net
	header		RCVD_IN_UCEPROTECT3	eval:check_rbl_txt('uceprotect3-lastexternal', 'dnsbl-3.uceprotect.net.')
	describe	RCVD_IN_UCEPROTECT3	Network listed in dnsbl-3.uceprotect.net
	tflags		RCVD_IN_UCEPROTECT3	net
	reuse		RCVD_IN_UCEPROTECT3
	score		RCVD_IN_UCEPROTECT3	0.25 # please adjust the score value

endif

For the last few replacements in your script, i notice you using spaces whereas the spam-report uses tabs for indenting lines....wouldn't it be better to use tabs? (VbTab)

[SorenR]

UCEPROTECT is a custom dnsbl i am using,

Code: Select all

ifplugin Mail::SpamAssassin::Plugin::DNSEval

	# UCEPROTECT1 (open relays/proxys/dialups) http://uceprotect.net
	header		RCVD_IN_UCEPROTECT1	eval:check_rbl_txt('uceprotect1-lastexternal', 'dnsbl-1.uceprotect.net.')
	describe	RCVD_IN_UCEPROTECT1	Listed in dnsbl-1.uceprotect.net
	tflags		RCVD_IN_UCEPROTECT1	net
	reuse		RCVD_IN_UCEPROTECT1
	score		RCVD_IN_UCEPROTECT1	2.5 # please adjust the score value
	
	# UCEPROTECT2 (open relays/proxys/dialups networks) http://uceprotect.net
	header		RCVD_IN_UCEPROTECT2	eval:check_rbl_txt('uceprotect2-lastexternal', 'dnsbl-2.uceprotect.net.')
	describe	RCVD_IN_UCEPROTECT2	Network listed in dnsbl-2.uceprotect.net
	tflags		RCVD_IN_UCEPROTECT2	net
	reuse		RCVD_IN_UCEPROTECT2
	score		RCVD_IN_UCEPROTECT2	0.5 # please adjust the score value

	# UCEPROTECT3 (bad networks) http://uceprotect.net
	header		RCVD_IN_UCEPROTECT3	eval:check_rbl_txt('uceprotect3-lastexternal', 'dnsbl-3.uceprotect.net.')
	describe	RCVD_IN_UCEPROTECT3	Network listed in dnsbl-3.uceprotect.net
	tflags		RCVD_IN_UCEPROTECT3	net
	reuse		RCVD_IN_UCEPROTECT3
	score		RCVD_IN_UCEPROTECT3	0.25 # please adjust the score value

endif

For the last few replacements in your script, i notice you using spaces whereas the spam-report uses tabs for indenting lines....wouldn't it be better to use tabs? (VbTab)

I guess it depends on the TAB settings. Your client/editor must use the same tab settings for the result to be perfect. Using spaces makes it look the same regardless of what you use to view it.

[RvdH]

Doubt it, spamassassin own rules use tabs as well, but easy to test so i will give it a go

[RvdH]

Doubt it, spamassassin own rules use tabs as well, but easy to test so i will give it a go

Like i already assumed, formatting of rules is totally irrelevant for the output