HOW TO: Using Regexp "AND with OR" rules, and testing 'TO' field value

This section contains user-submitted tutorials.
Post Reply
User avatar
jimimaseye
Moderator
Moderator
Posts: 9583
Joined: 2011-09-08 17:48

HOW TO: Using Regexp "AND with OR" rules, and testing 'TO' field value

Post by jimimaseye » 2016-02-02 21:36

Quite often a RULE is needed to check on a list of required or banned values (ie, an 'AND' match) whilst in conjunction and forming a list of other conditions that may need an 'OR' match.

eg,
if TO = GEOFF@mydomain and
(FROM contains Customer1 or FROM contains Customer2)

Hmailserver does not allow this combination. However using REGEXP you can supply the list (OR's) in a single test value to achieve the result

The following is a little hint that will help you use REGEXP to create a test against a list of values.

1, To test a field for a positive match against containing a list of values

ie, if field contains "THIS" or contains "THAT or contains "ANOTHER"

eg
  • if field contains
    *THIS* or
    *THAT* or
    *ANOTHER*
The following test will suffice:
FIELDTOTEST REGEXP (?i:^.*(THIS|THAT|ANOTHER).*$)

2, To test a field to ensure it DOES NOT contain a list of values

ie, does not contain 'THIS' and does not contain 'THAT' and does not contain 'ANOTHER'

eg
  • if field
    is not *THIS* and
    is not *THAT* and
    is not *ANOTHER*
FIELDTOTEST REGEXP (?i:^((?!THIS|THAT|ANOTHER).)*$)


Note1: the list values are separated with a pipe symbol " | ". Extra list values can be added using this separator symbol.
Note2: the letter "i" (3rd character of the expression) specifies that the text match is NOT case dependant. Removal of "i" will make it case dependant.


TIP about TO:
Often people see that entering the field "TO" to be tested (eg, where "TO = 'name@domain.tld'") doesnt actually work against the email values (therefore fail a match). You will find that doing either of the following 3 options may work:

a, instead of using 'predefined field' TO, you should use 'RECIPIENT LIST' or
b, 'predefined field' TO CONTAINS (instead of 'equals') or
c, instead of using 'predefined field' TO, you should use CUSTOMER HEADER FIELD "TO".

Notes:
option (b) works because email address fields (such as FROM and TO) often come in with the form "displayname <name@domain.tld>" and it is this value including name alias and the the angle brackets that is tested. (In other words: "name@domain.com" is NOT the same as "displayname <name@domain.com>"). Specifying CONTAINS will find the actual address within.

option (c) is particularly useful when collecting emails by External Download (rather than direct SMTP delivery) as it reads the physical 'TO:' header of the email rather than the actual account that is collecting the email (which may be different).

In Practice:

So to use the opening example, the Rule could be written as
  • if TO CONTAINS "GEOFF@mydomain" and
    FROM REGEXP (?i:^.*(Customer1|Customer2).*$)
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

glenluo
Senior user
Senior user
Posts: 274
Joined: 2011-07-03 12:10

Re: HOW TO: Using Regexp "AND with OR" rules, and testing 'TO' field value

Post by glenluo » 2022-07-18 03:54

Then how about contain and not contain at the same time at the header From,for example

1: Name part contains DHL and domain not contians @dhl.com
DHL EXPRESS <notice@otherdomain.com>

2: Name part contains SF EXPRESS and domain not contians @sf-express.com
SF EXPRESS <notice@spamdomain.com>

palinka
Senior user
Senior user
Posts: 3647
Joined: 2017-09-12 17:57

Re: HOW TO: Using Regexp "AND with OR" rules, and testing 'TO' field value

Post by palinka » 2022-07-18 06:10

glenluo wrote:
2022-07-18 03:54
Then how about contain and not contain at the same time at the header From,for example

1: Name part contains DHL and domain not contians @dhl.com
DHL EXPRESS <notice@otherdomain.com>

2: Name part contains SF EXPRESS and domain not contians @sf-express.com
SF EXPRESS <notice@spamdomain.com>
(^(DHL)((?!dhl\.com).)*$)|(^(SF\sEXPRESS)((?!sf-express\.com).)*$)

https://regex101.com/r/L7C3BT/2

Rii11
New user
New user
Posts: 1
Joined: 2022-08-03 12:58

Re: HOW TO: Using Regexp "AND with OR" rules, and testing 'TO' field value

Post by Rii11 » 2022-08-03 12:59

awesome!

User avatar
RvdH
Senior user
Senior user
Posts: 2311
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: HOW TO: Using Regexp "AND with OR" rules, and testing 'TO' field value

Post by RvdH » 2022-08-18 09:30

palinka wrote:
2022-07-18 06:10
glenluo wrote:
2022-07-18 03:54
Then how about contain and not contain at the same time at the header From,for example

1: Name part contains DHL and domain not contians @dhl.com
DHL EXPRESS <notice@otherdomain.com>

2: Name part contains SF EXPRESS and domain not contians @sf-express.com
SF EXPRESS <notice@spamdomain.com>
(^(DHL)((?!dhl\.com).)*$)|(^(SF\sEXPRESS)((?!sf-express\.com).)*$)

https://regex101.com/r/L7C3BT/2
I normally use SpamAssassin rules for this, easier to maintain (when you have lots of rules) and you can make more complex checks

Code: Select all

# Domains:
# dhlparcel.nl
# dhlparcel.com
# dhlparcel.de
describe PHISHING_FROM_DHL Trigger on phishing mails
header	 __PHISH_FROM_DHL_1 From:name =~ /^(\bdhl\s?parcel\b).*$/i
header   __PHISH_FROM_DHL_2 From:addr =~ /^(?!.*(\@|\.)dhlparcel\.(com|de|nl)$).*$/i
header   __PHISH_FROM_DHL_3 From:addr =~ /(?!(?:dhlparcel?\.(?:com|de|nl)$))(\bdhl\-?parcel\b)/i
meta     PHISHING_FROM_DHL ( __PHISH_FROM_DHL_1 + __PHISH_FROM_DHL_2 + __PHISH_FROM_DHL_3 >= 2)
score    PHISHING_FROM_DHL 5.0

describe PHISH_FROM_DHL	Trigger on phishing mails
header   PHISH_FROM_DHL	From:addr =~ /\@(dhlparcel\.(com|de|nl))$/i
score    PHISH_FROM_DHL	2.0

describe UNPHISH_FROM_DHL Untrigger on valid mails
header   __UNPHISH_FROM_DHL_A Return-Path:addr =~ /\@(dhlparcel\.(com|de|nl))>?$/i
meta     UNPHISH_FROM_DHL ( __UNPHISH_FROM_DHL_A && ( SPF_PASS || SPF_HELO_PASS && DKIM_VALID ))
score    UNPHISH_FROM_DHL -2.0

PHISH_FROM_DHL and UNPHISH_FROM_DHL speak for themselves i think, eg: PHISH_FROM_DHL if the FromAddress is either: @dhlparcel.com, @dhlparcel.de or @dhlparcel.nl
UNPHISH_FROM_DHL scores negative, if the Return-Path of the message is either: @dhlparcel.com, @dhlparcel.de or @dhlparcel.nl and the message passes SPF and DKIM

PHISHING_FROM_DHL, explained:
header 1 from name contains "dhl parcel"
header 2 from address does not end with "@dhlparcel.com, @dhlparcel.de or @dhlparcel.nl"
header 3 from address does not end with "@dhlparcel.com, @dhlparcel.de or @dhlparcel.nl" but contains the word dhlparcel or dhl-parcel somewhere in the address
meta If at least 2 out the 3 header checks get matched score
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

palinka
Senior user
Senior user
Posts: 3647
Joined: 2017-09-12 17:57

Re: HOW TO: Using Regexp "AND with OR" rules, and testing 'TO' field value

Post by palinka » 2022-08-18 16:51

RvdH wrote:
2022-08-18 09:30
I normally use SpamAssassin rules for this, easier to maintain (when you have lots of rules) and you can make more complex checks
More complex checks is true, but the easiest way (maintenance-wise) to deal with this is using Soren's dynamic black/white list. The php interface makes it ultra easy and truly dynamic.

viewtopic.php?f=20&t=33602

User avatar
RvdH
Senior user
Senior user
Posts: 2311
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: HOW TO: Using Regexp "AND with OR" rules, and testing 'TO' field value

Post by RvdH » 2022-08-18 16:55

palinka wrote:
2022-08-18 16:51
RvdH wrote:
2022-08-18 09:30
I normally use SpamAssassin rules for this, easier to maintain (when you have lots of rules) and you can make more complex checks
More complex checks is true, but the easiest way (maintenance-wise) to deal with this is using Soren's dynamic black/white list. The php interface makes it ultra easy and truly dynamic.

viewtopic.php?f=20&t=33602
Yes and no...spamassassin performance as a whole is lot more efficient compared to hmailserver/vbscript/json library and the alikes
And you can only block, with spamassassin you can block or score
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

palinka
Senior user
Senior user
Posts: 3647
Joined: 2017-09-12 17:57

Re: HOW TO: Using Regexp "AND with OR" rules, and testing 'TO' field value

Post by palinka » 2022-08-18 17:11

RvdH wrote:
2022-08-18 16:55
Yes and no...spamassassin performance as a whole is lot more efficient compared to hmailserver/vbscript/json library and the alikes
Agreed.
And you can only block, with spamassassin you can block or score
You can block, score and remove score (whitelist). Soren created vbs blacklist function which adds N points to the existing spam score, and whitelist function that removes spam score if it exists. They both work great. In fact, I use the whitelist a lot more than the blacklist.

User avatar
RvdH
Senior user
Senior user
Posts: 2311
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: HOW TO: Using Regexp "AND with OR" rules, and testing 'TO' field value

Post by RvdH » 2022-08-18 17:21

palinka wrote:
2022-08-18 17:11
RvdH wrote:
2022-08-18 16:55
Yes and no...spamassassin performance as a whole is lot more efficient compared to hmailserver/vbscript/json library and the alikes
Agreed.
And you can only block, with spamassassin you can block or score
You can block, score and remove score (whitelist). Soren created vbs blacklist function which adds N points to the existing spam score, and whitelist function that removes spam score if it exists. They both work great. In fact, I use the whitelist a lot more than the blacklist.
Ah, didn't know that
...well, i think i keep using what each program is meant for, eg: one as mailserver the other as antispam

As for this topic, just listed SA as option
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
SorenR
Senior user
Senior user
Posts: 5528
Joined: 2006-08-21 15:38
Location: Denmark

Re: HOW TO: Using Regexp "AND with OR" rules, and testing 'TO' field value

Post by SorenR » 2022-08-19 00:25

palinka wrote:
2022-08-18 17:11

You can block, score and remove score (whitelist). Soren created vbs blacklist function which adds N points to the existing spam score, and whitelist function that removes spam score if it exists. They both work great. In fact, I use the whitelist a lot more than the blacklist.
And it uses the database so you can update the lists on a live system without interrupting it ;-)
SørenR.

There are two types of people in this world:
1) Those who can extrapolate from incomplete data

Post Reply