SPF not working correctly

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
mikernet
Normal user
Normal user
Posts: 62
Joined: 2018-09-04 22:22

SPF not working correctly

Post by mikernet » 2020-02-21 20:55

Running hMailServer version 5.6.7. Never had an issue like this in many years, until now. Here is an example of the log section that shows the email being greylisted:

Code: Select all

"SMTPD"	2128	79087	"2020-02-17 01:29:03.833"	"40.107.223.80"	"SENT: 220 mail.singulink.com ESMTP"
"SMTPD"	2132	79087	"2020-02-17 01:29:03.865"	"40.107.223.80"	"RECEIVED: EHLO NAM11-DM6-obe.outbound.protection.outlook.com"
"SMTPD"	2132	79087	"2020-02-17 01:29:03.865"	"40.107.223.80"	"SENT: 250-mail.singulink.com[nl]250-SIZE 102400000[nl]250-STARTTLS[nl]250-AUTH LOGIN PLAIN[nl]250 HELP"
"SMTPD"	2128	79087	"2020-02-17 01:29:03.896"	"40.107.223.80"	"RECEIVED: STARTTLS"
"SMTPD"	2128	79087	"2020-02-17 01:29:03.896"	"40.107.223.80"	"SENT: 220 Ready to start TLS"
"SMTPD"	2176	79087	"2020-02-17 01:29:04.161"	"40.107.223.80"	"RECEIVED: EHLO NAM11-DM6-obe.outbound.protection.outlook.com"
"SMTPD"	2176	79087	"2020-02-17 01:29:04.161"	"40.107.223.80"	"SENT: 250-mail.singulink.com[nl]250-SIZE 102400000[nl]250-AUTH LOGIN PLAIN[nl]250 HELP"
"SMTPD"	2132	79087	"2020-02-17 01:29:04.193"	"40.107.223.80"	"RECEIVED: MAIL FROM:<XXXXXXX@olg.ca> SIZE=64138"
"SMTPD"	2132	79087	"2020-02-17 01:29:04.443"	"40.107.223.80"	"SENT: 250 OK"
"SMTPD"	2160	79087	"2020-02-17 01:29:04.474"	"40.107.223.80"	"RECEIVED: RCPT TO:<XXXXXXXX@XXXXXXXXX.com>"
"SMTPD"	2160	79087	"2020-02-17 01:29:04.505"	"40.107.223.80"	"SENT: 451 Please try again later."
"SMTPD"	2128	79087	"2020-02-17 01:29:04.537"	"40.107.223.80"	"RECEIVED: QUIT"
"SMTPD"	2128	79087	"2020-02-17 01:29:04.537"	"40.107.223.80"	"SENT: 221 goodbye"
You can see from this tool that the SPF record for olg.ca is fine:
https://www.dmarcanalyzer.com/spf/checker/

The SPF record includes spf.protection.outlook.com, which contains 40.107.0.0/16, and thus this IP should be passing just fine.

You can see that the included SPF record of _spf.salesforce.com contains the following "exists:" macro:

Code: Select all

v=spf1 exists:%{i}._spf.mta.salesforce.com -all
Could this be causing a problem for hMailServer? The macro is valid, but even if it wasn't I don't think that should prevent IPs from other valid included SPF records to all get thrown out...that would be a bit odd. Does hMailServer throw out the entire SPF record if it considers ANY of the included SPF records to be invalid?

Regardless, it seems to me that there is a bug here which should probably be fixed. I checked with multiple SPF validation tools and all of them say the records are good and the IP should pass.

User avatar
jimimaseye
Moderator
Moderator
Posts: 8644
Joined: 2011-09-08 17:48

Re: SPF not working correctly

Post by jimimaseye » 2020-02-21 21:13

07.223.80" "SENT: 451 Please try again later.
Thats your hms sending code 451. You have greylisting enabled by chance?

run this and post the results: https://www.hmailserver.com/forum/viewt ... 20&t=30914
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

mikernet
Normal user
Normal user
Posts: 62
Joined: 2018-09-04 22:22

Re: SPF not working correctly

Post by mikernet » 2020-02-21 22:37

Yes, I have greylisting enabled with bypass on SPF pass. The issue is that SPF should be passing for that email server IP and domain.

mikernet
Normal user
Normal user
Posts: 62
Joined: 2018-09-04 22:22

Re: SPF not working correctly

Post by mikernet » 2020-02-21 22:55

Let me know if you still want me to run that report, but I don't think this really has anything to do with my settings. hMailServer just fails to SPF pass mail coming from 40.107.223.80 for the olg.ca domain, which should pass no problem.

User avatar
jimimaseye
Moderator
Moderator
Posts: 8644
Joined: 2011-09-08 17:48

Re: SPF not working correctly

Post by jimimaseye » 2020-02-21 23:07

Run the report, yes. (We never assume anything without seeing it). That said it would be useful to see debug level logging for such an event too so we can see what is happening (if you can reproduce it) . Without this it isn't possible to judge.

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

mikernet
Normal user
Normal user
Posts: 62
Joined: 2018-09-04 22:22

Re: SPF not working correctly

Post by mikernet » 2020-02-21 23:12

Can you elaborate on how I get the debug level logging you're looking for?

Here is the diagnostic report output:

Code: Select all

2020-02-21   Hmailserver: 5.6.7-B2425

DOMAINS

   "Domain1.com" - atxxxxxxxxxxxxxx.ca            Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False     Catchall: info@Domain1.com
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:      True

   "Domain2.com" - chxxxxxxxxxxxxxxxxx.com        Enabled: True
      |- "Alias1.com" - chxxxxxxxxxx.com

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False   
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:      True

   "Domain3.com" - clxxxxxxxxxxxx.sixxxxxxx.com   Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False   
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:      True

   "Domain4.com" - haxxxxxxxxx.ca                 Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False   
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:      True

   "Domain5.com" - haxxxxxxxxx.com                Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False   
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:      True

   "Domain6.com" - luxxxxxxxxxxxxx.ca             Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False     Catchall: shawn@Domain6.com
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:      True

   "Domain7.com" - ofxxxxxxxxxxxxxxxxx.com        Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False   
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:      True

   "Domain8.com" - paxxxxxxxxxxxxxxxxx.ca         Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False     Catchall: london@Domain8.com
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:      True

   "Domain9.com" - poxxxxxxxx.ca                  Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False   
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:      True

   "Domain10.com" - sbxxxxxxxxxxxxx.com           Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False   
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:      True

   "Domain11.com" - sixxxxxxx.com                 Enabled: True

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: True    
                   Max message size:   100000   Header:   Relaxed  Plus addressing: False
                   Max size of accounts:    0   Body:     Relaxed
                                                Algorithm:   SHA1  Greylisting:      True
                                                Private key: e:\signing\Domain11.com\dkim_private_key.txt
                                                Selector:    mail

   "Domain12.com" - suxxxxxxxxxxxxxx.ca           Enabled: True
      |- "Alias2.com" - suxxxxxxxxxxxxxx.com

SIGNATURE         LIMITS                       DKIM               ADVANCED
  Enabled: False   Max size:                0   Enabled: False     Catchall: shawn@Domain12.com
                   Max message size:        0                      Plus addressing: False
                   Max size of accounts:    0                    
                                                                   Greylisting:      True
-----------------------------------------------------------------------------------------------

IP RANGES

IP: 127.0.0.1 - 127.0.0.1     Priority: 25     Name: My computer

  Allow connections                         Other
     SMTP:   True                              Antispam :   True
     POP3:   True                              Antivirus:   True
     IMAP:   True                              SSL/TLS:    False

  Allow Deliveries from                     Require Authentication from
     Local To Local       -  True              Local To Local       - False
     Local To External    -  True              Local To External    - False
     External To Local    -  True              External To Local    - False
     External To External -  True              External To External -  True


IP: 192.168.0.0 - 192.168.255.255     Priority: 24     Name: Internal Network

  Allow connections                         Other
     SMTP:   True                              Antispam :   True
     POP3:   True                              Antivirus:   True
     IMAP:   True                              SSL/TLS:    False

  Allow Deliveries from                     Require Authentication from
     Local To Local       -  True              Local To Local       -  True
     Local To External    -  True              Local To External    -  True
     External To Local    -  True              External To Local    - False
     External To External - False           


IP: 0.0.0.0 - 255.255.255.255     Priority: 10     Name: Internet

  Allow connections                         Other
     SMTP:   True                              Antispam :   True
     POP3:   True                              Antivirus:   True
     IMAP:   True                              SSL/TLS:    False

  Allow Deliveries from                     Require Authentication from
     Local To Local       -  True              Local To Local       -  True
     Local To External    -  True              Local To External    -  True
     External To Local    -  True              External To Local    - False
     External To External - False           


   !!  Warning:  DEFAULT DOMAIN is SET  !! - "Domain11.com"
------------------------------------------------------
AUTOBANNED Local Addresses:
    No entries

-----------------------------------------------------------------------------------------------

AUTOBAN
  Autoban Enabled: True       Max invalid logon attempts:     15
                              Minutes Before Reset:         1440  (24.00 hours, 1.00 days)
                              Minutes to Autoban:           1440  (24.00 hours, 1.00 days)

There is a total of 116 auto-ban IP ranges.
-----------------------------------------------------------------------------------------------

INCOMING RELAYS
   No entries
-----------------------------------------------------------------------------------------------

MIRRORING         Disabled
-----------------------------------------------------------------------------------------------

PROTOCOLS

SMTP
GENERAL             DELIVERY                  RFC COMPLIANCE            ADVANCED
No. Connections:  0  No Retries: 40 Mins: 10   Plain Text:         True  Bind: 
                     Host: EXTERNAL.TLD        Empty sender:       True  Batch recipients:   100
Max Msg Size:102400  Relay:-                   Incorrect endings:  True  Use STARTTLS:      True
                     (none entered)            Disc. on invalid:  False  Delivered-To hdr: False
                                                                         Loop limit:           5
                                                                         Recipient hosts:     15
  Routes:
     No routes defined.

POP3
  No. Connections: 0

IMAP
 GENERAL                   PUBLIC FOLDERS                    ADVANCED
  No. Connections:   0      Public folder name: #Public       IMAP sort:  True
                                                              IMAP Quota: True
                                                              IMAP Idle:  True
                                                              IMAP ACL:   True
                                                              Delim: "."
-----------------------------------------------------------------------------------------------

ANTISPAM

GENERAL                              SPAM TESTS              Score   SPAMASSASSIN
  Spam Mark:                  3       Use SPF:            True - 1    Use Spamassassin:    True
  Add X-HmailServer-Spam:     True    Check HELO host:    True - 1    Hostname:       127.0.0.1
  Add X-HmailServer-Reason:   True    Check MX records:   True - 1    Port:                 783
  Add X-HmailServer-Subject:  True    Verify DKIM:       False        Use SA score:        True
              Subject Text: "[SPAM]"
  Spam delete threshold: 10         Maximum message size: 2048

DNSBL ENTRIES:
            b.barracudacentral.org      Score: 2     Result: 127.0.0.*
                  psbl.surriel.com      Score: 2     Result: 127.0.0.*
                ubl.unsubscore.com      Score: 4     Result: 127.0.0.*
                   cbl.abuseat.org      Score: 2     Result: 127.0.0.*

SURBL ENTRIES:
   No entries

GREYLISTING:
  Greylisting:   True       Defer mins: 5       Days Unused: 3      Days Used: 180
                            Bypass SPF: True     Bypass A/MX: False

Greylist WHITELIST ENTRIES:
   IP Address: 104.47.*
   IP Address: 209.235.143.14
   IP Address: 40.107.*

Greylist DOMAINS enabled:
           Domain1.com
           Domain2.com
                 |--   Alias1.com
           Domain3.com
           Domain4.com
           Domain5.com
           Domain6.com
           Domain7.com
           Domain8.com
           Domain9.com
           Domain10.com
           Domain11.com
           Domain12.com
                 |--   Alias2.com

WHITELISTING
              0.0.0.0            to    255.255.255.255              *[@t]on[dot]aibn[dot]com
              0.0.0.0            to    255.255.255.255              *[@t]canadacomputers[dot]com
              0.0.0.0            to    255.255.255.255              *[@t]yorkproperty[dot]ca
              0.0.0.0            to    255.255.255.255              *[@t]yorkdev[dot]ca
              0.0.0.0            to    255.255.255.255              *[@t]test[dot]com
              0.0.0.0            to    255.255.255.255              *[@t]c1networx[dot]com
              0.0.0.0            to    255.255.255.255              *[@t]eteki[dot]com
              0.0.0.0            to    255.255.255.255              *[@t]sendgrid[dot]net
              0.0.0.0            to    255.255.255.255              *[@t]dynastygamers[dot]com
-----------------------------------------------------------------------------------------------

ANTIVIRUS

GENERAL:
  When found - Delete Attachments.

  Max Message Size: 0
     CLAM AV:   False
     CLAMWIN:   True       Executable: C:\Program Files (x86)\Sourcefire Inc\ClamAV\clamdscan.exe    Path: 
     CUSTOMAV:  False

  Block Attachments: True
               *.bat             Batch processing file
               *.cmd             Command file for Windows NT
               *.com             Command
               *.cpl             Windows Control Panel extension
               *.csh             CSH script
               *.exe             Executable file
               *.inf             Setup file
               *.lnk             Windows link file
               *.msi             Windows Installer file
               *.msp             Windows Installer patch
               *.reg             Registration key
               *.scf             Windows Explorer command
               *.scr             Windows Screen saver
-----------------------------------------------------------------------------------------------

SSL CERTIFICATES
   mail.Domain11.com
       Certificate: E:\Certificates\Domain11.com-chain.pem
       Private key: E:\Certificates\Domain11.com-key.pem
-----------------------------------------------------------------------------------------------

SSL/TLS
             SSL 3.0 :   True
             TLS 1.0 :   True
             TLS 1.1 :   True
             TLS 1.2 :   True                Verify Remote SSL/TLS Certs:   True
SslCipherList  :

ECDHE-RSA-AES128-GCM-SHA256     - ECDHE-ECDSA-AES128-GCM-SHA256   - ECDHE-RSA-AES256-GCM-SHA384     
ECDHE-ECDSA-AES256-GCM-SHA384   - DHE-RSA-AES128-GCM-SHA256       - DHE-DSS-AES128-GCM-SHA256       
kEDH+AESGCM                     - ECDHE-RSA-AES128-SHA256         - ECDHE-ECDSA-AES128-SHA256       
ECDHE-RSA-AES128-SHA            - ECDHE-ECDSA-AES128-SHA          - ECDHE-RSA-AES256-SHA384         
ECDHE-ECDSA-AES256-SHA384       - ECDHE-RSA-AES256-SHA            - ECDHE-ECDSA-AES256-SHA          
DHE-RSA-AES128-SHA256           - DHE-RSA-AES128-SHA              - DHE-DSS-AES128-SHA256           
DHE-RSA-AES256-SHA256           - DHE-DSS-AES256-SHA              - DHE-RSA-AES256-SHA              
AES128-GCM-SHA256               - AES256-GCM-SHA384               - ECDHE-RSA-RC4-SHA               
ECDHE-ECDSA-RC4-SHA             - AES128                          - AES256                          
RC4-SHA                         - HIGH                            - !aNULL                          
!eNULL                          - !EXPORT                         - !DES                            
!3DES                           - !MD5                            - !PSK;                           
-----------------------------------------------------------------------------------------------

TCPIP PORTS                                         Connection Sec
               0.0.0.0         / 25    / SMTP   -   StartTLS Optional   Cert: mail.Domain11.com
               0.0.0.0         / 110   / POP3   -   StartTLS Optional   Cert: mail.Domain11.com
               0.0.0.0         / 143   / IMAP   -   StartTLS Optional   Cert: mail.Domain11.com
               0.0.0.0         / 465   / SMTP   -   SSL/TLS             Cert: mail.Domain11.com
               0.0.0.0         / 587   / SMTP   -   StartTLS Optional   Cert: mail.Domain11.com
               0.0.0.0         / 993   / IMAP   -   SSL/TLS             Cert: mail.Domain11.com
               0.0.0.0         / 995   / POP3   -   SSL/TLS             Cert: mail.Domain11.com
               0.0.0.0         / 1025  / SMTP   -   StartTLS Optional   Cert: mail.Domain11.com
-----------------------------------------------------------------------------------------------

LOGGING      Logging Enabled: True

  Paths:-
    Current:  E:\Logs\hmailserver_2020-02-21.log
    Error:    E:\Logs\ERROR_hmailserver_2020-02-21.log
    Event:    E:\Logs\hmailserver_events.log - Last Event: 2020/02/21
    Awstats:  E:\Logs\hmailserver_awstats.log
                        APPLICATION -    True
                        SMTP        -    True
                        POP3        -      .
                        IMAP        -      .
                        TCPIP       -      .
                        DEBUG       -      .
                        AWSTATS     -      .
-----------------------------------------------------------------------------------------------

SYSTEM TESTS

Database type: MSSQL

IPv6 support is available in operating system.

ERROR: Backup directory has not been specified.

ERROR: Full paths are stored in the database.

-----------------------------------------------------------------------------------------------

HMAILSERVER.INI

[Directories]
Program folder:  C:\Program Files (x86)\hMailServer\
Database folder: 
Data folder:     E:\Data
Log folder:      E:\Logs
Temp folder:     E:\Temp
Event folder:    E:\Events

[Database]
Type=              MSSQL
Username=          sa
PasswordEncryption=1
Port=              0
Server=            mssql1
Internal=          0
-----------------------------------------------------------------------------------------------

Generated by HMSSettingsDiagnostics v1.98, Hmailserver Forum.

User avatar
jimimaseye
Moderator
Moderator
Posts: 8644
Joined: 2011-09-08 17:48

Re: SPF not working correctly

Post by jimimaseye » 2020-02-22 00:33

Can you elaborate on how I get the debug level logging you're looking for?
Yep.

Code: Select all

---------------------------------------------------------------------------------------------

LOGGING      Logging Enabled: True

  Paths:-
    Current:  E:\Logs\hmailserver_2020-02-21.log
    Error:    E:\Logs\ERROR_hmailserver_2020-02-21.log
    Event:    E:\Logs\hmailserver_events.log - Last Event: 2020/02/21
    Awstats:  E:\Logs\hmailserver_awstats.log
                        APPLICATION -    True
                        SMTP        -    True
                        POP3        -      .
                        IMAP        -      .
                        TCPIP       -      .
                        DEBUG       -      .   <<<<-------  HERE
                        AWSTATS     -      .
--------------------------------------------------------------------------------------------]
As above - enable it.

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
mattg
Moderator
Moderator
Posts: 20897
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: SPF not working correctly

Post by mattg » 2020-02-22 02:02

mikernet wrote:
2020-02-21 20:55
You can see that the included SPF record of _spf.salesforce.com contains the following "exists:" macro:

Code: Select all

v=spf1 exists:%{i}._spf.mta.salesforce.com -all
I think this is the problem

https://www.spf-record.com/syntax
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

mikernet
Normal user
Normal user
Posts: 62
Joined: 2018-09-04 22:22

Re: SPF not working correctly

Post by mikernet » 2020-02-22 02:10

That macro passes SPF validation on every test site I've tried, including macro resolution on the sample site I provided, so it must be following readily available standards.

Even if it wasnt though, I really dont think the entire SPF config shoul# be thrown away because of a bad included SPF record. A bad third party config shouldn't render someone's entire mail config broken when using other senders.

User avatar
mattg
Moderator
Moderator
Posts: 20897
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: SPF not working correctly

Post by mattg » 2020-02-22 03:22

The issue here is that hmailserver see that SPF as a fail

Then greylisting and your other antispam settings kick in

In my view greylisting is next to useless these days, especially if you select to bypass greylisting on an SPF pass

Any correctly formatted spf record that ends in +all will ALWAYS pass SPF checks
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

mikernet
Normal user
Normal user
Posts: 62
Joined: 2018-09-04 22:22

Re: SPF not working correctly

Post by mikernet » 2020-02-22 06:21

Greylisting with SPF bypass still manages to stop a significant portion of spam for us before it has a chance to go anywhere else, but given hMailServer's current non-conformant SPF implementation and lack of ability to handle this situation, it seems that we need to disable it now.

If someone has a +all record then yes, I expect any delivery server to pass, that's not a problem. Never delivering mail because it gets trapped in a greylisting cycle due to incorrectly handled SPF results by hMailServer is the bigger issue for me.

The link you sent before is not the complete SPF specification. The relevant section on macros in the SPF formal RFC spec can be found here: https://tools.ietf.org/html/rfc7208#page-28

I urge you to consider the following updates:
1) Implement the full SPF specification, both for "pass on SPF bypass" greylisting setting as well as the "anti-spam > spam tests > use SPF" setting, which I'm sure does not currently work correctly either
2) Do not throw away an entire SPF record if an "include:" record is invalid. Once again, an invalid third party record should not render all your own records or other included records to be invalid as well.

mikernet
Normal user
Normal user
Posts: 62
Joined: 2018-09-04 22:22

Re: SPF not working correctly

Post by mikernet » 2020-02-22 06:24

This made me wonder - if a third-party included record becomes "unavailable" and cannot be retrieved, does the whole SPF record get thrown out as well?

User avatar
mattg
Moderator
Moderator
Posts: 20897
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: SPF not working correctly

Post by mattg » 2020-02-22 09:23

OK, researching I can see that the RFCs have changed in 2014, and there is a new version of the spf testing library used in hMailserver

I've created a GitHub issue, and will do a pull request.
Don't know how long or if that will be implemented

https://github.com/hmailserver/hmailserver/issues/314
mikernet wrote:
2020-02-22 06:24
This made me wonder - if a third-party included record becomes "unavailable" and cannot be retrieved, does the whole SPF record get thrown out as well?
The only FAIL is where an SPF record is available, and it doesn't pass the tests.

Honestly, this is first I have heard of SPF Macros, and not many of us use SPF to bypass greylisting, because among other things we find that the spammers use valid SPF records, or find one of the millions of domains that have a +spf record.

I used to use greylisting a lot, but since the likes of gmail and Office365 have grown, and use multiple servers, greylisting is fairly useless.

This may make it. Upgrading an existing library to a newer version shouldn't be too hard.
We will see.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
jimimaseye
Moderator
Moderator
Posts: 8644
Joined: 2011-09-08 17:48

Re: SPF not working correctly

Post by jimimaseye » 2020-02-22 10:19

Whether or not the macro inclusion is the cause of the failure remains to be seen (hopefully the debug logging will give more info). However, according to that rfc in the event of a dns lookup failure (misconfiguration or recursion depth error etc) and a 'permerror' status returned, it is up to the implementation of the receiving server to decide on what choice it makes. Options are to accept and pass and therefore discard the possibility of it being spam or err on the side of caution and treat as suspect (reject) and apply existing antispam procedures. Arguments could be made for either.

Within hms spf checking is only used as PART of anti-spam measures and such failure alone will not trigger a rejection outright (it requires other checks and score determinations). (It would be interesting to see how spamassassin spf checking behaves on the same message).

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
SorenR
Senior user
Senior user
Posts: 3623
Joined: 2006-08-21 15:38
Location: Denmark

Re: SPF not working correctly

Post by SorenR » 2020-02-22 14:14

mattg wrote:
2020-02-22 09:23
Upgrading an existing library to a newer version shouldn't be too hard.
We will see.
I compared the original 1.10 library and the library from hMailServer 5.6.8 ...
- First of all the #define's and the #include's in RMSPF.C has been changed by Martin.
- There are massive codechanges from about line 2000 in RMSPF.C, Martin aparently changed a lot.

I just had to try replace the old version with the new version and compiling 5.6.8 failed big time. This is NOT going to be an easy task!

Version 1.10 -> http://www.pamho.net/source/RMSPF110S.ZIP
Version 1.12 -> http://www.pamho.net/source/RMSPF112S.ZIP
SørenR.

“Those who don't know history are doomed to repeat it.”
― Edmund Burke

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-22 18:11

I noticed mattg placed a pull request in github... I dunno if that is tested code or if he simply replaced the libs without building/testing
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
SorenR
Senior user
Senior user
Posts: 3623
Joined: 2006-08-21 15:38
Location: Denmark

Re: SPF not working correctly

Post by SorenR » 2020-02-22 18:17

RvdH wrote:
2020-02-22 18:11
I noticed mattg placed a pull request in github... I dunno if that is tested code or if he simply replaced the libs without building/testing
It's not.....

Attached files compile on my 5.6.8 and seems not to crash when run... 8)

PS... What does your .24 version cover ??? SURBL ??? :wink:
Attachments
SPF.rar
(24.09 KiB) Downloaded 409 times
SørenR.

“Those who don't know history are doomed to repeat it.”
― Edmund Burke

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-22 18:33

SorenR wrote:
2020-02-22 18:17
Attached files compile on my 5.6.8 and seems not to crash when run... 8)
The question should be if it validates SPF records properly :)
SorenR wrote:
2020-02-22 18:17
PS... What does your .24 version cover ??? SURBL ??? :wink:
Yup, its the altered SURBL regex
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
SorenR
Senior user
Senior user
Posts: 3623
Joined: 2006-08-21 15:38
Location: Denmark

Re: SPF not working correctly

Post by SorenR » 2020-02-22 18:55

RvdH wrote:
2020-02-22 18:33
SorenR wrote:
2020-02-22 18:17
Attached files compile on my 5.6.8 and seems not to crash when run... 8)
The question should be if it validates SPF records properly :)
SorenR wrote:
2020-02-22 18:17
PS... What does your .24 version cover ??? SURBL ??? :wink:
Yup, its the altered SURBL regex
Well... I do not have sufficient traffic to test that. Sometimes when I test my script changes I may sit for two hours before the right event occur. :roll:

I'll move the "fix" to my production server sometime later today and then we'll see. I have to reinstate all my SPF settings :roll:

Somehow it makes sense as I have seen different answers from hMailServer and SpamAssassin for quite some time and that is also why all SPF checking on my server is done by SpamAssassin :mrgreen:
SørenR.

“Those who don't know history are doomed to repeat it.”
― Edmund Burke

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-22 19:19

Well, that is most likely because spamassassin does a better job checking spf.....hmailserver doesn't do anything other then checking the sending ip against DNS spf record
There are quite a few macro's that could be used, for example %{h} that validates the message helo/ehlo, %{s} with senders email address and %{d} for senders domain

https://www.zytrax.com/books/dns/ch9/spf.html
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-22 20:32

For now we only have a claim spf isn't working, it might as well be a DNS problem... or did someone actually test hmailserver's behavior when it encounteres a spf record using the exists:%{1} macro implementation? Haven't seen a debug log (yet)
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
jimimaseye
Moderator
Moderator
Posts: 8644
Joined: 2011-09-08 17:48

Re: SPF not working correctly

Post by jimimaseye » 2020-02-22 20:52

RvdH wrote:
2020-02-22 20:32
For now we only have a claim spf isn't working, it as well be a DNS problem... or did someone actually test hmailserver's behavior when it encounteres a spf record using the exists:%{1} macro implementation? Haven't seen a debug log (yet)
I eluded to this earlier when op first made the claim but he hasn't provided the evidence yet :
jimimaseye wrote:
2020-02-21 23:07
Run the report, yes. (We never assume anything without seeing it). That said it would be useful to see debug level logging for such an event too so we can see what is happening (if you can reproduce it) . Without this it isn't possible to judge.

[Entered by mobile. Excuse my spelling.]
[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
SorenR
Senior user
Senior user
Posts: 3623
Joined: 2006-08-21 15:38
Location: Denmark

Re: SPF not working correctly

Post by SorenR » 2020-02-22 21:31

Well....

http://www.pamho.net/source/RMSPF112.TXT

I did spend some time comparing the original 1.10 and the 1.10 from hMailServer - Martin made a lot of changes to the code.

So far everything just blaze past my GreyListing with SPF so clearly something is off ... Until it is confirmed and verified that hMailServer do have a SPF problem I will revert my changes.
SørenR.

“Those who don't know history are doomed to repeat it.”
― Edmund Burke

User avatar
mattg
Moderator
Moderator
Posts: 20897
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: SPF not working correctly

Post by mattg » 2020-02-22 22:53

RvdH wrote:
2020-02-22 18:11
I noticed mattg placed a pull request in github... I dunno if that is tested code or if he simply replaced the libs without building/testing
Simple replace - no testing
RvdH wrote:
2020-02-22 20:32
For now we only have a claim spf isn't working, it might as well be a DNS problem... or did someone actually test hmailserver's behavior when it encounteres a spf record using the exists:%{1} macro implementation? Haven't seen a debug log (yet)
No checks done
I did however note that the existing hMailserver code doesn't check macros, and that the RFCs changed in 2014.

I didn't do a line by line compare, but I did check first few lines, last few lines and some random ones in between. Doing a full text compare as SorenR did is much more sensible.

Should I will remove the pull request?
RvdH wrote:
2020-02-22 19:19
...that is most likely because spamassassin does a better job checking spf...
I think we've all known this a long time

I was hoping that this was an easy fix.
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-22 22:54

As far i understand the exists:%{i} macro doesn't do much more as checking if the value in %{i} resolves to any A-record, the returned IP address does not need to match that of the sender. So this apparently is the worst option that you can define in your spf record as it security wise doesn't verify anything
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
mattg
Moderator
Moderator
Posts: 20897
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: SPF not working correctly

Post by mattg » 2020-02-22 23:10

probably second worst behind +all which allows ANY IP address
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-22 23:25

mattg wrote:
2020-02-22 23:10
probably second worst behind +all which allows ANY IP address
✅

As long as hmailserver ignores exists:%{i} and/or doesn't give false positives once it encounters such SPF record we should be good in my opinion, but I like many others here do not rely on hmailserver anti-spam masseurs, and have spamassassin in place to do the spam checking
Last edited by RvdH on 2020-02-22 23:35, edited 1 time in total.
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
mattg
Moderator
Moderator
Posts: 20897
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: SPF not working correctly

Post by mattg » 2020-02-22 23:29

Me too

I think the point of this thread though specifically is the 'don't greylist on spf pass' option

Greylisting in my view, has been fairly ineffective since gMail and Office365 and the likes have really ramped up their domain hosting. (with their multiple servers)

not greylisting for spf pass is one of the few solutions to that, but that does require spf checking to work in hmailserver
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-22 23:40

mattg wrote:
2020-02-22 23:29
Me too

I think the point of this thread though specifically is the 'don't greylist on spf pass' option
True, but we only have seen a claim that says hmailserver fails spf verification if a SPF record containing exists:%{i} exists :)
mattg wrote:
2020-02-22 23:29
Greylisting in my view, has been fairly ineffective since gMail and Office365 and the likes have really ramped up their domain hosting. (with their multiple servers)
Therefor i have my spfverify.exe, that enables dynamic greywhitelisting for the big ones (eg: gmail.com, outlook.com)
Basically spfverify.exe checks sending ip against HELO/EHLO domain SPF record, and because most of those use TLS, once they send the 2nd HELO/EHLO command they are whitelisted from greylisting when verified
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
SorenR
Senior user
Senior user
Posts: 3623
Joined: 2006-08-21 15:38
Location: Denmark

Re: SPF not working correctly

Post by SorenR » 2020-02-23 18:13

mattg wrote:
2020-02-22 09:23
OK, researching I can see that the RFCs have changed in 2014, and there is a new version of the spf testing library used in hMailserver

I've created a GitHub issue, and will do a pull request.
Don't know how long or if that will be implemented

https://github.com/hmailserver/hmailserver/issues/314
mikernet wrote:
2020-02-22 06:24
This made me wonder - if a third-party included record becomes "unavailable" and cannot be retrieved, does the whole SPF record get thrown out as well?
The only FAIL is where an SPF record is available, and it doesn't pass the tests.

Honestly, this is first I have heard of SPF Macros, and not many of us use SPF to bypass greylisting, because among other things we find that the spammers use valid SPF records, or find one of the millions of domains that have a +spf record.

I used to use greylisting a lot, but since the likes of gmail and Office365 have grown, and use multiple servers, greylisting is fairly useless.

This may make it. Upgrading an existing library to a newer version shouldn't be too hard.
We will see.
https://datatracker.ietf.org/doc/rfc7208/ This explains why a new RFC was issued in 2014 when in fact the original from 2006 is still very much valid.

RFC 7208 was made due to updates in ... or as a result of ... RFC 8616, RFC 8553 and RFC 7372. From what I can read then RFC 4408 dated April 2006 should still be valid. The update to RFC 4408 by RFC 6652 in december 2018 appear not relevant to the way hMailServer lookup the SPF record.

Testing RMSPF 1.12 with hMailServer was done again today but as it currently functions as a black box all I can say is "it does NOT give the expected result".

RMSPF 1.12 is dated around the same time as the release of RFC 4408 so perhaps we can hope the author was aware of it.
SørenR.

“Those who don't know history are doomed to repeat it.”
― Edmund Burke

mikernet
Normal user
Normal user
Posts: 62
Joined: 2018-09-04 22:22

Re: SPF not working correctly

Post by mikernet » 2020-02-23 20:02

RvdH wrote:
2020-02-22 22:54
As far i understand the exists:%{i} macro doesn't do much more as checking if the value in %{i} resolves to any A-record, the returned IP address does not need to match that of the sender. So this apparently is the worst option that you can define in your spf record as it security wise doesn't verify anything
That's not the macro though. As stated in the op, the macros is exists:%{i}._spf.mta.salesforce.com. So if the email server is 1.2.3.4, it will verify that 1.2.3.4._spf.mta.salesforce.com exists, and that DNS server is setup to only respond with valid records for authorized email servers, which is a perfectly valid and reasonable way to handle SPF.

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-23 20:21

mikernet wrote:
2020-02-23 20:02
RvdH wrote:
2020-02-22 22:54
As far i understand the exists:%{i} macro doesn't do much more as checking if the value in %{i} resolves to any A-record, the returned IP address does not need to match that of the sender. So this apparently is the worst option that you can define in your spf record as it security wise doesn't verify anything
That's not the macro though. As stated in the op, the macros is exists:%{i}._spf.mta.salesforce.com. So if the email server is 1.2.3.4, it will verify that 1.2.3.4._spf.mta.salesforce.com exists, and that DNS server is setup to only respond with valid records for authorized email servers, which is a perfectly valid and reasonable way to handle SPF.
Nope, the mechanism is: %{i}._spf.mta.salesforce.com, So if the email server is 1.2.3.4, it will verify that 1.2.3.4._spf.mta.salesforce.com exists by checking if a A-record exist for that domain
But a A-record lookup for 1.2.3.4._spf.mta.salesforce.com might return a different ip address, and is not limited to the original 1.2.3.4 that send the mail

https://www.zytrax.com/books/dns/apd/rfc7208.txt
5.7. "exists"

This mechanism is used to construct an arbitrary domain name that is
used for a DNS A record query. It allows for complicated schemes
involving arbitrary parts of the mail envelope to determine what is
permitted.

exists = "exists" ":" domain-spec

The <domain-spec> is expanded as per Section 7. The resulting domain
name is used for a DNS A RR lookup (even when the connection type is
IPv6). If any A record is returned, this mechanism matches.

Domains can use this mechanism to specify arbitrarily complex
queries. For example, suppose example.com publishes the record:

v=spf1 exists:%{ir}.%{l1r+-}._spf.%{d} -all
Last edited by RvdH on 2020-02-23 20:32, edited 1 time in total.
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
jimimaseye
Moderator
Moderator
Posts: 8644
Joined: 2011-09-08 17:48

Re: SPF not working correctly

Post by jimimaseye » 2020-02-23 20:29

We are still tackling a problem that we haven't yet proven exists. We are still assuming the macro is the cause of the original failure and are still waiting for the proof that .
(We never assume anything without seeing it). That said it would be useful to see debug level logging for such an event too so we can see what is happening (if you can reproduce it) . Without this it isn't possible to judge. 
[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

mikernet
Normal user
Normal user
Posts: 62
Joined: 2018-09-04 22:22

Re: SPF not working correctly

Post by mikernet » 2020-02-23 20:31

RvdH wrote:
2020-02-23 20:21
mikernet wrote:
2020-02-23 20:02
RvdH wrote:
2020-02-22 22:54
As far i understand the exists:%{i} macro doesn't do much more as checking if the value in %{i} resolves to any A-record, the returned IP address does not need to match that of the sender. So this apparently is the worst option that you can define in your spf record as it security wise doesn't verify anything
That's not the macro though. As stated in the op, the macros is exists:%{i}._spf.mta.salesforce.com. So if the email server is 1.2.3.4, it will verify that 1.2.3.4._spf.mta.salesforce.com exists, and that DNS server is setup to only respond with valid records for authorized email servers, which is a perfectly valid and reasonable way to handle SPF.
Nope, the mechanism is: %{i}._spf.mta.salesforce.com, So if the email server is 1.2.3.4, it will verify that 1.2.3.4._spf.mta.salesforce.com exists by checking a A-record exist for that domain
But a A-record lookup for 1.2.3.4._spf.mta.salesforce.com might return a different ip address, and is not limited to the original 1.2.3.4 that send the mail
It doesn't matter what IP it returns as long as it WON'T return an A record for DNS entries that correspond to invalid email servers. The returned IP address doesn't mean anything other than "yes, the IP address that corresponds to the DNS record you requested is a valid email server".

How exactly is a nefarious sender going to take advantage of that? If a nefarious sender connects to hMailServer from IP 1.2.3.4 then the SPF record check will test 1.2.3.4._spf.mta.salesforce.com which in turn will NOT exist, since 1.2.3.4 is not a SalesForce email server.

Code: Select all

*** dc1.singulink.com can't find 1.2.3.4._spf.mta.salesforce.com: Non-existent domain
Last edited by mikernet on 2020-02-23 20:35, edited 2 times in total.

mikernet
Normal user
Normal user
Posts: 62
Joined: 2018-09-04 22:22

Re: SPF not working correctly

Post by mikernet » 2020-02-23 20:33

jimimaseye wrote:
2020-02-23 20:29
Wee are still tackling a problem that we haven't yet proven exists. We arte still assuming the macro is the cause of the original failure and are still waiting for the proof that .
(We never assume anything without seeing it). That said it would be useful to see debug level logging for such an event too so we can see what is happening (if you can reproduce it) . Without this it isn't possible to judge. 
[Entered by mobile. Excuse my spelling.]
I'm working on a debug trace but that won't happen until Monday. The problem 100% exists that something in the VALID olg.ca SPF record is causing hMailServer to fail the SPF check though.

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-23 20:40

mikernet wrote:
2020-02-23 20:31
RvdH wrote:
2020-02-23 20:21
mikernet wrote:
2020-02-23 20:02


That's not the macro though. As stated in the op, the macros is exists:%{i}._spf.mta.salesforce.com. So if the email server is 1.2.3.4, it will verify that 1.2.3.4._spf.mta.salesforce.com exists, and that DNS server is setup to only respond with valid records for authorized email servers, which is a perfectly valid and reasonable way to handle SPF.
Nope, the mechanism is: %{i}._spf.mta.salesforce.com, So if the email server is 1.2.3.4, it will verify that 1.2.3.4._spf.mta.salesforce.com exists by checking a A-record exist for that domain
But a A-record lookup for 1.2.3.4._spf.mta.salesforce.com might return a different ip address, and is not limited to the original 1.2.3.4 that send the mail
It doesn't matter what IP it returns as long as it WON'T return an A record for DNS entries that correspond to invalid email servers. The returned IP address doesn't mean anything other than "yes, the IP address that corresponds to the DNS record you requested is a valid email server".

How exactly is a nefarious sender going to take advantage of that? If a nefarious sender connects to hMailServer from IP 1.2.3.4 then the SPF record check will test 1.2.3.4._spf.mta.salesforce.com which in turn will NOT exist, since 1.2.3.4 is not SalesForce email server.
Sure, fine by me...i don't care much if it returns the expected ip address, but it my opinion it should (most i tested do)

For example: 52.31.115.240 send a mail, spf exists:%{i}._spf.sparkpostmail.com is in place there
52.31.115.240._spf.sparkpostmail.com resolves to A RR 52.31.115.240
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

mikernet
Normal user
Normal user
Posts: 62
Joined: 2018-09-04 22:22

Re: SPF not working correctly

Post by mikernet » 2020-02-23 20:43

RvdH wrote:
2020-02-23 20:40

Sure, fine by me...i don't care much if it returns the expected ip address, but it my opinion it should (most i tested do)

For example: 52.31.115.240 send a mail, spf exists:%{i}._spf.sparkpostmail.com is in place there
52.31.115.240._spf.sparkpostmail.com resolves to A RR 52.31.115.240
It literally makes no difference for the security of this particular mechanism, which you stated is problematic for some reason that I can't figure out. That simply isn't true. The IP returned is completely irrelevant and returning the same IP as the DNS test request would do nothing to improve the security of the process.

User avatar
SorenR
Senior user
Senior user
Posts: 3623
Joined: 2006-08-21 15:38
Location: Denmark

Re: SPF not working correctly

Post by SorenR » 2020-02-23 20:45

mikernet wrote:
2020-02-23 20:33
I'm working on a debug trace but that won't happen until Monday. The problem 100% exists that something in the VALID olg.ca SPF record is causing hMailServer to fail the SPF check though.
Wireshark ?? I believe you should be able to filter "dns.spf" ...
SørenR.

“Those who don't know history are doomed to repeat it.”
― Edmund Burke

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-23 20:46

mikernet wrote:
2020-02-23 20:43
RvdH wrote:
2020-02-23 20:40

Sure, fine by me...i don't care much if it returns the expected ip address, but it my opinion it should (most i tested do)

For example: 52.31.115.240 send a mail, spf exists:%{i}._spf.sparkpostmail.com is in place there
52.31.115.240._spf.sparkpostmail.com resolves to A RR 52.31.115.240
It literally makes no difference for the security of this particular mechanism, which you stated is problematic for some reason that I can't figure out. That simply isn't true. The IP returned is completely irrelevant and returning the same IP as the DNS test request would do nothing to improve the security of the process.
I find that weird....but hey, that might be just me :)
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

mikernet
Normal user
Normal user
Posts: 62
Joined: 2018-09-04 22:22

Re: SPF not working correctly

Post by mikernet » 2020-02-23 20:47

SorenR wrote:
2020-02-23 20:45
mikernet wrote:
2020-02-23 20:33
I'm working on a debug trace but that won't happen until Monday. The problem 100% exists that something in the VALID olg.ca SPF record is causing hMailServer to fail the SPF check though.
Wireshark ?? I believe you should be able to filter "dns.spf" ...
I need OLG to send me an email first :) They won't be in the office again until Monday.

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-23 20:59

mikernet wrote:
2020-02-23 20:47
SorenR wrote:
2020-02-23 20:45
mikernet wrote:
2020-02-23 20:33
I'm working on a debug trace but that won't happen until Monday. The problem 100% exists that something in the VALID olg.ca SPF record is causing hMailServer to fail the SPF check though.
Wireshark ?? I believe you should be able to filter "dns.spf" ...
I need OLG to send me an email first :) They won't be in the office again until Monday.
FYI, spfverify and dynamic greylistwhitelisting mentioned above would have worked for your case
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
SorenR
Senior user
Senior user
Posts: 3623
Joined: 2006-08-21 15:38
Location: Denmark

Re: SPF not working correctly

Post by SorenR » 2020-02-24 17:19

I'm afraid we will have to rework - or look for an alternative SPF Lookup solution.

It appears the author of the current solution (Roger Moser) is no longer serving the e-mail and conferencing needs of sincere followers of His Divine Grace A.C. Bhaktivedanta Swami Prabhupada ... ISKCON ... commonly known as the "Hare Krishna Movement".
SørenR.

“Those who don't know history are doomed to repeat it.”
― Edmund Burke

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-24 20:44

SorenR wrote:
2020-02-24 17:19
I'm afraid we will have to rework - or look for an alternative SPF Lookup solution.

It appears the author of the current solution (Roger Moser) is no longer serving the e-mail and conferencing needs of sincere followers of His Divine Grace A.C. Bhaktivedanta Swami Prabhupada ... ISKCON ... commonly known as the "Hare Krishna Movement".
Because? v1.10 already is able to parse spf exists:%{i} records, you referenced the v1.12 release notes for used library yourself so you should have seen it didn't add anything for that specific task

My guess this greylist problem as referred to by the OP as nothing to do with the SPF records used....i suspect it to be a problem with DNS (lookup)
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
jimimaseye
Moderator
Moderator
Posts: 8644
Joined: 2011-09-08 17:48

Re: SPF not working correctly

Post by jimimaseye » 2020-02-24 21:22

RvdH wrote:
2020-02-24 20:44
My guess this greylist problem as referred to by the OP as nothing to do with the SPF records used....i suspect it to be a problem with DNS (lookup)
Yep. Still waiting for the proof. Lots of supposition, and a smidge of coincidence, but no proof. And the fact that the code (quoted) shows macros already handled in 5.6 suggests we are eating time on this theory.

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

tunis
Senior user
Senior user
Posts: 251
Joined: 2015-01-05 20:22
Location: Sweden

Re: SPF not working correctly

Post by tunis » 2020-02-25 09:43

Maybe the SPF record has more then 10 lookups?
RMSPF.cpp#L137

https://sendgrid.com/.../spf-limitations/
HMS 5.6.8 B2494.25 on Windows Server 2019 Core VM.
HMS 5.6.8 B2494.24 on Windows Server 2016 Core VM.
HMS 5.6.7 B2425.16 on Windows Server 2012 R2 Core VM.

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-25 10:07

tunis wrote:
2020-02-25 09:43
Maybe the SPF record has more then 10 lookups?
RMSPF.cpp#L137

https://sendgrid.com/.../spf-limitations/
i count just 4 lookups...

include:_spf.fireeyecloud.com
include:spf.protection.outlook.com
include:_spf.salesforce.com
exists:%{i}._spf.mta.salesforce.com
The "include", "a", "mx", "ptr", and "exists" mechanisms as well as the "redirect" modifier do count against this limit. The "all", "ip4", and "ip6" mechanisms do not require DNS lookups and therefore do not count against this limit
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
SorenR
Senior user
Senior user
Posts: 3623
Joined: 2006-08-21 15:38
Location: Denmark

Re: SPF not working correctly

Post by SorenR » 2020-02-25 12:39

Came across this site https://vamsoft.com/support/tools/spf-policy-tester

Click on the "advanced" tab and use these data from the forum.

Sender IP Address: 40.107.223.80
Sender Address: XXXXXXX@olg.ca
HELO/EHLO Domain: NAM11-DM6-obe.outbound.protection.outlook.com

Yes, RFC says that if sender is not found the HELO/EHLO greeting should be used. :roll:

I can't include the output here due to formatting issues but --> Click on TEST ... A complete evaluation is made. Interesting in a sort of weird geek'ish fasion :mrgreen: 8)

It appears that only 2 of 10 lookups are used. If we assume the evaluation is performed (top down) the same way hMailServer is doing it (still need to see a WireShark trace of the DNS data exchange) then the "_spf.salesforce.com" record is not even queried thus the macro is never "seen" AND this reveals a completely different problem alltogether.
SørenR.

“Those who don't know history are doomed to repeat it.”
― Edmund Burke

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-25 13:41

SorenR wrote:
2020-02-25 12:39
Yes, RFC says that if sender is not found the HELO/EHLO greeting should be used. :roll:
Should, but hmailserver does not check SPF for HELO/EHLO altogether ....therefor my initial remark saying SPF testing in hmailserver is very limited/basic
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

mikernet
Normal user
Normal user
Posts: 62
Joined: 2018-09-04 22:22

Re: SPF not working correctly

Post by mikernet » 2020-02-25 13:47

I don't think I've ever had a problem with HELO not being used before. I would be surprised if a legit mail sender would rely on that behavior. Would that not completely eliminate the purpose of SPF anyway?

The basic test in the link you provided which does not use HELO passes fine though.

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-25 13:53

https://vamsoft.com/support/tools/spf-policy-tester seems to stop processing once a match (pass result) is found, and therefor is not expanding/walking all spf record values
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
SorenR
Senior user
Senior user
Posts: 3623
Joined: 2006-08-21 15:38
Location: Denmark

Re: SPF not working correctly

Post by SorenR » 2020-02-25 13:56

mikernet wrote:
2020-02-25 13:47
I don't think I've ever had a problem with HELO not being used before. I would be surprised if a legit mail sender would rely on that behavior. Would that not completely eliminate the purpose of SPF anyway?

The basic test in the link you provided which does not use HELO passes fine though.
It would only be a problem if there is no sender... How often do you get emails with no sender?

A blank sender usually means a Delivery Notification in the form of a NDR ...
SørenR.

“Those who don't know history are doomed to repeat it.”
― Edmund Burke

mikernet
Normal user
Normal user
Posts: 62
Joined: 2018-09-04 22:22

Re: SPF not working correctly

Post by mikernet » 2020-02-25 13:57

Oh sorry, misread what he said.

User avatar
SorenR
Senior user
Senior user
Posts: 3623
Joined: 2006-08-21 15:38
Location: Denmark

Re: SPF not working correctly

Post by SorenR » 2020-02-25 13:59

RvdH wrote:
2020-02-25 13:53
https://vamsoft.com/support/tools/spf-policy-tester seems to stop processing once a match (pass result) is found, and therefor is not expanding/walking all spf record values
Anything wrong with that?
SørenR.

“Those who don't know history are doomed to repeat it.”
― Edmund Burke

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-25 14:00

SorenR wrote:
2020-02-25 13:59
Anything wrong with that?
SorenR wrote:
2020-02-25 13:59
It appears that only 2 of 10 lookups are used. If we assume the evaluation is performed (top down) the same way hMailServer is doing it (still need to see a WireShark trace of the DNS data exchange) then the "_spf.salesforce.com" record is not even queried thus the macro is never "seen" AND this reveals a completely different problem alltogether.
You noticed it didn't expand _spf.salesforce.com, i only giving the reason why
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
SorenR
Senior user
Senior user
Posts: 3623
Joined: 2006-08-21 15:38
Location: Denmark

Re: SPF not working correctly

Post by SorenR » 2020-02-25 14:26

RvdH wrote:
2020-02-25 14:00
SorenR wrote:
2020-02-25 13:59
Anything wrong with that?
SorenR wrote:
2020-02-25 13:59
It appears that only 2 of 10 lookups are used. If we assume the evaluation is performed (top down) the same way hMailServer is doing it (still need to see a WireShark trace of the DNS data exchange) then the "_spf.salesforce.com" record is not even queried thus the macro is never "seen" AND this reveals a completely different problem alltogether.
You noticed it didn't expand _spf.salesforce.com, i only giving the reason why
I get that ... I have not been really deep into the rmspf.cpp code but is that not the way it works? It would seem a complete waste of time and cpu cycles to continue to the end and if that is the case then we have a different problem with SPF ...

Code: Select all

+0 ms		Starting SPF policy evaluation.
			Policy: "v=spf1 ip4:34.223.9.0/24 ip4:34.223.11.128/26 ip4:34.223.11.192/26 ip4:34.223.12.0/25 
			ip4:38.27.116.128/27 ip4:165.254.91.16/28 ip4:38.27.116.96/27 ip4:165.254.91.96/27 
			ip4:149.13.95.32/27 ip4:154.57.155.16/28 ip4:209.135.212.132 ip4:162.210.234.132 ip4:100.25.99.0/25 
			ip4:100.24.127.128/25 ip4:3.122.63.0/24 ip4:52.215.218.128/25 ip4:63.34.31.0/25 ip4:63.34.218.0/24 
			ip4:3.123.5.0/24 ip4:34.223.36.0/24 ip4:3.93.93.0/24 ip4:3.112.99.0/24 ip4:3.112.100.0/25 
			ip4:3.112.100.128/25 -all"
+0 ms		The policy passed syntax validation with 1 warning(s).
			Warning #1: The length of the policy (484 characters) exceeds the recommended limit of 450 
			characters. This may cause the evaluation to fail (see RFC7208, Section 3.4.).
SørenR.

“Those who don't know history are doomed to repeat it.”
― Edmund Burke

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-25 15:28

SorenR wrote:
2020-02-25 14:26
I get that ... I have not been really deep into the rmspf.cpp code but is that not the way it works? It would seem a complete waste of time and cpu cycles to continue to the end and if that is the case then we have a different problem with SPF ...
Most likely

Off-topic...
I myself am a bit curious how that SPFMAXLOOKUPS is calculated, for a SPF record like "v=spf1 mx -all" you have to do 2 lookups, first query and resolve the MX record(s), next get the A-record for that record(s)
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

User avatar
SorenR
Senior user
Senior user
Posts: 3623
Joined: 2006-08-21 15:38
Location: Denmark

Re: SPF not working correctly

Post by SorenR » 2020-02-25 15:47

RvdH wrote:
2020-02-25 15:28
SorenR wrote:
2020-02-25 14:26
I get that ... I have not been really deep into the rmspf.cpp code but is that not the way it works? It would seem a complete waste of time and cpu cycles to continue to the end and if that is the case then we have a different problem with SPF ...
Most likely

Off-topic...
I myself am a bit curious how that SPFMAXLOOKUPS is calculated, for a SPF record like "v=spf1 mx -all" you have to do 2 lookups, first query and resolve the MX record(s), next get the A-record for that record(s)
https://tools.ietf.org/html/rfc7208#section-4.6.4
SørenR.

“Those who don't know history are doomed to repeat it.”
― Edmund Burke

User avatar
SorenR
Senior user
Senior user
Posts: 3623
Joined: 2006-08-21 15:38
Location: Denmark

Re: SPF not working correctly

Post by SorenR » 2020-02-25 16:21

SørenR.

“Those who don't know history are doomed to repeat it.”
― Edmund Burke

User avatar
RvdH
Senior user
Senior user
Posts: 1085
Joined: 2008-06-27 14:42
Location: Netherlands

Re: SPF not working correctly

Post by RvdH » 2020-02-25 16:41

For hmailserver?
the ARSoft.Tools.Net is written in C#, and hmailserver is written in C++
CIDR to RegEx: d-fault.nl/CIDRtoRegEx
DNS Lookup: d-fault.nl/DNSTools
DNSBL Lookup: d-fault.nl/DNSBLLookup
GEOIP Lookup: d-fault.nl/GeoipLookup

mikernet
Normal user
Normal user
Posts: 62
Joined: 2018-09-04 22:22

Re: SPF not working correctly

Post by mikernet » 2020-02-25 21:36

Code: Select all

"DEBUG"	63464	"2020-02-25 14:33:14.018"	"Creating session 940"
"DEBUG"	63464	"2020-02-25 14:33:14.018"	"Executing event OnClientConnect"
"DEBUG"	63464	"2020-02-25 14:33:14.440"	"Event completed"
"DEBUG"	63464	"2020-02-25 14:33:14.440"	"TCP connection started for session 939"
"SMTPD"	63464	939	"2020-02-25 14:33:14.440"	"40.107.223.81"	"SENT: 220 mail.singulink.com ESMTP"
"SMTPD"	83224	939	"2020-02-25 14:33:14.486"	"40.107.223.81"	"RECEIVED: EHLO NAM11-DM6-obe.outbound.protection.outlook.com"
"SMTPD"	83224	939	"2020-02-25 14:33:14.486"	"40.107.223.81"	"SENT: 250-mail.singulink.com[nl]250-SIZE 102400000[nl]250-STARTTLS[nl]250-AUTH LOGIN PLAIN[nl]250 HELP"
"SMTPD"	63408	939	"2020-02-25 14:33:14.518"	"40.107.223.81"	"RECEIVED: STARTTLS"
"SMTPD"	63408	939	"2020-02-25 14:33:14.518"	"40.107.223.81"	"SENT: 220 Ready to start TLS"
"DEBUG"	63408	"2020-02-25 14:33:14.518"	"Performing SSL/TLS handshake for session 939. Verify certificate: False"
"SMTPD"	83116	939	"2020-02-25 14:33:14.799"	"40.107.223.81"	"RECEIVED: EHLO NAM11-DM6-obe.outbound.protection.outlook.com"
"SMTPD"	83116	939	"2020-02-25 14:33:14.799"	"40.107.223.81"	"SENT: 250-mail.singulink.com[nl]250-SIZE 102400000[nl]250-AUTH LOGIN PLAIN[nl]250 HELP"
"SMTPD"	83036	939	"2020-02-25 14:33:14.830"	"40.107.223.81"	"RECEIVED: MAIL FROM:<XXXXX@olg.ca> SIZE=31985"
"DEBUG"	83036	"2020-02-25 14:33:14.955"	"Spam test: SpamTestDNSBlackLists, Score: 0"
"DEBUG"	83036	"2020-02-25 14:33:14.971"	"Spam test: SpamTestHeloHost, Score: 1"
"DEBUG"	83036	"2020-02-25 14:33:14.986"	"Spam test: SpamTestMXRecords, Score: 0"
"DEBUG"	83036	"2020-02-25 14:33:15.033"	"Spam test: SpamTestSPF, Score: 0"
"DEBUG"	83036	"2020-02-25 14:33:15.033"	"Total spam score: 1"
"SMTPD"	83036	939	"2020-02-25 14:33:15.049"	"40.107.223.81"	"SENT: 250 OK"
"SMTPD"	63408	939	"2020-02-25 14:33:15.080"	"40.107.223.81"	"RECEIVED: RCPT TO:<XXXXXX@sbgamingsystems.com>"
"SMTPD"	63408	939	"2020-02-25 14:33:15.111"	"40.107.223.81"	"SENT: 451 Please try again later."
"SMTPD"	83224	939	"2020-02-25 14:33:15.143"	"40.107.223.81"	"RECEIVED: QUIT"
"DEBUG"	83224	"2020-02-25 14:33:15.143"	"Deleting message file."
"SMTPD"	83224	939	"2020-02-25 14:33:15.143"	"40.107.223.81"	"SENT: 221 goodbye"
"DEBUG"	63408	"2020-02-25 14:33:15.158"	"Ending session 939"
"DEBUG"	83924	"2020-02-25 14:33:20.096"	"No messages to index."

Post Reply