hMailServer 5.7

Use this forum if you want to discuss a problem or ask a question related to a hMailServer beta release.
User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

hMailServer 5.7

Post by martin » 2019-08-11 21:34

hMailServer 5.7 builds are now available for download on the build server (x64+newest openssl) at https://build.hmailserver.com/. If someone is curious you can try it out. The next step is to continue go through the issues listed and GitHub and see if there are any which needs fixing in 5.7 which I've started doing.

I think a large majority of the bugs/requests will be closed without action due to lack of time, but there are some bugs such as the POP3/RETR-issue i'll look into. I will also look into the TLS1.2-connectivity when connecting to database engines, since this is something which pops up frequently. If you have some other suggestions (which may not be obvious when I go through github) then please let me know.
Martin Knafve

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: hMailServer 5.7

Post by RvdH » 2019-08-13 08:28

OnHelo & OnClientLogon Script Handlers, both very useful for scripting, blocking, reporting abusive IP's

+ Below a small list of small and easy fixes i have incorporated in 'my' custom builds
  1. Add Return-Path header as topmost header before sending the message to SA for SPF checking (+ delete Return-Path header after the SA check )
  2. Fix ExternalFetcher DELE when no RETR, pull pull #254
  3. SMTP multiply max message size with 1024 issue #267
  4. Add email address variable to SignatureAdder.cpp pull #265
  5. DKIM on acccount-rule 'reply' not applied #172 issue #172
  6. preserve RewriteEnvelopeFromWhenForwarding setting when forwarding from account rule
  7. The logical flow should be to disregard "Require SMTP authentication" if "Allow deliveries from" is unselected issue #287
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: hMailServer 5.7

Post by mattg » 2019-08-13 09:26

RvdH wrote:
2019-08-13 08:28
OnHelo & OnClientLogon Script Handlers, both very useful for scripting, blocking, reporting abusive IP's
+1 from me
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
derelvis
Normal user
Normal user
Posts: 42
Joined: 2018-11-19 19:15

Re: hMailServer 5.7

Post by derelvis » 2019-08-13 09:37

RvdH wrote:
2019-08-13 08:28
2. Fix ExternalFetcher DELE when no RETR, pull pull #254
Thanks!!
HMS 5.7.0-B2484 (x64) (DB version 5700) - MySQL 8.0.15
ClamWin0.99.4 + Clamd service - SpamAssassin 3.4.1.38
Windows Server 2012 - 64Bit

User avatar
Dravion
Senior user
Senior user
Posts: 2071
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: hMailServer 5.7

Post by Dravion » 2019-08-13 10:19

derelvis wrote:
2019-08-13 09:37
RvdH wrote:
2019-08-13 08:28
2. Fix ExternalFetcher DELE when no RETR, pull pull #254
Thanks!!

So, this is your fix?

SetReceiveBinary(false);

https://github.com/hmailserver/hmailser ... 13f0a2d899

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: hMailServer 5.7

Post by RvdH » 2019-08-13 10:44

Dravion wrote:
2019-08-13 10:19
derelvis wrote:
2019-08-13 09:37
RvdH wrote:
2019-08-13 08:28
2. Fix ExternalFetcher DELE when no RETR, pull pull #254
Thanks!!

So, this is your fix?

SetReceiveBinary(false);

https://github.com/hmailserver/hmailser ... 13f0a2d899
Apparently, look here, https://github.com/hmailserver/hmailserver/issues/253
this fix has been in my 5.6.x custom builds for quite a while
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: hMailServer 5.7

Post by martin » 2019-08-13 11:21

I have merged this into master and also added it for 5.6.8 to give derelvis a chance to test it earlier (since he/she may not want to upgrade to 5.7 yet). I am creating new builds now which I'll run through the test suite to check that there's no regressions.
Martin Knafve

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: hMailServer 5.7

Post by RvdH » 2019-08-13 13:27

RvdH wrote:
2019-08-13 08:28
OnHelo & OnClientLogon Script Handlers, both very useful for scripting, blocking, reporting abusive IP's

+ Below a small list of small and easy fixes i have incorporated in 'my' custom builds
  1. Add Return-Path header as topmost header before sending the message to SA for SPF checking (+ delete Return-Path header after the SA check )
  2. Fix ExternalFetcher DELE when no RETR, pull pull #254
  3. SMTP multiply max message size with 1024 issue #267
  4. Add email address variable to SignatureAdder.cpp pull #265
  5. DKIM on acccount-rule 'reply' not applied #172 issue #172
  6. preserve RewriteEnvelopeFromWhenForwarding setting when forwarding from account rule
  7. The logical flow should be to disregard "Require SMTP authentication" if "Allow deliveries from" is unselected issue #287
I got another one, not really a bug more a sort off annoyance
https://github.com/hmailserver/hmailserver/issues/115
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
derelvis
Normal user
Normal user
Posts: 42
Joined: 2018-11-19 19:15

Re: hMailServer 5.7

Post by derelvis » 2019-08-13 18:00

martin wrote:
2019-08-13 11:21
I have merged this into master and also added it for 5.6.8 to give derelvis a chance to test it earlier (since he/she may not want to upgrade to 5.7 yet). I am creating new builds now which I'll run through the test suite to check that there's no regressions.
As already mentioned in the other thread I will have to try the 5.7 version due to already being on DB version 5700.
Thanks for offering the build and the possibility to try it out :-)
HMS 5.7.0-B2484 (x64) (DB version 5700) - MySQL 8.0.15
ClamWin0.99.4 + Clamd service - SpamAssassin 3.4.1.38
Windows Server 2012 - 64Bit

palinka
Senior user
Senior user
Posts: 4461
Joined: 2017-09-12 17:57

Re: hMailServer 5.7

Post by palinka » 2019-08-13 19:18

mattg wrote:
2019-08-13 09:26
RvdH wrote:
2019-08-13 08:28
OnHelo & OnClientLogon Script Handlers, both very useful for scripting, blocking, reporting abusive IP's
+1 from me
+1 from me too. Extremely useful.

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: hMailServer 5.7

Post by RvdH » 2019-08-16 01:47

@martin

I have a pull request for 5.7.x ready, as you might have seen?
https://github.com/hmailserver/hmailser ... eaef3f0b20
This is working, verified and tested code in the 5.6.x branch, you might want to review it though...i did what i did, and all with minimal knowledge of c++
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: hMailServer 5.7

Post by martin » 2019-08-16 12:00

Yep I saw. I intend to review it during weekend.
Martin Knafve

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: hMailServer 5.7

Post by martin » 2019-08-16 16:56

I've merged the changes now. Did some fixes to indentation and added integration tests for it. Thanks!
Martin Knafve

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: hMailServer 5.7

Post by RvdH » 2019-08-16 17:13

martin wrote:
2019-08-16 16:56
I've merged the changes now. Did some fixes to indentation and added integration tests for it. Thanks!
I noticed the indentation changes, how did this happen?
Is there a setting for it somewhere in VS to get it synced with your indentation?

EDIT:
Open 'Tools' in the Visual Studio Menu and click on 'Options'
Click on 'Text Editor', then select 'C/C++' and open the 'Tabs' settings.

What are your settings?

EDIT 2:
Ah, OK...mine is set to 4 (Think this is the default)
Last edited by RvdH on 2019-08-16 17:17, edited 2 times in total.
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: hMailServer 5.7

Post by martin » 2019-08-16 17:15

The code I merged contained a mix of tabs/spaces.

In Visual Studio, the settings are under Tools ->Options->Text Editor->C/C++->Tabs where I have always used "Insert spaces" and tab size/indent size set to 3.

At some point I should probably update all code to use whatever is default in Visual Studio...
Martin Knafve

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: hMailServer 5.7

Post by jimimaseye » 2019-08-16 23:04

So... I'm struggling to keep up. What with official hmailserver 5.6.8, official Hmailserver 5.7, unofficial hmailserver 5.6.8.xxx (RvdH), and heaven only knows how many unofficial dravion hmailservers, what is the latest official 5.7 (found on build.hmailservet.com) offering? Does it have some or all of RvdH patches (OnHelo event etc).

Is the latest 5.7 compilation (which i believe is x64 only) good to go and stable? Or does it contain recently new untested fixes?

Will you be compiling a changelog for 5.7 to which we can refer like you do for the production and beta 5.6.8 releases?

(Isnt it time to change names of the forks/unofficial ones so people can refer to them by name? Eg, Rmailserver or Dmailserver)

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: hMailServer 5.7

Post by RvdH » 2019-08-17 02:24

Wait for it... ;)

All 'my and community' requested changes/fixes are incorporated in 5.7.x today, i believe we all have to wait for martin until he decides 5.7.x is ready as a mainstream, non-alpha, beta or full release
If you're on 5.6.x, either official or my 'custom' build, stick with for the time being until 5.7.x is officially released
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
derelvis
Normal user
Normal user
Posts: 42
Joined: 2018-11-19 19:15

Re: hMailServer 5.7

Post by derelvis » 2019-08-17 08:34

martin wrote:
2019-08-13 11:21
I have merged this into master and also added it for 5.6.8 to give derelvis a chance to test it earlier (since he/she may not want to upgrade to 5.7 yet). I am creating new builds now which I'll run through the test suite to check that there's no regressions.
Short update:
using build 5.7.0-B2469 x64 since thursday morning in production mode now on a Windows Server 2012 without any crashes, error logs or issues.
Very stable in my case. Thanks for the good work, the pull requests and fixes to all of you!
HMS 5.7.0-B2484 (x64) (DB version 5700) - MySQL 8.0.15
ClamWin0.99.4 + Clamd service - SpamAssassin 3.4.1.38
Windows Server 2012 - 64Bit

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: hMailServer 5.7

Post by martin » 2019-08-17 09:42

Nice. RvdH provided me with a pull request for the changes he mentioned above, so they are in 5.7 branch now (so far untested).
Martin Knafve

User avatar
ap44
New user
New user
Posts: 16
Joined: 2018-07-28 13:30
Location: Germany

Re: hMailServer 5.7

Post by ap44 » 2019-08-17 11:33

Short update:
using build 5.7.0-B2469 x64 since thursday morning in production mode now on a Windows Server 2012 without any crashes, error logs or issues.
Very stable in my case. Thanks for the good work, the pull requests and fixes to all of you!
do you fetch mails (pop3) from other servers?
hMailServer-5.7.0-B2604-x64 (MariaDB 10.8.x), SpamAssassin in productive use :D
"SENT: Too many invalid commands. Bye!" :mrgreen:

User avatar
derelvis
Normal user
Normal user
Posts: 42
Joined: 2018-11-19 19:15

Re: hMailServer 5.7

Post by derelvis » 2019-08-17 16:11

Yes, from an external Exchange Server for several domains and from a mail provider for another domain.
HMS 5.7.0-B2484 (x64) (DB version 5700) - MySQL 8.0.15
ClamWin0.99.4 + Clamd service - SpamAssassin 3.4.1.38
Windows Server 2012 - 64Bit

User avatar
ap44
New user
New user
Posts: 16
Joined: 2018-07-28 13:30
Location: Germany

Re: hMailServer 5.7

Post by ap44 » 2019-08-17 22:50

@derelvis Thank you for your feedback!

Now we've completely switched to v5.7.0 and it looks pretty good so far! :)
hMailServer-5.7.0-B2604-x64 (MariaDB 10.8.x), SpamAssassin in productive use :D
"SENT: Too many invalid commands. Bye!" :mrgreen:

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: hMailServer 5.7

Post by mattg » 2019-08-18 03:45

I've moved to 5.7.0-B2481 (x64)

I see an issue with some (not all) of my external accounts

gmail is one
"APPLICATION" 85448 "2019-08-18 11:26:31.078" "The IP address for external account gmail could not be resolved. Aborting fetch."

The server is 'pop.gmail.com' and was working previosuly
When I go to Utilities >> MX Query and tyry to find pop.gmail.com it doesn't respond, but the following is logged

"TCPIP" 138056 "2019-08-18 11:35:18.804" "DNS MX lookup: pop.gmail.com"
"TCPIP" 138056 "2019-08-18 11:35:19.288" "DNS - MX Result: 0 IP addresses were found."




All of the external accounts that I am having issue with have FQDN in the form pop.xxxxx.com


Opening a command prompt and typing

Code: Select all

nslookup -type=mx
pop.gmail.com
Gets me a response

Code: Select all

Non-authoritative answer:
pop.gmail.com   canonical name = gmail-pop.l.google.com

l.google.com
        primary name server = ns1.google.com
        responsible mail addr = dns-admin.google.com
        serial  = 263925741
        refresh = 900 (15 mins)
        retry   = 900 (15 mins)
        expire  = 1800 (30 mins)
        default TTL = 60 (1 min)
So then I changed my External account server to the canonical name 'gmail-pop.l.google.com' and then the certificate fails verification checks. Unchecking the verification checks allows messages to download

Same trick seems to have worked on all of the accounts that have failed to find DNS for external account downloads
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
derelvis
Normal user
Normal user
Posts: 42
Joined: 2018-11-19 19:15

Re: hMailServer 5.7

Post by derelvis » 2019-08-18 09:36

@app44: your're welcome :-) nice to hear

@mattg: Can confirm the part of failing mx query for pop.gmail.com in build 2469. I do not use gmail on our server so I did no further tests.
HMS 5.7.0-B2484 (x64) (DB version 5700) - MySQL 8.0.15
ClamWin0.99.4 + Clamd service - SpamAssassin 3.4.1.38
Windows Server 2012 - 64Bit

User avatar
katip
Senior user
Senior user
Posts: 1161
Joined: 2006-12-22 07:58
Location: Istanbul

Re: hMailServer 5.7

Post by katip » 2019-08-18 09:39

mattg wrote:
2019-08-18 03:45
So then I changed my External account server to the canonical name 'gmail-pop.l.google.com' and then the certificate fails verification checks. Unchecking the verification checks allows messages to download

Same trick seems to have worked on all of the accounts that have failed to find DNS for external account downloads
build 2481
same problem here with gmail + hotmail
outlook.office365.com don't work, but cdg-efz.ms-acdc.office.com did the trick.

another problem:
remote admin throws below error
Untitled 1.png
Untitled 1.png (3.62 KiB) Viewed 186827 times
on server it works fine.
Katip
--
HMS 5.7, MariaDB 10.4.10, SA 4.0.0, ClamAV 0.103.8

User avatar
Dravion
Senior user
Senior user
Posts: 2071
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: hMailServer 5.7

Post by Dravion » 2019-08-18 09:51

@Kaip This script might fix your Remote login
problem

https://www.hmailserver.com/forum/viewt ... 21&t=31511
Last edited by Dravion on 2019-08-18 10:15, edited 1 time in total.

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: hMailServer 5.7

Post by martin » 2019-08-18 09:57

The CNAME issue issue is reported here (since before):
https://github.com/hmailserver/hmailserver/issues/217

I started fixing it yesterday. I'm doing a fairly large change, so probably won't have time to finish it today (but rather tomorrow or tuesday).
Martin Knafve

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: hMailServer 5.7

Post by RvdH » 2019-08-18 10:05

What OS are you on?

Dunno if it is related, but the spfverify.exe utility i wrote a while back for dynamic greylistwhitelisting has some issues resolving CNAME records (pop.gmail.com is CNAME record as well) on Windows Server 2012, on Windows 10 (or Server 2016) this seems no issue. spfverify.exe uses Windows P/Invoke methods of dnsapi.dll to query DNS

On Windows Server 2012 running the command below results in a immediate app crash

Code: Select all

spfverify.exe 208.117.48.80 em.spotify.com
On Windows Server 2016 running the same command executes successfully

Code: Select all

spfverify.exe 208.117.48.80 em.spotify.com
IP address matches SPF record(s) ip range for domain (ExitCode 0)
em.spotify.com is a CNAME for u54769.wl.sendgrid.net

On Windows Server 2012 and Windows Server 2016 running the command below executes successfully for both

Code: Select all

spfverify.exe 208.117.48.80 u54769.wl.sendgrid.net
IP address matches SPF record(s) ip range for domain (ExitCode 0)
Last edited by RvdH on 2019-08-18 10:19, edited 1 time in total.
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
katip
Senior user
Senior user
Posts: 1161
Joined: 2006-12-22 07:58
Location: Istanbul

Re: hMailServer 5.7

Post by katip » 2019-08-18 10:17

katip wrote:
2019-08-18 09:39
another problem:
remote admin throws below error
Untitled 1.png
on server it works fine.
Dravion wrote:
2019-08-18 09:51
Mayve this will do the trick

https://www.hmailserver.com/forum/viewt ... 21&t=31511
no avail :(
it was ever working fine with official builds and your x64 lately.
server 2008R2, client win10
Katip
--
HMS 5.7, MariaDB 10.4.10, SA 4.0.0, ClamAV 0.103.8

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: hMailServer 5.7

Post by mattg » 2019-08-18 11:36

RvdH wrote:
2019-08-18 10:05
What OS are you on?
Unsure who you are asking - But I'm running hMailserver on Windows 10 pro
derelvis wrote:
2019-08-18 09:36
Can confirm the part of failing mx query for pop.gmail.com in build 2469.
As Martin has shown it is a 5.7 error.
I was on 5.6.8 until this morning (~8 hours ago)
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: hMailServer 5.7

Post by RvdH » 2019-08-18 11:55

mattg wrote:
2019-08-18 11:36
As Martin has shown it is a 5.7 error.
I was on 5.6.8 until this morning (~8 hours ago)
A bit premature to switch, don't you think? Martin clearly stated 5.7.x is currently like a alpha build and should not be used in production as of yet
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: hMailServer 5.7

Post by mattg » 2019-08-18 12:48

I figured if it went terribly wrong, I'd roll back.

Good news is that the OnHelo and OnClientLogon are both working well, without needing to change my scripts
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: hMailServer 5.7

Post by martin » 2019-08-18 14:37

The CNAME issue is fixed in 5.7 build 2482.

This is a relatively large change, since I had to clean up the DNS-related code to fix it properly. Could be that there are some side effects. The change itself affects all DNS resolution, and not just the resolution performed when POP3-downloads are being performed. I've tested that basic email delivery still works though, and all integration test passes.

I'm running the build on my server and if you're among the brave ones who use 5.7 already, it may still make sense to wait some day so that I run into any issue first. :)
Martin Knafve

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: hMailServer 5.7

Post by jimimaseye » 2019-08-18 15:13

martin wrote:
2019-08-18 14:37
The CNAME issue is fixed in 5.7 build 2482.
Out of interest, why did the ported code in 5.7 behave different to when it was in 5.6?
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: hMailServer 5.7

Post by martin » 2019-08-18 15:28

jimimaseye wrote:
2019-08-18 15:13
Out of interest, why did the ported code in 5.7 behave different to when it was in 5.6?
As part of changes for 5.7 (years ago), some duplicated code was removed and some code was made more consistent (DNS queries were made in 2 different ways in the DNSResolver class which was reduced to 1). As part of those changes, hMailServer stopped following CNAME records when performing external fetch. Not sure what you mean by ported, but this has nothing to do with x86->x64 if that's what you are referring to.

The change I did today was to fix the above issue and to simplify the code further to make it easier to understand. It's a fairly big change but I think the code is easier to understand now.
Martin Knafve

User avatar
jimimaseye
Moderator
Moderator
Posts: 10060
Joined: 2011-09-08 17:48

Re: hMailServer 5.7

Post by jimimaseye » 2019-08-18 15:36

martin wrote:
2019-08-18 15:28
jimimaseye wrote:
2019-08-18 15:13
Out of interest, why did the ported code in 5.7 behave different to when it was in 5.6?
As part of changes for 5.7 (years ago), some duplicated code was removed and some code was made more consistent (DNS queries were made in 2 different ways in the DNSResolver class which was reduced to 1). As part of those changes, hMailServer stopped following CNAME records when performing external fetch. Not sure what you mean by ported, but this has nothing to do with x86->x64 if that's what you are referring to.

The change I did today was to fix the above issue and to simplify the code further to make it easier to understand. It's a fairly big change but I think the code is easier to understand now.
Ok, fair enough. ("Ported" - I mean transferred or copied to a new version/environment).

If I get time I will try to replicate the original condition and test with which I report in the issue.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: hMailServer 5.7

Post by martin » 2019-08-18 15:41

The host name which you mentioned in your original post (popmail.libero.it) now has a A record but no CNAME. So if you were to try it today with the same host and same hMailServer version as before I don't think would get the error. I have verified that the external fetch can connect to that server though (first thing I tested). If you want to test with a server where the issue occur, then pop.gmail.com is a better option.
Martin Knafve

User avatar
katip
Senior user
Senior user
Posts: 1161
Joined: 2006-12-22 07:58
Location: Istanbul

Re: hMailServer 5.7

Post by katip » 2019-08-18 16:23

katip wrote:
2019-08-18 09:39
another problem:
remote admin throws below error
Untitled 1.png
on server it works fine.
issue looks like to be the same here :
https://www.hmailserver.com/forum/viewtopic.php?t=28767
and here :
https://github.com/hmailserver/hmailserver/issues/122
on client i did a full install and it connects now properly.

BTW i confirm that external POP download from "outlook.office365.com" and "pop.gmail.com" is working now with b2482.
Katip
--
HMS 5.7, MariaDB 10.4.10, SA 4.0.0, ClamAV 0.103.8

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: hMailServer 5.7

Post by RvdH » 2019-08-20 21:58

@martin, i have another pull request ready for IMAPCommandLogin.cpp
The code as you have it now returns the login error before the onClientLogon Event in IMAPCommandLogin.cpp, actually i don't see the usage of the change you made there compared to 5.6.x branch

I believe the the way 5.6.x handles this is better, but you might have a different opinion, so please review it
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
ap44
New user
New user
Posts: 16
Joined: 2018-07-28 13:30
Location: Germany

Re: hMailServer 5.7

Post by ap44 » 2019-08-22 18:02

Conclusion after almost 5 days in productive use. 5.7 works just as stable and reliable as the 5.6.8.

Good work, great result!
hMailServer-5.7.0-B2604-x64 (MariaDB 10.8.x), SpamAssassin in productive use :D
"SENT: Too many invalid commands. Bye!" :mrgreen:

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: hMailServer 5.7

Post by mattg » 2019-08-23 05:30

I'm feeling the same

upgraded seamlessly to b2484 yesterday morning (~30 hours ago)
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: hMailServer 5.7

Post by RvdH » 2019-08-23 13:07

RvdH wrote:
2019-08-20 21:58
@martin, i have another pull request ready for IMAPCommandLogin.cpp
The code as you have it now returns the login error before the onClientLogon Event in IMAPCommandLogin.cpp, actually i don't see the usage of the change you made there compared to 5.6.x branch

I believe the the way 5.6.x handles this is better, but you might have a different opinion, so please review it
Can someone on the 5.7.x branch confirm that when a user tries to login to IMAP without entering the full email address as account name (and without default domain set) the login error is returned before the OnClientLogon event? eg: login with 'info' only instead of 'info@domain.x'
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: hMailServer 5.7

Post by mattg » 2019-08-23 14:10

RvdH wrote:
2019-08-23 13:07
Can someone on the 5.7.x branch confirm that when a user tries to login to IMAP without entering the full email address as account name (and without default domain set) the login error is returned before the OnClientLogon event? eg: login with 'info' only instead of 'info@domain.x'
Yes
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: hMailServer 5.7

Post by mattg » 2019-08-24 02:43

But thinking about this I don't see an issue

If someone tries to log in on port 143, and gets the user name incorrect - then this happens

Code: Select all

"DEBUG"	16012	"2019-08-23 22:09:07.195"	"Pre-creating session 5837"
"TCPIP"	16012	"2019-08-23 22:09:07.195"	"TCP - 192.168.0.199 connected to 10.10.10.150:143."
"DEBUG"	16012	"2019-08-23 22:09:07.195"	"Executing event OnClientConnect"
"DEBUG"	16012	"2019-08-23 22:09:07.336"	"Event completed"
"DEBUG"	16012	"2019-08-23 22:09:07.336"	"TCP connection started for session 5825"
"IMAPD"	16012	5825	"2019-08-23 22:09:07.336"	"192.168.0.199"	"SENT: * OK IMAPrev1"
"IMAPD"	15888	5825	"2019-08-23 22:09:07.352"	"192.168.0.199"	"RECEIVED: 1 capability"
"IMAPD"	15888	5825	"2019-08-23 22:09:07.352"	"192.168.0.199"	"SENT: * CAPABILITY IMAP4 IMAP4rev1 CHILDREN IDLE QUOTA SORT ACL STARTTLS NAMESPACE RIGHTS=texk[nl]1 OK CAPABILITY completed"
"IMAPD"	16012	5825	"2019-08-23 22:09:07.352"	"192.168.0.199"	"RECEIVED: 2 STARTTLS"
"IMAPD"	16012	5825	"2019-08-23 22:09:07.352"	"192.168.0.199"	"SENT: 2 OK Begin TLS negotiation now"
"DEBUG"	15668	"2019-08-23 22:09:07.352"	"Performing SSL/TLS handshake for session 5825. Verify certificate: False"
"TCPIP"	15668	"2019-08-23 22:09:07.461"	"TCPConnection - TLS/SSL handshake completed. Session Id: 5825, Remote IP: 192.168.0.199, Version: TLSv1.2, Cipher: ECDHE-RSA-AES128-GCM-SHA256, Bits: 128"
"IMAPD"	15888	5825	"2019-08-23 22:09:07.461"	"192.168.0.199"	"RECEIVED: 3 capability"
"IMAPD"	15888	5825	"2019-08-23 22:09:07.461"	"192.168.0.199"	"SENT: * CAPABILITY IMAP4 IMAP4rev1 CHILDREN IDLE QUOTA SORT ACL STARTTLS NAMESPACE RIGHTS=texk[nl]3 OK CAPABILITY completed"
"DEBUG"	14540	"2019-08-23 22:09:24.230"	"No messages to index."
"IMAPD"	15668	5825	"2019-08-23 22:09:47.649"	"192.168.0.199"	"RECEIVED: 5 login "server" ***"
"IMAPD"	15668	5825	"2019-08-23 22:09:47.649"	"192.168.0.199"	"SENT: 5 NO Invalid user name. Please use full email address as user name."
"DEBUG"	15668	"2019-08-23 22:10:33.836"	"The read operation failed. Bytes transferred: 0 Remote IP: 192.168.0.199, Session: 5825, Code: 2, Message: End of file"
"DEBUG"	15668	"2019-08-23 22:10:33.836"	"Ending session 5825"
I've done my Nerds.dk tests for country of origin in the OnClientConnect, and have banned them if desired.

The only thing surprising to me is the delay to drop the connection
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
Dravion
Senior user
Senior user
Posts: 2071
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: hMailServer 5.7

Post by Dravion » 2019-08-24 06:22

mattg wrote:
2019-08-24 02:43
I've done my Nerds.dk tests for country of origin in the OnClientConnect, and have banned them if desired.

The only thing surprising to me is the delay to drop the connection
It's because all Network Events in hMailServer specific to SMTP, POP3 and IMAP are handled asynchronous because the underlying
ASIO Library (part of BOOST C++ Framework) handles it as Proactor Pattern

Reference:
https://en.wikipedia.org/wiki/Proactor_pattern

In short:
All incoming connections can be paused and there is a loop which checks the state of the connection and if required react on it.
This means, a reaction of hMailServer is never executed immediately and delays are common.

VBScript on the other hand is processed immediately but it depends on ASIO (SMTP, POP3 and IMAP) responses which finally
results even in delays for VBScripts.

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: hMailServer 5.7

Post by RvdH » 2019-08-24 07:21

mattg wrote:
2019-08-24 02:43
But thinking about this I don't see an issue

If someone tries to log in on port 143, and gets the user name incorrect - then this happens

.....
Maybe for you this is not an issue, but the whole point for the OnClientLogon Event is to get triggered when client login fails/succeeds and right now it doesn't (as seen in your logs)

I, for example have a few honeypot addresses, whenever some external IP address tries to login in one of those accounts on any of the protocols the external IP address is banned, and reported to blocklist.de and abuseipdb.com (all triggered in the OnClientLogon event)
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: hMailServer 5.7

Post by mattg » 2019-08-24 09:27

Ah Ok

I only log successful log-ons via OnCLientLogon, but I see your use case
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
martin
Developer
Developer
Posts: 6846
Joined: 2003-11-21 01:09
Location: Sweden
Contact:

Re: hMailServer 5.7

Post by martin » 2019-08-24 09:53

I've merged the pull request. :)
Martin Knafve

User avatar
RvdH
Senior user
Senior user
Posts: 3235
Joined: 2008-06-27 14:42
Location: The Netherlands

Re: hMailServer 5.7

Post by RvdH » 2019-08-24 09:55

cool 8)
Now i dare to start thinking about switching to 5.7.x :mrgreen:
CIDR to RegEx: d-fault.nl/cidrtoregex
DNS Lookup: d-fault.nl/dnstools
DKIM Generator: d-fault.nl/dkimgenerator
DNSBL Lookup: d-fault.nl/dnsbllookup
GEOIP Lookup: d-fault.nl/geoiplookup

palinka
Senior user
Senior user
Posts: 4461
Joined: 2017-09-12 17:57

Re: hMailServer 5.7

Post by palinka » 2019-08-25 13:30

RvdH wrote:
2019-08-24 09:55
cool 8)
Now i dare to start thinking about switching to 5.7.x :mrgreen:
Me too. Your enhancements on official 64 bit... looking pretty tasty. I'll set some time aside next weekend.

palinka
Senior user
Senior user
Posts: 4461
Joined: 2017-09-12 17:57

Re: hMailServer 5.7

Post by palinka » 2019-08-25 13:32

RvdH wrote:
2019-08-24 07:21
I, for example have a few honeypot addresses, whenever some external IP address tries to login in one of those accounts on any of the protocols the external IP address is banned, and reported to blocklist.de and abuseipdb.com (all triggered in the OnClientLogon event)
I get a text message when my mom messes up the password and gets locked out. LOL but it works. :mrgreen:

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: hMailServer 5.7

Post by mattg » 2019-08-26 01:41

I can confirm the new build works the way you want, ie I can log attempts to log in that are unsuccessful from OnCLintConnect

5.7.0_B2485(x64)

I see there is now a B2486...but with no changes?
is that right?
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

hMailserver-User
Normal user
Normal user
Posts: 38
Joined: 2015-04-25 08:49

Re: hMailServer 5.7

Post by hMailserver-User » 2019-08-26 22:00

Hi!

Just to be sure: i can simply update my 5.6.8 installation - i'm using it with mysql x86 - with 5.7?
Thx!

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: hMailServer 5.7

Post by mattg » 2019-08-26 23:39

Please wait until it is a BETA at least - it is currently very much still ALPHA

But yes just that.

HOWEVER I had to track down the 64 bit libMySQL.dll manually, (this was flagged as to-fix in the hMailserver 5.7 builds - don't know if that is done yet), which also required me to update my MySQL to the latest of the installed version
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
Lee Thompson
Normal user
Normal user
Posts: 38
Joined: 2009-01-15 11:18

Re: hMailServer 5.7

Post by Lee Thompson » 2019-09-04 09:27

Is the MySQL drop in replacement MariaDB (https://mariadb.org/) compatible with 5.7? (Either the 5.5.x or 10.x branch?)

How does 5.7 handle upgrading from 5.6.x (with the old database stuff)?

User avatar
Dravion
Senior user
Senior user
Posts: 2071
Joined: 2015-09-26 11:50
Location: Germany
Contact:

Re: hMailServer 5.7

Post by Dravion » 2019-09-04 14:32

Lee Thompson wrote:
2019-09-04 09:27
Is the MySQL drop in replacement MariaDB (https://mariadb.org/) compatible with 5.7? (Either the 5.5.x or 10.x branch?)

How does 5.7 handle upgrading from 5.6.x (with the old database stuff)?
Since MySQL 8 moved so far from recent MariaDB ABI , it's hard to find common ground.
The safest way is to stick with libmysql for latest vers. 5.7.24+ for 64 and 32-Bit.

glenluo
Senior user
Senior user
Posts: 350
Joined: 2011-07-03 12:10

Re: hMailServer 5.7

Post by glenluo » 2019-09-06 07:52

For SURBL,there is a small bug,maybe the DNS issue,for example,when i set dbl.spamhaus.org. The sender send an email containing the domain youtube.com,then SURBL check return an public IP,such as : 22.33.44.55,the the result is MATCH.
I think it need to check whether it resturn A record like 127.0.XX.XX.

User avatar
mattg
Moderator
Moderator
Posts: 22437
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: hMailServer 5.7

Post by mattg » 2019-09-06 08:52

glenluo wrote:
2019-09-06 07:52
For SURBL,there is a small bug,maybe the DNS issue,for example,when i set dbl.spamhaus.org. The sender send an email containing the domain youtube.com,then SURBL check return an public IP,such as : 22.33.44.55,the the result is MATCH.
I think it need to check whether it resturn A record like 127.0.XX.XX.
I can't see what is returned. Did you find that with Wireshark or similar...?

Here is the logs for a spam message that scored with my SURBL earlier today

2 of 3 SURBL servers picked this message
dbl.spamhaus.org didn't pick these, and upon looking, I can't see ANY time that dbl.spamhaus.org has triggered so far this month.

Perhaps this is a dbl.spamhaus.org issue

Code: Select all

"DEBUG"	16420	"2019-09-06 11:17:35.727"	"SURBL: Execute"
"DEBUG"	16420	"2019-09-06 11:17:35.727"	"SURBL: Found URL: jemma.su"
"DEBUG"	16420	"2019-09-06 11:17:35.727"	"SURBL: Found URL: wordpress.com"
"DEBUG"	16420	"2019-09-06 11:17:35.727"	"SURBL: Found URL: tumblr.com"
"DEBUG"	16420	"2019-09-06 11:17:35.727"	"SURBL: Found URL: eroticasearch.net"
"DEBUG"	16420	"2019-09-06 11:17:35.727"	"SURBL: 4 unique addresses found."
"DEBUG"	16420	"2019-09-06 11:17:35.727"	"SURBL: Lookup: eroticasearch.net.multi.surbl.org"
"DEBUG"	16420	"2019-09-06 11:17:35.758"	"SURBL: Lookup: jemma.su.multi.surbl.org"
"DEBUG"	16420	"2019-09-06 11:17:35.914"	"SURBL: Match found"
"DEBUG"	16420	"2019-09-06 11:17:35.914"	"SURBL: Execute"
"DEBUG"	16420	"2019-09-06 11:17:35.930"	"SURBL: Found URL: jemma.su"
"DEBUG"	16420	"2019-09-06 11:17:35.930"	"SURBL: Found URL: wordpress.com"
"DEBUG"	16420	"2019-09-06 11:17:35.930"	"SURBL: Found URL: tumblr.com"
"DEBUG"	16420	"2019-09-06 11:17:35.930"	"SURBL: Found URL: eroticasearch.net"
"DEBUG"	16420	"2019-09-06 11:17:35.930"	"SURBL: 4 unique addresses found."
"DEBUG"	16420	"2019-09-06 11:17:35.930"	"SURBL: Lookup: eroticasearch.net.dbl.spamhaus.org"
"DEBUG"	16420	"2019-09-06 11:17:35.961"	"SURBL: Lookup: jemma.su.dbl.spamhaus.org"
"DEBUG"	16420	"2019-09-06 11:17:35.977"	"SURBL: Lookup: tumblr.com.dbl.spamhaus.org"
"DEBUG"	16420	"2019-09-06 11:17:35.993"	"SURBL: Lookup: wordpress.com.dbl.spamhaus.org"
"DEBUG"	16420	"2019-09-06 11:17:36.024"	"SURBL: Match not found"
"DEBUG"	16420	"2019-09-06 11:17:36.024"	"SURBL: Execute"
"DEBUG"	16420	"2019-09-06 11:17:36.024"	"SURBL: Found URL: jemma.su"
"DEBUG"	16420	"2019-09-06 11:17:36.024"	"SURBL: Found URL: wordpress.com"
"DEBUG"	16420	"2019-09-06 11:17:36.024"	"SURBL: Found URL: tumblr.com"
"DEBUG"	16420	"2019-09-06 11:17:36.024"	"SURBL: Found URL: eroticasearch.net"
"DEBUG"	16420	"2019-09-06 11:17:36.024"	"SURBL: 4 unique addresses found."
"DEBUG"	16420	"2019-09-06 11:17:36.024"	"SURBL: Lookup: eroticasearch.net.uribl.spameatingmonkey.net"
"DEBUG"	16420	"2019-09-06 11:17:36.414"	"SURBL: Match found"
"DEBUG"	16420	"2019-09-06 11:17:36.414"	"Spam test: SpamTestSURBL, Score: 4"
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

glenluo
Senior user
Senior user
Posts: 350
Joined: 2011-07-03 12:10

Re: hMailServer 5.7

Post by glenluo » 2019-09-06 09:45

It seems that as long as it returns an IP,the SURBL result is "Match found".
When it happens,i use nslookup to check youtube.com.dbl.spamhaus.org,it returns an public IP.
I login my other server oversea,it return null result,so it is not spamhaus issue,it maybe the dns hijacking issue.

palinka
Senior user
Senior user
Posts: 4461
Joined: 2017-09-12 17:57

Re: hMailServer 5.7

Post by palinka » 2019-09-06 12:26

DBL should respond with normal return codes.

https://www.spamhaus.org/faq/section/Spamhaus%20DBL#291
What do the 127.*.*.* Return Codes mean?
The DBL uses DNS return codes in the 127.0.1.0/24 range. Queries regarding any domain listed in DBL and all IP queries will return a response code. If no code is returned (NXDOMAIN) the domain is not listed in DBL. DBL return codes in current and future use are:

Code: Select all

Return Codes	Data Source
127.0.1.2	spam domain
127.0.1.4	phish domain
127.0.1.5	malware domain
127.0.1.6	botnet C&C domain
127.0.1.102	abused legit spam
127.0.1.103	abused spammed redirector domain
127.0.1.104	abused legit phish
127.0.1.105	abused legit malware
127.0.1.106	abused legit botnet C&C
127.0.1.255	IP queries prohibited!
This table will be updated as specific DBL categories are added and 127.0.1.* return codes are assigned to them.

The following special codes indicate an error condition and should not be taken to imply that the queried domain is "listed":

Code: Select all

Return Code	Zone	Description
127.255.255.252	Any	Typing error in DNSBL name
127.255.255.254	Any	Anonymous query through public resolver
127.255.255.255	Any	Excessive number of queries
I definitely get these return codes, but I'm using the spamhaus dbl in a filter in OnHELO, so i do get to see what they send back.

No response, per the above, means no listing.

I haven't upgraded to 5.7 yet, and i don't use spamhaus for hmailserver SURBL checks, for whatever that's worth.

Post Reply