Distribution lists + DKIM issues

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
lifeofguenter
New user
New user
Posts: 10
Joined: 2019-06-28 09:56

Distribution lists + DKIM issues

Post by lifeofguenter » 2019-06-29 08:41

Hi all,

It seems that distribution lists don't work well with DKIM verification by hmailserver, especially if the signing method is set to "simple" by the sender.

I am not entirely sure, but it seems because the content of the email is modified a bit and not kept as original. There might be two possibilities to solve this issue:
  • do the dkim verification on message-accept, before it gets passed on to the members of the list
  • do it like gmail, verify before, and "re-send" it while changing the dkim so the receiver has an accurate dkim hash
Thoughts?

User avatar
jimimaseye
Moderator
Moderator
Posts: 8515
Joined: 2011-09-08 17:48

Re: Distribution lists + DKIM issues

Post by jimimaseye » 2019-06-29 09:04

My thoughts:

Simply saying "it doesn't work" doesn't help.

Process? Evidence? Problem? Logs?

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

lifeofguenter
New user
New user
Posts: 10
Joined: 2019-06-28 09:56

Re: Distribution lists + DKIM issues

Post by lifeofguenter » 2019-06-29 09:37

you are right @jimimaseye!

So the DKIM fail is actually happening because the body is being altered - I am no sure yet if its happening due the distribution list or not.

The original message (simplified):

Code: Select all

Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit


DOMAIN: xxx


-----------------------------------------------------------------------------------
                    DOMAIN WILL BE CLOSED / DELETED IN 48 HOURS                    
-----------------------------------------------------------------------------------
When it goes through my hmailserver + distribution list:

Code: Select all

Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-hMailServer-Spam: YES
X-hMailServer-Reason-2: Rejected by DKIM. - (Score: 5)
X-hMailServer-Reason-Score: 5


DOMAIN: xxx


-------------------------------------------------------------------------=
----------
                    DOMAIN WILL BE CLOSED / DELETED IN 48 HOURS          =
         =20
-------------------------------------------------------------------------=
----------
E.g. it is somehow re-encoding the message. If I manually change the body, then I get it dkim-passed again (checked via: http://www.appmaildev.com/en/dkimfile)

Any ideas why the encoding is being changed?

User avatar
jimimaseye
Moderator
Moderator
Posts: 8515
Joined: 2011-09-08 17:48

Re: Distribution lists + DKIM issues

Post by jimimaseye » 2019-06-29 09:45

Who is sending the message? Presumably someone external as your hmailserver is running spam and dkim checks against it. And what are the recipients in the 2 examples?

You're still not telling us the full picture.

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

lifeofguenter
New user
New user
Posts: 10
Joined: 2019-06-28 09:56

Re: Distribution lists + DKIM issues

Post by lifeofguenter » 2019-06-29 10:19

jimimaseye let me know what is missing for the full picture :)
  • this is an automated email sent by inwx.de
  • on my gmail account the email passes dkim (verification done by gmail)
  • on my hmailserver the email does not pass dkim (verification done by hmailserver)
  • those are two separate emails that were sent (I switched the email address directly on inwx.de) - gmail does not know anything about hmailserver and vice-versa
  • there are no external checks in place neither on gmail nor on my hmailserver
  • I however use http://www.appmaildev.com/en/dkimfile to verify dkim offline as an alternative way - just for this forum
  • the recipients in both cases are "distribution lists", in gmail/gsuite this is called "groups"
  • gmail does not alter the body
  • hmailserver _seems_ to be altering the body at some point which is why dkim is failing
I do not know yet if its something hmailserver is doing or if the sender itself is doing something wrong (also a possibility) and at this point I am also not sure if this is an issue with hmailserver in general or the "distribution list" feature. Other dkim emails are coming in fine. So if hmailserver is 100% not re-writing emails with the distribution-list feature then I will chat to inwx.de

User avatar
jimimaseye
Moderator
Moderator
Posts: 8515
Joined: 2011-09-08 17:48

Re: Distribution lists + DKIM issues

Post by jimimaseye » 2019-06-29 10:37

Hmailserver doesn't change any body at all. Antivirus does. At point of receipt and dkim check hms doesn't even know or care that the recipient is a distribution list address.

So an external sender sending in to a hmailserver distribution list address? And what of that same sender sends the same message to an actual (not a distribution list) account in hmailserver?

[Entered by mobile. Excuse my spelling.]
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

lifeofguenter
New user
New user
Posts: 10
Joined: 2019-06-28 09:56

Re: Distribution lists + DKIM issues

Post by lifeofguenter » 2019-06-29 11:32

So I have clamav integrated via hmailserver - but as to my understanding it would not change the body if no virus is found, let alone change the encoding?
So an external sender sending in to a hmailserver distribution list address?
Correct.
And what of that same sender sends the same message to an actual (not a distribution list) account in hmailserver?
Did that - same results. So it does not seem to be an issue with distribution-lists. Sent an email to inwx now :)

Thanks for helping, greatly appreciated the advice :)

User avatar
jimimaseye
Moderator
Moderator
Posts: 8515
Joined: 2011-09-08 17:48

Re: Distribution lists + DKIM issues

Post by jimimaseye » 2019-06-29 11:38

Ok no problem. But yes the error is with them or some other intervention en route to hms.
5.7 on test.
SpamassassinForWindows 3.4.0 spamd service
AV: Clamwin + Clamd service + sanesecurity defs : https://www.hmailserver.com/forum/viewtopic.php?f=21&t=26829

lifeofguenter
New user
New user
Posts: 10
Joined: 2019-06-28 09:56

Re: Distribution lists + DKIM issues

Post by lifeofguenter » 2019-07-01 13:19

hmm I am not so convinced. I am getting the same issues with other emails. And other email providers are not having issues with those emails? Something is re-encoding the email to "quoted-printable" something maybe hmailserver is doing?

User avatar
SorenR
Senior user
Senior user
Posts: 3562
Joined: 2006-08-21 15:38
Location: Denmark

Re: Distribution lists + DKIM issues

Post by SorenR » 2019-07-01 13:41

lifeofguenter wrote:
2019-07-01 13:19
hmm I am not so convinced. I am getting the same issues with other emails. And other email providers are not having issues with those emails? Something is re-encoding the email to "quoted-printable" something maybe hmailserver is doing?
hMailServer do not change the encoding of emails unless you explicitly told it to do so by using the scripting engine.

You could try switch off SpamAssassin and see if that makes a difference, it is the only process where hMailServer hand off the email to an external process and get more than a simple status code back.
SørenR.

“With age comes wisdom, but sometimes age comes alone.”
- Oscar Wilde

User avatar
mattg
Moderator
Moderator
Posts: 20782
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Distribution lists + DKIM issues

Post by mattg » 2019-07-01 13:42

lifeofguenter wrote:
2019-06-29 08:41
There might be two possibilities to solve this issue:

do the dkim verification on message-accept, before it gets passed on to the members of the list
do it like gmail, verify before, and "re-send" it while changing the dkim so the receiver has an accurate dkim hash
From what I can tell hMailserver does both of these things

hMailserver does check the actual connecting IP, and the SMTP FROM, checking the DKIM of that (as opposed to the message headers FROM). Perhaps something happens due to that.

ALSO what barand of router do you use? Does it do any 'mail inspection' or similar...?
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
SorenR
Senior user
Senior user
Posts: 3562
Joined: 2006-08-21 15:38
Location: Denmark

Re: Distribution lists + DKIM issues

Post by SorenR » 2019-07-01 13:52

mattg wrote:
2019-07-01 13:42
lifeofguenter wrote:
2019-06-29 08:41
There might be two possibilities to solve this issue:

do the dkim verification on message-accept, before it gets passed on to the members of the list
do it like gmail, verify before, and "re-send" it while changing the dkim so the receiver has an accurate dkim hash
From what I can tell hMailserver does both of these things

hMailserver does check the actual connecting IP, and the SMTP FROM, checking the DKIM of that (as opposed to the message headers FROM). Perhaps something happens due to that.

ALSO what barand of router do you use? Does it do any 'mail inspection' or similar...?
Cisco is notoriously known for f*** up DKIM when using ESMTP inspection on their security appliances. I fell in the trap many years ago with my ASA 5505 and it took a few days to figure out why.

Googling the matter it seems they also have a problem with TLS. :roll:
SørenR.

“With age comes wisdom, but sometimes age comes alone.”
- Oscar Wilde

lifeofguenter
New user
New user
Posts: 10
Joined: 2019-06-28 09:56

Re: Distribution lists + DKIM issues

Post by lifeofguenter » 2019-07-01 13:57

hMailServer do not change the encoding of emails unless you explicitly told it to do so by using the scripting engine.
I am not trying to not believe you. But I am not being able to replicate this issue on another server.
You could try switch off SpamAssassin
I do not have SpamAssassin enabled
ALSO what barand of router do you use? Does it do any 'mail inspection' or similar...?
good point. I am using a fritzbox. But I am connecting to imap via TLS - so it should not be able to interfere. On my workstation/machine I am running Debian with no AntiVirus scanner installed.

Additionally, just to make sure its not something happening between hMailServer <> my PC (be it my client, router, ISP, etc.) I RDP'ed onto my mail server and had a look at the eml:

Image

That means the email is stored as "quoted-printable". So that is something either happening before hMailServer or with hMailServer (be it either internal/external/whatever). Before hMailServer I have nothing installed. E.g. it is directly connected to the internet (strato.de vserver).

The server requires (START)TLS on all ports - even 25 - that should theoretically additionally avoid any scrambling happening before hMailServer

User avatar
SorenR
Senior user
Senior user
Posts: 3562
Joined: 2006-08-21 15:38
Location: Denmark

Re: Distribution lists + DKIM issues

Post by SorenR » 2019-07-01 18:30

lifeofguenter wrote:
2019-07-01 13:57
hMailServer do not change the encoding of emails unless you explicitly told it to do so by using the scripting engine.
I am not trying to not believe you. But I am not being able to replicate this issue on another server.
It's very easy to prove/disprove. The source is available on Github, any C++ programmer should be able to tell you exactly what is going on.
SørenR.

“With age comes wisdom, but sometimes age comes alone.”
- Oscar Wilde

User avatar
mattg
Moderator
Moderator
Posts: 20782
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: Distribution lists + DKIM issues

Post by mattg » 2019-07-02 00:00

ALSO, more of us would have the same issue (and we don't), AND we could deliberately replicate what you do and get the same outcome.

What language are these emails written in?
Is it possible to get the source message from both your hmailserver and your gmail (not just the headers) to compare the entire message sources side by side?

It certainly looks like some sort of buffer overrun. What version of hMailsevrer are you running?
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

lifeofguenter
New user
New user
Posts: 10
Joined: 2019-06-28 09:56

Re: Distribution lists + DKIM issues

Post by lifeofguenter » 2019-07-02 00:15

Ok I think I found the issue: https://stbuehler.de/blog/article/2011/ ... oding.html

According to https://github.com/hmailserver/hmailser ... .cpp#L1444 + https://en.wikipedia.org/wiki/Extended_SMTP#8BITMIME + confirmed by checking the headers that hMailServer sends: 8bit is not supported.

Which is strange because I am seeing some tests with 8bit + dkim: https://github.com/hmailserver/hmailser ... ext&q=8bit - but maybe I am misinterpreting the tests.

TIL: I always thought that the header information "content-transfer-encoding" is not important for the transport but only important for decoding messages later. Turns out, this is mostly the case except for 8bit which is an extension/something new.
  • So depending on a servers configuration, it will create a email to send and encode it in 8bit.
  • it will connect to hMailServer - during the handshake it will now learn that hMailServer is not capabable of 8bit
  • sending server will now fallback and re-encode the 8bit email to 7bit/qp.
  • However the dkim implementation (bug with sender) is not recognizing the body change and thus not re-caluclating the dkim-hash
  • hMailServer receives 7bit/qp email with outdated dkim-hash of the previous 8bit email
So: not fault of hMailServer but faulty dkim implementation of sender. The fallback process is described in the RFC: https://tools.ietf.org/html/rfc6152 - so its really only the dkim plugin that they use thats faulty.

However: for better compatibility I would maybe suggest for hMailServer to support 8bit? E.g. RFC 6152 / 8BITMIME?

palinka
Senior user
Senior user
Posts: 1915
Joined: 2017-09-12 17:57

Re: Distribution lists + DKIM issues

Post by palinka » 2019-07-02 01:37

Good job. That explains a lot. I had given up on hms checking dkim and just let spamassassin deal with it.

lifeofguenter
New user
New user
Posts: 10
Joined: 2019-06-28 09:56

Re: Distribution lists + DKIM issues

Post by lifeofguenter » 2019-07-02 10:47

Good job. That explains a lot. I had given up on hms checking dkim and just let spamassassin deal with it.
Thanks. This is though really not an issue with the DKIM implementation on hMailServer but with certain DKIM implementation that are sending to hMailServer. The only thing that hMailServer can do to avoid this issue is to support 8bit transfer-encoding.

Post Reply