An error occured while executing 'Task-DeliveryTask'

[spyd3r0x]

Hi all,

I recently moved hMailServer to a new server. I'm running 5.6.7 - Build 2425, on Win Server 2019, with SQL Express 2019. The hMail service is running as the Local System account.

I've been running it for many years, without a problem, on an old server. Now that I've moved it to the new server, I'm getting an odd error:

"ERROR" 8336 "2020-04-13 09:10:08.425" "Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'Task-DeliveryTask'"

This error gets written to the logs on a regular basis while email is being sent out. It also creates a dump file:

"ERROR" 8348 "2020-04-13 06:05:17.287" "Severity: 1 (Critical), Code: HM5519, Source: StackLogger::Log, Description: An error has been detected. A mini dump has been written to D:\hMailServer\Logs\minidump_2020-04-13 060517_{AFF7BEE5-EF53-495D-9620-54D1DB51BF1B}.dmp"

The messages that cause the error then get added to the Delivery Queue, with a "Next try" time of ASAP. They then never go out. If I try to manually send one, it sets a "Next try" time of that moment, and then never sends. Most likely because the referenced .eml file doesn't actually exist on the server...

I copied my config over from the older, much slower server, where it ran fine for years. I have AV scanning disabled on hMail's Data and Temp folders, but also tried disabling AV altogether. It didn't help. One thing I haven't tried yet is completely uninstalling AV. Note that I don't have AV enabled within hMailServer.

I tried lowering the performance threading settings. Running 10, 10, 15 right now. I believe I had it at 25, 75, 15 before. No difference.

I took a look at one of the dump files, but didn't get much from it, other than that a memory pointer could not be read. It looks like the error may be related to ntdll!NtWaitForSingleObject.

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.
Exception Information: The thread tried to read from or write to a virtual address for which it does not have the appropriate access.

I'll run a memory diagnostic the next time I can take the server down, but I had run one before setting this server up, so I don't think that is the issue.

I enabled DEBUG logging, and see this, which makes me think it's a DKIM issue. Many emails get signed just fine, though...

Code: Select all

"DEBUG"	8252	"2020-04-13 09:55:47.952"	"Executing task DeliveryTask in work queue SMTP delivery queue"
"DEBUG"	8252	"2020-04-13 09:55:47.952"	"Delivering message..."
"DEBUG"	8252	"2020-04-13 09:55:47.953"	"Applying rules"
"DEBUG"	8252	"2020-04-13 09:55:47.953"	"Performing local delivery"
"DEBUG"	8252	"2020-04-13 09:55:47.954"	"Local delivery completed"
"DEBUG"	8252	"2020-04-13 09:55:47.954"	"Signing message using DKIM..."
"ERROR"	8252	"2020-04-13 09:55:47.993"	"Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'Task-DeliveryTask'"
"DEBUG"	8252	"2020-04-13 09:55:47.993"	"Logging exception.."
"DEBUG"	8252	"2020-04-13 09:55:47.994"	"Minidump creation aborted. The max count (10) is reached and no log is older than 4 hours."
"DEBUG"	8252	"2020-04-13 09:55:47.994"	"Completed logging of exception..."

Anyone have any thoughts or ideas?

Thanks!

[jimimaseye]

It is likely to be permissions related.

Remove dkim signing and try again. If it works then b it is permissions (or location) of the certificates that is causing the problem.

Other thing to now is that hmailserver typically run vbs under SYSTEM account (not 'local') and this is the account that requires the access.

[Entered by mobile. Excuse my spelling.]

[spyd3r0x]

I tried disabling DKIM and still get the same error, it just doesn't come after a DKIM operation.

Code: Select all

"DEBUG"	8332	"2020-04-13 10:43:01.199"	"Executing task DeliveryTask in work queue SMTP delivery queue"
"DEBUG"	8332	"2020-04-13 10:43:01.199"	"Delivering message..."
"DEBUG"	8332	"2020-04-13 10:43:01.200"	"Applying rules"
"DEBUG"	8332	"2020-04-13 10:43:01.200"	"Performing local delivery"
"DEBUG"	8332	"2020-04-13 10:43:01.200"	"Local delivery completed"
"ERROR"	8332	"2020-04-13 10:43:01.301"	"Severity: 2 (High), Code: HM4208, Source: ExceptionHandler::Run, Description: An error occured while executing 'Task-DeliveryTask'"
"DEBUG"	8332	"2020-04-13 10:43:01.301"	"Logging exception.."
"ERROR"	8332	"2020-04-13 10:43:01.455"	"Severity: 1 (Critical), Code: HM5519, Source: StackLogger::Log, Description: An error has been detected. A mini dump has been written to D:\hMailServer\Logs\minidump_2020-04-13 104301_{A2BB3BDD-C27B-42BD-937C-EFC8A8B3B66A}.dmp"
"DEBUG"	8332	"2020-04-13 10:43:01.455"	"Completed logging of exception..."

[spyd3r0x]

Oh, and SYSTEM has Full Control over the entire hMailServer directory, so doesn't seem like it's a permissions issue?

[Dravion]

This is a hmailserver.exe System Service crash.

The new Minidump cannot be written because you have a lot of them already in your log Directory.
Clear your hMailServer log folder and if this happen again, attach a new Minidump file to your next reply.

[spyd3r0x]

This is a hmailserver.exe System Service crash.

The new Minidump cannot be written because you have a lot of them already in your log Directory.
Clear your hMailServer log folder and if this happen again, attach a new Minidump file to your next reply.

Yeah, every time I clear them out, 10 more pop out in short order, hehe. I've attached a couple in a .zip file.

Thanks for looking!

[Dravion]

Ok, it took a look inside of it.

From a Development point of view, this is a Windows Memory Access Problem.

hMailServer is trying to read specific data from a location which it doesn't has granted permission to it.
In Programmers terms, its a pointer running wild.

If something like this occurs windows will forcefully terminate the process which is causing this type of trouble.

But this does not give us a hint what the origins of the problem was, but the log entry 'Task-DeliveryTask'
give a little hint of what might happen. The DeliveryTask is invoked if Outgoing Emails are ready to deploy and is trying to send all Emails out
which are in hMailServer send que.

It could help to Empty the Delivery Que completely and re-send all Emails.

[spyd3r0x]

I had to give it a few days, to make sure all regular retries were cleared out. Once they were, I cleared all of the failed messages from the queue, made sure there were no emails in the Data directory, and restarted the service. Once new emails started going out today, the same service crash started happening.

Any other ideas?

I also have a new error in the logs today, but I suspect it's unrelated:

"ERROR" 12552 "2020-04-15 06:05:16.604" "Severity: 3 (Medium), Code: HM5516, Source: DNSResolver::GetARecords, Description: Attempted DNS lookup for empty host name."

[Dravion]

I had to give it a few days, to make sure all regular retries were cleared out. Once they were, I cleared all of the failed messages from the queue, made sure there were no emails in the Data directory, and restarted the service. Once new emails started going out today, the same service crash started happening.

Any other ideas?

I also have a new error in the logs today, but I suspect it's unrelated:

"ERROR" 12552 "2020-04-15 06:05:16.604" "Severity: 3 (Medium), Code: HM5516, Source: DNSResolver::GetARecords, Description: Attempted DNS lookup for empty host name."

This is a completely different Error.

This is an DNS-Error which means hMailServer (which relies on your Windows DNS-Settings) can't resolve a DNS-Domain A Resource Record, required to identify the
need DNS MX Entry to be able to send an Email.

1) Check "CAREFULLY" your Email address you are trying to resolve
2) Try to send an Email to exactly the same Email recipient by other means (not via hMailServer) to see if the receiving Email address can be contacted.

OR just don't send Email to non existent Email addresses.

[spyd3r0x]

Yeah, I figured the GetARecords error was just an MX-related error. I'm not too worried about that one, as I can track down which list those are coming from.

Any other thoughts regarding the Task-DeliveryTask crash that I still have going?

[Dravion]

Yeah, I figured the GetARecords error was just an MX-related error. I'm not too worried about that one, as I can track down which list those are coming from.

Any other thoughts regarding the Task-DeliveryTask crash that I still have going?

Yeah but first you need to preset your latest log entries (besides DNS Errors) so we can talk about it.

Just attach your newest Crashdumps to your next reply.

Generally spoken:
If a crashdump is written, the following is happening (always):

1) The process in question (in your case hmailserver.exe) was FORCEFULLY terminated because it was about to access a block of Memory it doesn't has permission to.
2) On Crash Situation of hmailserver.exe, the System program wefault.exe jumps in and saved all known states of the crashing program and created a crashdump.

A Program Crashdump is simply a technical situation report about the last milliseconds of a crashing process.
Its not meant to be to gice a complete explanation what went wrong but it can help to puzzle together the overall picture, so its important to have the most recent
Crash dump file as possible.

[nicolasft]

Hello,
I'm having the same problem as you do on my server with hMailServer,
Same configuration as Win Server 2019 + SQL Server
On an old machine with windows server 2012 there is no problem and the configuration of both hMailServer are the same,
Did you get any progress?

[spyd3r0x]

Hello,
I'm having the same problem as you do on my server with hMailServer,
Same configuration as Win Server 2019 + SQL Server
On an old machine with windows server 2012 there is no problem and the configuration of both hMailServer are the same,
Did you get any progress?

Nope, no progress yet, but your post makes me think it's a Server 2019 issue...since we're both running on 2019, and both had working configs on older server versions.

[spyd3r0x]

Just attach your newest Crashdumps to your next reply.

Here's my latest crash file. Thanks!

[nicolasft]

Hello,
I'm having the same problem as you do on my server with hMailServer,
Same configuration as Win Server 2019 + SQL Server
On an old machine with windows server 2012 there is no problem and the configuration of both hMailServer are the same,
Did you get any progress?

Nope, no progress yet, but your post makes me think t's a Server 2019 issue...since we're both running on 2019, and both had working configs on older server versions.

Yes, I suspect too
I have the same hMailServer with the same settings running on a windows server 2012 and it works perfectly
The only things that change are
1) The Windows Version
2) The amount of memory
3) Connecting to a Remote SQL Server

[spyd3r0x]

Yes, I suspect too
I have the same hMailServer with the same settings running on a windows server 2012 and it works perfectly
The only things that change are
1) The Windows Version
2) The amount of memory
3) Connecting to a Remote SQL Server

Most of the hardware specs on my new server have changed versus my old server. More memory, faster CPU, faster HDDs, and so on. My software differences are running Server 2019 and SQL Express 2019 (local install). I was previously on Server 2012 and SQL Express 2014 (local install).

I was wondering if SQL could be the issue, though it seems unlikely. What version is your remote SQL?

[nicolasft]

Yes, I suspect too
I have the same hMailServer with the same settings running on a windows server 2012 and it works perfectly
The only things that change are
1) The Windows Version
2) The amount of memory
3) Connecting to a Remote SQL Server

Most of the hardware specs on my new server have changed versus my old server. More memory, faster CPU, faster HDDs, and so on. My software differences are running Server 2019 and SQL Express 2019 (local install). I was previously on Server 2012 and SQL Express 2014 (local install).

I was wondering if SQL could be the issue, though it seems unlikely. What version is your remote SQL?

Its a SQL Server 2017

I tried now to clean the entire DATA folder to see if something was not "stuck" there but for me the error basically happens when he starts trying to send emails that he didn’t get before,
In this case, enter the list of "Delivery Queue" and do not leave the debbug log, basically says the same thing for everyone

"DEBUG" 2076 "2020-04-15 16: 04: 12.402" "The write operation failed. Bytes transferred: 0 Remote IP: XXXXXXXXXX, Session: 122, Code: 10054, Message: An existing connection was forcibly closed by the remote host "

[nicolasft]

And after that the An error occured while executing 'Task-DeliveryTask'

[SorenR]

I had to give it a few days, to make sure all regular retries were cleared out. Once they were, I cleared all of the failed messages from the queue, made sure there were no emails in the Data directory, and restarted the service. Once new emails started going out today, the same service crash started happening.

Any other ideas?

I also have a new error in the logs today, but I suspect it's unrelated:

"ERROR" 12552 "2020-04-15 06:05:16.604" "Severity: 3 (Medium), Code: HM5516, Source: DNSResolver::GetARecords, Description: Attempted DNS lookup for empty host name."

This is a completely different Error.

[nicolasft]

spyderox

I made a change that seems to have had an effect,
I don't know yet if it has a relationship but after I changed I still didn't have the error,
Check in the windows settings on the network adapter, in the power part, if the option that windows can disable the adapter to save energy is not checked
I need a few more hours to make sure

[mattg]

That will definitely impact on hMailserver

[spyd3r0x]

spyderox

I made a change that seems to have had an effect,
I don't know yet if it has a relationship but after I changed I still didn't have the error,
Check in the windows settings on the network adapter, in the power part, if the option that windows can disable the adapter to save energy is not checked
I need a few more hours to make sure

Nice suggestion! I'm giving it a go, now. I tried disabling Data Execution Prevention over the weekend. That had no effect.

[Dravion]

spyderox

I made a change that seems to have had an effect,
I don't know yet if it has a relationship but after I changed I still didn't have the error,
Check in the windows settings on the network adapter, in the power part, if the option that windows can disable the adapter to save energy is not checked
I need a few more hours to make sure

Nice suggestion! I'm giving it a go, now. I tried disabling Data Execution Prevention over the weekend. That had no effect.

Data Execution Protection (DEP) is not a Problem.

The Problem you are suffering from a Windows Operating System Access Violation Exception.
This kind of Error is only raised by the Windows Kernel if a Process is illegally trying to access a block of so called Virtual Memory (in most cases
RAM-Module Memory nowdays) which it doesn't has permissions to access.

DISCLAIMER:
This has nothing to do with File system permissions, Network permissions or granting process privileges by Registry or Process policy.

I works like this:
if a Windows App is launched (regarles if it is Notepad.exe, Minesweeper.exe or hMailServer.exe) it requests a so called address space.
This is a area of Memory (mostly RAM Module Memory) which any process has as Memory Budget (any Process has its own Memory budget).

For example. If Solitair.exe (the Windows Card Game) is trying to allocate a block of Memory which belongs to mspaint.exe (the
Windows Paint Program) Solitair.exe will raise a Access Violation the same Way as your hMailServer Process has done.

How can this be possible?
In most cases, this has to do with a programming technique called Pointer. A pointer is a Variable which points to a position in
Memory. In the next step, a function which depends on such a pointer address can try to access this position in Memory, but if
the Pointer is pointing to a Position in Memory which is plain and simple wrong it will face the consequences which means the entire
process (in your case hMailServer.exe) is shutdown because a AccessViolation was triggered. This is important: Imagine any
process could interfere with each other process, your Windows would crash every day multiple times.

What can cause a pointer running wild?
In most cases, its a programming Error. The Programmer needs to make sure the pointer is always pointing to correct Memory
positions but this is more complex as it sounds. In this case, this Error only seems to be occur if specific Data (Emails) are processed
which hMailServer cannot process correctly. Thats why i recommended you to Empty your Delivery que to eliminate the faulty Data
in your Emails which caused the Problem at the first place.

Is this a perfect solution?
No. This is only a Workaround, but not a Solution.
To make a Solution possible, we need to pinpoint the Root cause of your Error within hMailServer.exe code and figuring out a
way we can catch problematic Email data before it can harm the entire hMailServer.exe process.

[spyd3r0x]

Data Execution Protection (DEP) is not a Problem.

The Problem you are suffering from a Windows Operating System Access Violation Exception.
This kind of Error is only raised by the Windows Kernel if a Process is illegally trying to access a block of so called Virtual Memory (in most cases
RAM-Module Memory nowdays) which it doesn't has permissions to access.

...

Thats why i recommended you to Empty your Delivery que to eliminate the faulty Data
in your Emails which caused the Problem at the first place.

Thank you for continuing to respond I'm right with you regarding the issue. It's clear what is happening, but what isn't clear is why.

I tried disabling DEP simply because it is related to blocking illegal access to memory. I wasn't seeing any DEP errors in the Event Logs, but figured it was worth a shot.

I've cleared the delivery queue multiple times, and it hasn't made a difference. Brand new emails result in the same issue. Nothing has changed with our configuration, except moving hMailServer to a new server. Clients, email content, and hMail configurations are all the same as they have been for years.

[Dravion]

Data Execution Prevention (DEP) addresses a different Problem.

The Microprocessor we use in our Computers (van Neumann Architecture) doesn't care if a peace of Data is a Machine instruction (executable code)
or plain Data (like your Emails content, or a Picture or sound data ect.).
Hackers can exploit this feature by pretending it's data but in reality it's executable code which will immediately executed inside the tricked process.
An Attacker could smuggle a handcrafted peace of exploit code into an Email and trigger direct Machine execution. To prevent this kind of shit (which is a big security Problem) DEP was invented. If a peace of Data is marked as Data, it cannot be executed and if something is marked as executable code, it will not handled like Data. Its not a 100% completely bulletproof thing to mitigate any evil but it helps a bit.

ps:
Regardless if security measures like DEP (Data Execution Prevention) or ASLR (Address space layout randomization) is enabled,
this cannot cure hMailServers Problem you are facing. The best way to fix your Problem is to understand what exactly is happening.
This can be very time consuming trial and error process and needs a lot of patients.

[Dravion]

Ok, a few things you can checkout.

AccessViolations (like you are facing with your hMailServer Installation) is indeed a Memory Problem, but there are chances this is not only a
Software Problem. In fact, we see a lot of freshly, new installed Servers which are running on faulty Memory or buggy Drivers.
Yes, its true, a lot of new Systems have faulty Memory and you only will experience such an Hardware Memory Module Error if
you have an App which uses lots of Memory (like hMailServer or another Memory heavy process).

Check your Memory Modules carefully and don't use a crappy Memory checker.

This is a good one
https://www.memtest.org/download/5.31b/ ... 1b.iso.zip

1) Download it, unzip it and burn the ISO to a CD-ROM or DVD.
2) Shutdown your Windows Server and Start from your freshly burned CD/DVD and wait a few hours.

if you have more then 16 GBytes of RAM you have to wait 8 hours or more (run it over night)
Take a picture with your Smartphone or a Digicam and attache the picture at your next reply.

[spyd3r0x]

I finally had a chance to run a memtest last night. I actually used the newer UEFI version of memtest (from memtest.com). 32GB of memory, and it found no errors.