Notice: Not officially supported, but possible and (from my point of view) stable.
All you need is .net 2.0 32-bit support on R2 core. For this you have to type at command prompt:
Start /w ocsetup ServerCore-WOW64
Start /w ocsetup NetFx2-ServerCore-WOW64
After that you're able to install hmailserver. Ignore install errors of IE6, the setup tries this because of .net 2.0. You already installed that, so...
Some buttons within the localhost server administrator using explorer views etc. won't work, but you'll get over this. Remotely administered you'll notice no difference between core or no-core.
For setting DCOM permissions for webadmin by command line see http://hmailserver.com/forum/viewtopic.php?f=12&t=18679