IMAP with mobile phones

Use this forum if you have installed hMailServer and want to ask a question related to a production release of hMailServer. Before posting, please read the troubleshooting guide. A large part of all reported issues are already described in detail here.
Post Reply
User avatar
katip
Senior user
Senior user
Posts: 1173
Joined: 2006-12-22 07:58
Location: Istanbul

IMAP with mobile phones

Post by katip » 2023-11-11 10:43

Production server with ~90K mail process/day (awstats)
Mail storage ~3.5 mio mails ~4 TB data ~150 users
Dedicated metal, server H/W, plenty of RAM, Win2019
~150 workstations - all POP
~50 phones - all IMAP, Android/iPhone mixed

Using 5.7 builds, starting from some months ago, during the day (work hours), sooner or later CPU quickly peaks to 95-99% and stays there, RAM usage never exceeds 2GB, data directory starts to fill with zero bytes mails, whole system almost dead, HMS must be killed from Task Manager, plenty of errors logged.
And starting from about 1 month ago, it was no more "sooner or later", but "very soon".

No need to tell that i tried anything possible such as disabling all rules, scripts, tasks, ClamAV, SA... did memcheck, diskscan... downgraded to last officials/betas and previous community builds (Rvdh), both 32/64 bit... fine (very fine) tuned MySQL.. nothing changed.

5 days ago i instructed all phone users to switch to POP and blocked 143/993 on FW.
I tell you what, since then i have not a single error logged (something unseen to me, 5 days in a row without an ERROR_Log file!!) and whole system working like swiss clock.

Currently running 5.7.0 B2643.6 by Rvdh

One hint: Messages with 20-30MB size are very common in our communication (mostly pdf/Office attachments) and phone (IMAP) users increased from a few to about 50 during this year.

yes, HMS rocks - without IMAP! I'm fine with it, never been an IMAP fan anyway.
My humble conclusion: For our workload, HMS IMAP has some serious issues - at least on mobile phones.
Katip
--
HMS 5.7, MariaDB 10.4.10, SA 4.0.0, ClamAV 0.103.8

palinka
Senior user
Senior user
Posts: 4591
Joined: 2017-09-12 17:57

Re: IMAP with mobile phones

Post by palinka » 2023-11-11 11:48

50 imap users doesn't sound like much.

One thing I've noticed about imap is that almost all errors are caused by bad programming of clients.

I have horde in between hmailserver and mobile clients. Clients connect to horde using exchange activesync on port 443. Horde translates commands from the client into imap commands to hmailserver. This has worked very well for me. Of course, my volume is nowhere close to yours.

All users (all family) either use activesync on mobile or roundcube on PC. I'm the only user with an actual imap client (emclient).

You can experiment with horde without any risk at all to your production server. Install horde on a webserver anywhere in your network.

Anyway, that's just some food for thought.

User avatar
katip
Senior user
Senior user
Posts: 1173
Joined: 2006-12-22 07:58
Location: Istanbul

Re: IMAP with mobile phones

Post by katip » 2023-11-11 12:28

palinka wrote:
2023-11-11 11:48
One thing I've noticed about imap is that almost all errors are caused by bad programming of clients.
+1
in fact K9 works very well since recent and i strongly encourage users to use it, they do so as far as i notice. however inevitably there are many iPhones too. it's not easy to track down who causes this evil by doing what. logging don't help much, you're drown in lines. and what does it help? very likely culprit is iPhone. but you can't impose people not to use this crap. boss & sons use iPhone too :P

yes, Horde may be a workaround. i was thinking to try Dovecot in proxy mode on one of our Ubuntu, when i have time. anyway, till then all is fine now. we'll see..
Katip
--
HMS 5.7, MariaDB 10.4.10, SA 4.0.0, ClamAV 0.103.8

palinka
Senior user
Senior user
Posts: 4591
Joined: 2017-09-12 17:57

Re: IMAP with mobile phones

Post by palinka » 2023-11-11 12:43

katip wrote:
2023-11-11 12:28
logging don't help much, you're drown in lines.
Yes :evil:
yes, Horde may be a workaround. i was thinking to try Dovecot in proxy mode on one of our Ubuntu, when i have time. anyway, till then all is fine now. we'll see..
Anything that doesn't directly interfere with your production server has to be a good thing to experiment with. I don't know anything about dovecot, but you can install horde on a separate server and it will connect to your production server BY NAME (very important if you want to use tls). Anyway, if you get it up and running, it can't do any harm to your hmailserver. Its completely outside. You can enable imap on hmailserver and leave a hole in your firewall on 143 only for the server horde is on.

If you need any tips, let me know. I seem to be the only one using horde. Its a bear to install the first time (and the second and third and fourth :lol: ), but once its running and tuned it is rock solid. I never have to maintain it - ever - aside from adjusting individual user settings for this and that I play around with. It just runs. And the icing on the cake is groupware contacts, calendar and tasks.

User avatar
katip
Senior user
Senior user
Posts: 1173
Joined: 2006-12-22 07:58
Location: Istanbul

Re: IMAP with mobile phones

Post by katip » 2023-11-13 18:34

palinka wrote:
2023-11-11 12:43
Anything that doesn't directly interfere with your production server has to be a good thing to experiment with.
Success! With Dovecot as IMAP proxy...
I instructed all phone users to switch back to IMAP and no anomaly till now - it's been a whole "Monday" work day meanwhile - with all phone-team IMAP access on. Zero error, reasonable CPU/RAM.

Horde workaround very likely would help the same. I preferred Dovecot because its setup as proxy is very simple with a few config lines (in contrary to its fine tuned full setup which is kinda atom science :lol: ).

It seems like HMS needs an IMAP "shield/helper" to cope with malicious clients' antics (in our case 99% iPhone my guess). Probably this kind of setup solves also notorious IMAP problems with MS based clients (older Outlooks?). Not tested though, we exclusively use TB.

I keep monitoring closely and hope not having spoken too soon :roll:
Katip
--
HMS 5.7, MariaDB 10.4.10, SA 4.0.0, ClamAV 0.103.8

palinka
Senior user
Senior user
Posts: 4591
Joined: 2017-09-12 17:57

Re: IMAP with mobile phones

Post by palinka » 2023-11-13 18:42

katip wrote:
2023-11-13 18:34
palinka wrote:
2023-11-11 12:43
Anything that doesn't directly interfere with your production server has to be a good thing to experiment with.
Success! With Dovecot as IMAP proxy...
I instructed all phone users to switch back to IMAP and no anomaly till now - it's been a whole "Monday" work day meanwhile - with all phone-team IMAP access on. Zero error, reasonable CPU/RAM.

Horde workaround very likely would help the same. I preferred Dovecot because its setup as proxy is very simple with a few config lines (in contrary to its fine tuned full setup which is kinda atom science :lol: ).

It seems like HMS needs an IMAP "shield/helper" to cope with malicious clients' antics (in our case 99% iPhone my guess). Probably this kind of setup solves also notorious IMAP problems with MS based clients (older Outlooks?). Not tested though, we exclusively use TB.

I keep monitoring closely and hope not having spoken too soon :roll:
Good job. Glad its working. Hmailserver is very strict. That's a good thing 99% of the time.

User avatar
mattg
Moderator
Moderator
Posts: 22456
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: IMAP with mobile phones

Post by mattg » 2023-12-03 03:28

katip wrote:
2023-11-13 18:34
Success! With Dovecot as IMAP proxy...
Can you share your dovecot config?
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
katip
Senior user
Senior user
Posts: 1173
Joined: 2006-12-22 07:58
Location: Istanbul

Re: IMAP with mobile phones

Post by katip » 2023-12-03 07:53

mattg wrote:
2023-12-03 03:28
katip wrote:
2023-11-13 18:34
Success! With Dovecot as IMAP proxy...
Can you share your dovecot config?
install dovecot-imapd (+ dovecot-pop3d if you like to proxy pop too)
go directly to /etc/dovecot/dovecot.conf (i ignored all existing defaults) and do your edits
create a user "imapproxy"
restart dovecot. that's all.

at first connect i had an error due to auto created /home/imapproxy rights, because owner/group is root. change it to 777. user directories are created in it upon first connections and get owner/group "imapproxy" with 700.
i got the clues from this guy: https://www.flomain.de/2015/07/how-to-p ... of-scalix/
you may want to read dovecot related parts only. good luck.

Code: Select all

## Dovecot 1.0 configuration file
#base_dir = /var/run/dovecot/
#protocols = imap imaps
#disable_plaintext_auth = no
## SSL/TLS settings
## Comment these out if you do not wish to provide SSL secured connections.
#ssl_cert_file = /etc/dovecot/cert.pem
#ssl_key_file = /etc/dovecot/key.pem
## Disable SSL/TLS support.
##ssl_disable = no
#ssl_disable = yes
## Login processes
#login_dir = /var/run/dovecot/login
#login_process_per_connection = no
#login_processes_count = 3
## Authentication processes
## If you are not moving mailboxes from host to one on daily basis you can
## use authentication cache pretty safely.
#auth_cache_size = 4096
#auth_cache_ttl = 7200
## Set limit for MySQL lookup processes
#auth_worker_max_count = 30
#auth default {
#  mechanisms = plain
#  # Userdb settings are not used with proxy but there need to be something.
#  userdb static {
#    args = static uid=5000 gid=5000 home=/dev/null
#  }
#  passdb sql {
#    args = /etc/dovecot/sql.conf
#  }
#  user = root
#  count = 1
#}

###

ssl = yes
ssl_cert = </etc/dovecot/mail.mydomain.com-chain.pem
ssl_key = </etc/dovecot/mail.mydomain.com-key.pem
#starttls = any-cert

###

disable_plaintext_auth = no
auth_mechanisms = plain login
mail_uid = imapproxy
mail_gid = imapproxy
# Only IMAP or both
protocols = imap pop3
mail_location = imapc:~/imapc

###

# Something useful
protocol imap {
  mail_max_userip_connections = 20
}

###

# Change the line below to reflect the IP address of your HMS.
imapc_host = 192.168.x.x
# i think entering standart ports doesn't matter, it connects to either of them even without any entered
# imapc_port = 993
# imapc_port = 143

# If also pop3
pop3c_host = 192.168.x.x
# pop3c_port = 995
#pop3c_port = 110
# This is a good thing. If "no" (default), after a POP3 download & leave on server, IMAP shows all as "read"
pop3_no_flag_updates = yes

###

passdb {
 driver = imap
 # Change the line below to reflect the IP address of your HMS.
 args = host=192.168.x.x
 default_fields = userdb_imapc_user=%u userdb_imapc_password=%w
}
userdb {
 driver = prefetch
}

###

mail_home = /home/imapproxy/%u

# eof
Katip
--
HMS 5.7, MariaDB 10.4.10, SA 4.0.0, ClamAV 0.103.8

User avatar
katip
Senior user
Senior user
Posts: 1173
Joined: 2006-12-22 07:58
Location: Istanbul

Re: IMAP with mobile phones

Post by katip » 2023-12-06 13:26

mattg wrote:
2023-12-03 03:28
katip wrote:
2023-11-13 18:34
Success! With Dovecot as IMAP proxy...
Can you share your dovecot config?
FWIW i found this, may be worth to try:
mail_prefetch_count = nn
i started to try with 25. we'll see..

BTW, this came out as useless as it works only on backend installatons, not proxies.
mail_max_userip_connections = xx
excellent resource: https://doc.dovecot.org/settings/core/
JFYI
Katip
--
HMS 5.7, MariaDB 10.4.10, SA 4.0.0, ClamAV 0.103.8

User avatar
mattg
Moderator
Moderator
Posts: 22456
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: IMAP with mobile phones

Post by mattg » 2024-01-31 10:53

I finally have this working - thanks for your help

Took me a while to get the StartTLS connection to my hMailserver to work when AUTH occurs
I think that "source_ip=%r" seems to be too new to be useful; My Ubuntu version ships with an older version of Dovecot.
There are no errors, but I get the Dovecot Proxy IP address in my Hmailserver logs.

That means that I don't auto block international IMAP connection attempts with my hMailserver.
I'll block them at the Proxy, or perhaps the firewall.

Code: Select all

## Dovecot 1.0 configuration file
#base_dir=/var/run/dovecot/

## SSL/TLS settings
ssl=yes
ssl_cert=</etc/letsencrypt/live/example.com/fullchain.pem
ssl_key=</etc/letsencrypt/live/example.com/privkey.pem
ssl_verify_client_cert=no
#starttls=any-cert

# Only IMAP or both IMAP and POP3
protocols=imap
mail_location=imapc:~/imapc
disable_plaintext_auth=no
auth_mechanisms=plain login
mail_uid=imapproxy
mail_gid=mail

# Something useful
protocol imap {
  mail_max_userip_connections=20
}

### IMAPC = IMAP hollow Client - used with AUTH credentials. See below for AUTH
# Change the line below to reflect the IP address of your HMS.
imapc_host=192.168.0.150

## Login processes
#login_dir=/var/run/dovecot/login
# Change the line below to reflect the IP address of your HMS.
login_trusted_networks=192.168.0.150
login_proxy_max_disconnect_delay=30 secs
imap_id_retain=yes
imapc_user=%u
imapc_password=%w
imapc_ssl=starttls
imapc_ssl_verify=no

service imap-login {
  service_count = 1
  process_min_avail = 10
  process_limit = 250
  vsz_limit = 128M
}
# If also pop3
# Change the line below to reflect the IP address of your HMS.
#pop3c_host=192.168.0.150
# This is a good thing. If "no" (default), after a POP3 download & leave on server, IMAP shows all as "read"
#pop3_no_flag_updates=yes

###
# Thunderbird work around
imap_client_workarounds=tb-extra-mailbox-sep tb-lsub-flags

###
# Auth settings
#auth_debug=yes
#auth_debug_passwords=yes
auth_failure_delay=3s

passdb {
 driver=imap
 # Change the line below to reflect the IP address of your HMS.
 args= host=192.168.0.150 ssl=starttls allow_invalid_cert=yes
 default_fields= userdb_imapc_user=%u userdb_imapc_password=%w source_ip=%r
}

userdb {
 driver=prefetch
}

###
# Location of user proxy directories. Will need suitable UNIX permissions
mail_home=/home/imapproxy/%u
# eof
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
katip
Senior user
Senior user
Posts: 1173
Joined: 2006-12-22 07:58
Location: Istanbul

Re: IMAP with mobile phones

Post by katip » 2024-01-31 13:15

mattg wrote:
2024-01-31 10:53
I finally have this working - thanks for your help

Took me a while to get the StartTLS connection to my hMailserver to work when AUTH occurs
I think that "source_ip=%r" seems to be too new to be useful; My Ubuntu version ships with an older version of Dovecot.
Glad to hear this news, you're welcome..
"source_ip=%r" is also new to me. Anyway, it was somehow frustrating that connecting IP appeared as proxy IP in HMS logs. I had to setup a "fail2ban" to track & block brute force. It simply works, there are a number of options, I think it's also possible to manage connecting IPs generally, but never tried.

Meanwhile things changed at our end. I built a dedicated power metal with 12th Gen. i7-12700 2.11 GHz CPU, WD SSD, plenty of RAM, HMS datadir on a WD 8TB HDD... box is almost the best for it's price here. As a result, all trouble vanished ("money opens all doors" we say). Even at peak hours CPU hits 3-5% max, rarely 8-10% momentary, RAM 1.9-2.5GB max. IMAP errors diminished dramatically and no need to tell about general performance increase. So about 2 weeks ago I shut down Dovecot. But in any case it was a good experience for future reference as a possible front-end.
HMS 5.7 B2643.6 (rvdh) on 2019 server
Katip
--
HMS 5.7, MariaDB 10.4.10, SA 4.0.0, ClamAV 0.103.8

itkrmr
New user
New user
Posts: 3
Joined: 2018-11-09 14:17

Re: IMAP with mobile phones

Post by itkrmr » 2024-05-27 15:46

Hello. I tried this method. I managed to set up only without a secure connection. Does anyone have an example of how to use lets encrypt certificates? I use Certify the web to generate certificates, and then export them to PEM files.

User avatar
SorenR
Senior user
Senior user
Posts: 6348
Joined: 2006-08-21 15:38
Location: Denmark

Re: IMAP with mobile phones

Post by SorenR » 2024-05-27 17:50

itkrmr wrote:
2024-05-27 15:46
Hello. I tried this method. I managed to set up only without a secure connection. Does anyone have an example of how to use lets encrypt certificates? I use Certify the web to generate certificates, and then export them to PEM files.
My server is using .crt and .key files... They come directly from Let'sEncrypt using .acme.sh on my Synology (Linux) NAS. IIRC there is also a Windows version of the same.
SørenR.

Woke is Marxism advancing through Maoist cultural revolution.

User avatar
mattg
Moderator
Moderator
Posts: 22456
Joined: 2007-06-14 05:12
Location: 'The Outback' Australia

Re: IMAP with mobile phones

Post by mattg » 2024-05-31 08:56

ALSO ensure that you use the fullchain.pem as this loads the CA certs as well
Just 'cause I link to a page and say little else doesn't mean I am not being nice.
https://www.hmailserver.com/documentation

User avatar
katip
Senior user
Senior user
Posts: 1173
Joined: 2006-12-22 07:58
Location: Istanbul

Re: IMAP with mobile phones

Post by katip » 2024-05-31 09:27

FWIW, I noticed that iPhone/iPad also don't like Dovecot's index cache (empty body or no new mail at all).
Solution:
in dovecot.conf replace

Code: Select all

mail_location=imapc:~/imapc
with

Code: Select all

mail_location = imapc:~/imapc:INDEX=MEMORY
reload service and all is fine. even with any iCrap :lol:
Katip
--
HMS 5.7, MariaDB 10.4.10, SA 4.0.0, ClamAV 0.103.8

Post Reply