Cmaild Appears To Hang When Processing Over 500 Messages

RickSanson · Joined: 14 Mar 2007 Posts: 36

Hi!

I am trying to work through a problem where C-Mail falls appears to fall down over 500 messages.

Using top this is what I see...

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU CMD
19644 root 23 0 1404M 1.0G 824 R 33.4 89.1 15:07 1 perl

This is a dual processor box with 2gb ram. 500 messages are processed in about 10 minutes and the "perl" ram consumption varies but over 500 messages perl's ram consumption climbs and then remains fixed at the above value (1404M).

Please help! I need to move as many as 5000!

admin · Site Admin Joined: 22 Mar 2006 Posts: 130

RickSanson · Joined: 14 Mar 2007 Posts: 36

I'm doing a few more runs in testing but here is what I've found:

cmaild (listed as the command "perl") apparently puts all message elements into memory. The more messages to send the more memory it requires until it exceeds system memory which it then begins to consume swap space. Once all swap space is consumed the perl process is simply dropped without notice.

I notice than at 500 messages perl requires about 1.25MB ram per 100 messages until above around 800. 1500 messages consumed "4GB" ram (physical is 2GB). So above 800 messages the process consumes about 2.5MB per 100 messages.

This logs at this time only report back a successful run but when the perl process is dropped no log is written...

Thank you for your help!

RickSanson · Joined: 14 Mar 2007 Posts: 36

Hi!

I built another machine that has two non-parity strip set for both the root and the swap partitions. I did thsi to see if I could get anything out of the swap...

I did 500 messages (html only) using 369MB ram with 0 swap
I did 1000 messages (html only) using 1410MB ram with 256KB swap
I then tried 2000 messages which wound up to 3067MB ram and 1.7GB swap --- Perl died right there, I got an Out of memory message (from top) then saw the perl process come back began to consume more memory and swap but with little processor utilization...

Here is the tail of the cmail_daemon.log

send to:rick@blah.com
send to:rick@blah.com
send to:rick@blah.com
.. Successfully send: 1008 emails, 0 with errors. Spend 1546 seconds.

2008-01-11 14:30:25 STOP deamon

2008-01-11 14:31:40 Deamon START and found 1 active Send Requests

Please help! I'm in serious trouble with this...

RickSanson · Joined: 14 Mar 2007 Posts: 36

Is anyone sending over 1000 messages?

Anyone? Any ideas?

RickSanson · Joined: 14 Mar 2007 Posts: 36

Just as a follow up...

Using flat files (csv) is NOT efficient. The problem here is that there is no garbage collection and the scripts (perl) will consume memory until the machine pukes, kills perl without warning...

However with MySQL I've pushed 20,000 messages per hour consuming not more than 1% memory...