Into.the.Void.

Dear Internet,

I need your help!
I have a debian stable (4.0) server with apache2 (Version: 2.2.3-4+etch6) running which is hosting more than 10 different sites. The problem is that in the apache2 error log I can see a lot of segfaults. All sites though continue to work properly and nobody has ever complained about them.

Some logs:

[Tue Feb 03 18:30:36 2009] [notice] child pid 1353 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:37 2009] [notice] child pid 29343 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:37 2009] [notice] child pid 1350 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:38 2009] [notice] child pid 1349 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:38 2009] [notice] child pid 1352 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:39 2009] [notice] child pid 1354 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:41 2009] [notice] child pid 1380 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:42 2009] [notice] child pid 1378 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:42 2009] [notice] child pid 1714 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:44 2009] [notice] child pid 1715 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:44 2009] [notice] child pid 1718 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:45 2009] [notice] child pid 1720 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:45 2009] [notice] child pid 1721 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:46 2009] [notice] child pid 1723 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:47 2009] [notice] child pid 1724 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:47 2009] [notice] child pid 1725 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:49 2009] [notice] child pid 1726 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:49 2009] [notice] child pid 1728 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:50 2009] [notice] child pid 1729 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:50 2009] [notice] child pid 1730 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:51 2009] [notice] child pid 1358 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:51 2009] [notice] child pid 1733 exit signal Segmentation fault (11)

In order to find out what causes the segfaults I have enabled the following options:
inside /etc/apache2/apache2.conf
CoreDumpDirectory /tmp-apache/
$ ls -Fla / | grep tmp-apache
drwxrwxrwx 2 www-data www-data 4096 2009-01-31 11:01 tmp-apache/

I have changed the ulimit settings inside /etc/security/limits.conf
* soft core unlimited
* hard core unlimited

I have even added a ulimit -c unlimited setting inside /etc/init.d/apache2.
But still I get no core dumps inside /tmp-apache2/ from the segfaulting children.
If I manually kill -11 apache.pid then I can see a core file inside /tmp-apache/

I have only seen one or two core dumps generated by apache and using gdb I could see that they both “blamed” a function of /usr/lib/apache2/modules/libphp5.so. In my quest to find which site/code causes the segfaults I have recompiled apache2 to enable mod_whatkilledus. But no core dump was created in /tmp-apache/ for more than a week even if the segfaults keep happening.

I have reduced my modules, removed mod_python, mod_perl, etc and still these segfaults keep occuring but no core dumps. I suspect that the only time I got a core was when a parent and not a child process segfaulted. I don’t think that my apache2 children dump core when they segfault.

Is there anything I could have done and I haven’t done it ? Is there a way I can force apache2 children to dump core or any other way to determine what causes these segfaults ? All these without of course closing down the sites one by one to see when the segfaults stop…

Thanks in advance to anyone that replies!

P.S. blog’s database is making some tricks…I hope it’s ok now and the post is fully published

Filed by kargig at 22:41 under Linux
14 Comments | 17,175 views