03/02/2009
Help needed on apache2 segfaults
Dear Internet,
I need your help!
I have a debian stable (4.0) server with apache2 (Version: 2.2.3-4+etch6) running which is hosting more than 10 different sites. The problem is that in the apache2 error log I can see a lot of segfaults. All sites though continue to work properly and nobody has ever complained about them.
Some logs:
[Tue Feb 03 18:30:36 2009] [notice] child pid 1353 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:37 2009] [notice] child pid 29343 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:37 2009] [notice] child pid 1350 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:38 2009] [notice] child pid 1349 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:38 2009] [notice] child pid 1352 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:39 2009] [notice] child pid 1354 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:41 2009] [notice] child pid 1380 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:42 2009] [notice] child pid 1378 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:42 2009] [notice] child pid 1714 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:44 2009] [notice] child pid 1715 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:44 2009] [notice] child pid 1718 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:45 2009] [notice] child pid 1720 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:45 2009] [notice] child pid 1721 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:46 2009] [notice] child pid 1723 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:47 2009] [notice] child pid 1724 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:47 2009] [notice] child pid 1725 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:49 2009] [notice] child pid 1726 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:49 2009] [notice] child pid 1728 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:50 2009] [notice] child pid 1729 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:50 2009] [notice] child pid 1730 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:51 2009] [notice] child pid 1358 exit signal Segmentation fault (11)
[Tue Feb 03 18:30:51 2009] [notice] child pid 1733 exit signal Segmentation fault (11)
In order to find out what causes the segfaults I have enabled the following options:
inside /etc/apache2/apache2.conf
CoreDumpDirectory /tmp-apache/
$ ls -Fla / | grep tmp-apache
drwxrwxrwx 2 www-data www-data 4096 2009-01-31 11:01 tmp-apache/
I have changed the ulimit settings inside /etc/security/limits.conf
* soft core unlimited
* hard core unlimited
I have even added a ulimit -c unlimited setting inside /etc/init.d/apache2.
But still I get no core dumps inside /tmp-apache2/ from the segfaulting children.
If I manually kill -11 apache.pid then I can see a core file inside /tmp-apache/
I have only seen one or two core dumps generated by apache and using gdb I could see that they both “blamed” a function of /usr/lib/apache2/modules/libphp5.so. In my quest to find which site/code causes the segfaults I have recompiled apache2 to enable mod_whatkilledus. But no core dump was created in /tmp-apache/ for more than a week even if the segfaults keep happening.
I have reduced my modules, removed mod_python, mod_perl, etc and still these segfaults keep occuring but no core dumps. I suspect that the only time I got a core was when a parent and not a child process segfaulted. I don’t think that my apache2 children dump core when they segfault.
Is there anything I could have done and I haven’t done it ? Is there a way I can force apache2 children to dump core or any other way to determine what causes these segfaults ? All these without of course closing down the sites one by one to see when the segfaults stop…
Thanks in advance to anyone that replies!
P.S. blog’s database is making some tricks…I hope it’s ok now and the post is fully published
Filed by kargig at 22:41 under Linux
14 Comments | 17,148 views
Initial thought: run the same scripts without apache, ie run them directly through php, and see what happens?
Thanks for your reply, but these sites are mostly CMS, so it’s nearly impossible to check every single php file by hand and I can’t see a reason why this manual process would provide me with information on the apache segfaults either…
A segfault in mod_php5? this is very interesting. Can I have a look please? 🙂
If you somehow help me get those segfaults from apache2 children to dump core … 🙂
start httpd in single process mode under strace:
strace httpd -X > err.log 2>&1
the err.log will show you what’s being opened and the last hit to the server log should be close to your culprit as httpd will be dead when it seg faults.
Did you change your kernel just before segfaults started to occur ? Recent linux kernel have a new epoll limit which can cause apache to segfault when overloaded.
Nope, the machine is running stock debian etch kernel: 2.6.18-6-486.
same problem here , running gentoo hardened with php suhosin :
quaoar modules # php -v
PHP 5.2.8-pl2-gentoo (cli) (built: Mar 31 2009 01:34:12)
Copyright (c) 1997-2008 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2008 Zend Technologies
quaoar modules # apache2 -v
Server version: Apache/2.2.10 (Unix)
Server built: Mar 31 2009 01:03:51
this happens mostly when running drupal with ubercart modules and image handling.
Apr 15 13:12:55 quaoar grsec: From 1xx.2xx.1xx.xxx: signal 11 sent to /usr/sbin/apache2[apache2:23170] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:9200] uid/euid:0/0 gid/egid:0/0
Apr 15 13:12:55 quaoar grsec: From 1xx.2xx.1xx.xxx: denied resource overstep by requesting 8392704 for RLIMIT_STACK against limit 8388608 for /usr/sbin/apache2[apache2:23170] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:9200] uid/euid:0/0 gid/egid:0/0
Apr 15 13:12:55 quaoar apache2[23170]: segfault at 7b063d62afa0 ip 6cef6513b81d sp 7b063d62af60 error 6 in libphp5.so[6cef64dc4000+78d000]
Did we ever found out what was causing it ?
[…] Hello, Μετά από καιρό για 1η (η 2η) φορά με απογοήτευσε το update του apache2 (2.2.12-1). Πετούσε συνέχεια Segmentation fault (11) όπως εδώ. […]
A certain website with an old version of xoops. I am not sure which exact php file created the problem but I am glad that I got rid of the problem (with an upgrade to the xoops installation by the webmonster of the site)
same problem here, gentoo hardened / grsec
apache-2.2.11-r2 / php-5.2.10
the problem appears very often with phpmyadmin-2.11.9.5
could be suhosin related too
We ran into the same problem. I verified that it was PHP-related by disabling mod-php and waiting to see if the segfaults continued. They did not. So I re-enabled mod-php and it started again. Thanks to Kargig who pointed out that old php file was causing his problem. This got me looking at the server and I found quite a few old files out there hanging around. And since it’s a large server, I don’t have the luxury of trying to isolate the specific one. So I moved a lot of the old stuff out of the path of apache. So far, so good. Everything seems to be working ok for now. It’s been nearly half an hour since the last segfault error message.
Resolved by removing the Google mod_pagespeed module…
Regards,
HUjuice