Apache SIGABRT on Gentoo

By , last updated July 31, 2019

I’ve been trying to set up a PressBooks test site on my Gentoo dec box. I may or may not write a book about programming, but PressBooks is a WordPress plugin for writing and exporting books. I was curious and decided to give it a go.

All day I’ve been trying to export a book to get a feel for PressBooks. Both to get a feel for it, and to see if this is something worthwhile or if I should be looking at other solutions.

The SIGABRT

The problem I’ve been having was that each time I tried to export the book, in either format (It being PDF, mPDF, ODT, XML or whatever), I got an error from the browser saying it didn’t receive any data from the server.

This was the response from Chrome.

No data received

ERR_EMPTY_RESPONSE

It was a similar message with Firefox.

The connection was reset

The connection to the server was reset while the page was loading.

  • The site could be temporarily unavailable or too busy. Try again in a few moments.
  • If you are unable to load any pages, check your computer’s network connection.
  • If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the Web.

Whats happening on the server is that the Apache child process dies a painful death.

The errors

That this was a rather severe error was quite apparent by the no response errors from both Chrome and Firefox. The server logs for this domain where also empty, neither the domain_access_log or domain_error_log had any information about this issue. It was as it never happened.

Upon further inspection, the general server error_log (usually /var/log/apache2/error_log) was recently modified, and lo and behold, this error revealed itself in all its glory.

*** Error in `/usr/sbin/apache2': free(): invalid next size (fast): 0x00007f9041de1280 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x74893)[0x7f905525f893]
/lib64/libc.so.6(+0x79f56)[0x7f9055264f56]
/lib64/libc.so.6(+0x7b1e2)[0x7f90552661e2]
/usr/lib64/libmagic.so.1(+0x445d)[0x7f904ddb545d]
/usr/lib64/libmagic.so.1(magic_close+0x1f)[0x7f904ddb4510]
/usr/lib64/apache2/modules/libphp5.so(finfo_resource_destructor+0x33)[0x7f904ce87fef]
/usr/lib64/apache2/modules/libphp5.so(list_entry_destructor+0x8a)[0x7f904d2580f8]
/usr/lib64/apache2/modules/libphp5.so(zend_hash_del_key_or_index+0x205)[0x7f904d2554bc]
/usr/lib64/apache2/modules/libphp5.so(_zend_list_delete+0x6d)[0x7f904d257c3c]
/usr/lib64/apache2/modules/libphp5.so(zif_finfo_close+0xc1)[0x7f904ce888ec]
/usr/lib64/apache2/modules/libphp5.so(+0x62fc29)[0x7f904d289c29]
/usr/lib64/apache2/modules/libphp5.so(+0x630a68)[0x7f904d28aa68]
/usr/lib64/apache2/modules/libphp5.so(execute_ex+0x9ed)[0x7f904d2880f2]
/usr/lib64/apache2/modules/libphp5.so(zend_execute+0x9af)[0x7f904d288af8]
/usr/lib64/apache2/modules/libphp5.so(zend_call_function+0x115e)[0x7f904d22f119]
/usr/lib64/apache2/modules/libphp5.so(zif_call_user_func_array+0x9a)[0x7f904d0bc61d]
/usr/lib64/apache2/modules/libphp5.so(+0x62fc29)[0x7f904d289c29]
/usr/lib64/apache2/modules/libphp5.so(+0x63784c)[0x7f904d29184c]
/usr/lib64/apache2/modules/libphp5.so(execute_ex+0x9ed)[0x7f904d2880f2]
/usr/lib64/apache2/modules/libphp5.so(zend_execute+0x9af)[0x7f904d288af8]
/usr/lib64/apache2/modules/libphp5.so(zend_execute_scripts+0x269)[0x7f904d2448a0]
/usr/lib64/apache2/modules/libphp5.so(php_execute_script+0x43f)[0x7f904d1b5d9d]
/usr/lib64/apache2/modules/libphp5.so(+0x70c0f0)[0x7f904d3660f0]
/usr/sbin/apache2(ap_run_handler+0x61)[0x43f6f7]
/usr/sbin/apache2(ap_invoke_handler+0x179)[0x44005a]
/usr/sbin/apache2(ap_internal_redirect+0x69)[0x45171d]
/usr/lib64/apache2/modules/mod_rewrite.so(+0xdfac)[0x7f90504e2fac]
/usr/sbin/apache2(ap_run_handler+0x61)[0x43f6f7]
/usr/sbin/apache2(ap_invoke_handler+0x179)[0x44005a]
/usr/sbin/apache2(ap_process_request+0x6d)[0x450b6a]
/usr/sbin/apache2[0x44da53]
/usr/sbin/apache2(ap_run_process_connection+0x61)[0x448c9d]
/usr/sbin/apache2(ap_process_connection+0x67)[0x449133]
/usr/sbin/apache2[0x4580cc]
/usr/sbin/apache2[0x4583f0]
/usr/sbin/apache2[0x458673]
/usr/sbin/apache2(ap_mpm_run+0x4bb)[0x458b7c]
/usr/sbin/apache2(main+0xae1)[0x42565c]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f905520ca95]
/usr/sbin/apache2[0x424279]
======= Memory map: ========
00400000-0046e000 r-xp 00000000 09:7f 2768872                            /usr/sbin/apache2
0066d000-0066e000 r--p 0006d000 09:7f 2768872                            /usr/sbin/apache2
0066e000-00673000 rw-p 0006e000 09:7f 2768872                            /usr/sbin/apache2
00673000-00676000 rw-p 00000000 00:00 0
0100c000-012f4000 rw-p 00000000 00:00 0                                  [heap]
012f4000-015ca000 rw-p 00000000 00:00 0                                  [heap]
015ca000-015eb000 rw-p 00000000 00:00 0                                  [heap]
015eb000-01786000 rw-p 00000000 00:00 0                                  [heap]
7f9041da8000-7f904274e000 rw-p 00000000 00:00 0
7f904277e000-7f9042cd3000 rw-p 00000000 00:00 0
7f9042cd7000-7f9042d7a000 rw-p 00000000 00:00 0
7f9042d7a000-7f9042dc9000 rw-p 00000000 00:00 0
7f9042dc9000-7f90492f5000 r--p 00000000 09:7f 3147096                    /usr/lib64/locale/locale-archive
7f90492f5000-7f9049300000 r-xp 00000000 09:7f 2142984                    /lib64/libnss_files-2.20.so
7f9049300000-7f90494ff000 ---p 0000b000 09:7f 2142984                    /lib64/libnss_files-2.20.so
7f90494ff000-7f9049500000 r--p 0000a000 09:7f 2142984                    /lib64/libnss_files-2.20.so
7f9049500000-7f9049501000 rw-p 0000b000 09:7f 2142984                    /lib64/libnss_files-2.20.so
7f9049501000-7f904950b000 r-xp 00000000 09:7f 2142995                    /lib64/libnss_nis-2.20.so
7f904950b000-7f904970a000 ---p 0000a000 09:7f 2142995                    /lib64/libnss_nis-2.20.so
7f904970a000-7f904970b000 r--p 00009000 09:7f 2142995                    /lib64/libnss_nis-2.20.so
7f904970b000-7f904970c000 rw-p 0000a000 09:7f 2142995                    /lib64/libnss_nis-2.20.so
7f904970c000-7f9049713000 r-xp 00000000 09:7f 2142997                    /lib64/libnss_compat-2.20.so
7f9049713000-7f9049912000 ---p 00007000 09:7f 2142997                    /lib64/libnss_compat-2.20.so
7f9049912000-7f9049913000 r--p 00006000 09:7f 2142997                    /lib64/libnss_compat-2.20.so
7f9049913000-7f9049914000 rw-p 00007000 09:7f 2142997                    /lib64/libnss_compat-2.20.so
7f9049914000-7f9049918000 r-xp 00000000 09:7f 1876001                    /usr/lib64/libgpg-error.so.0.11.0
7f9049918000-7f9049b17000 ---p 00004000 09:7f 1876001                    /usr/lib64/libgpg-error.so.0.11.0
7f9049b17000-7f9049b18000 r--p 00003000 09:7f 1876001                    /usr/lib64/libgpg-error.so.0.11.0
7f9049b18000-7f9049b19000 rw-p 00004000 09:7f 1876001                    /usr/lib64/libgpg-error.so.0.11.0
7f9049b19000-7f9049bca000 r-xp 00000000 09:7f 1884412                    /usr/lib64/libgcrypt.so.11.8.3
7f9049bca000-7f9049dc9000 ---p 000b1000 09:7f 1884412                    /usr/lib64/libgcrypt.so.11.8.3
7f9049dc9000-7f9049dca000 r--p 000b0000 09:7f 1884412                    /usr/lib64/libgcrypt.so.11.8.3
7f9049dca000-7f9049dd1000 rw-p 000b1000 09:7f 1884412                    /usr/lib64/libgcrypt.so.11.8.3
7f9049dd1000-7f9049e1a000 r-xp 00000000 09:7f 1988967                    /usr/lib64/libxslt.so.1.1.28
7f9049e1a000-7f904a01a000 ---p 00049000 09:7f 1988967                    /usr/lib64/libxslt.so.1.1.28
7f904a01a000-7f904a01b000 r--p 00049000 09:7f 1988967                    /usr/lib64/libxslt.so.1.1.28
7f904a01b000-7f904a01c000 rw-p 0004a000 09:7f 1988967                    /usr/lib64/libxslt.so.1.1.28
7f904a01c000-7f904a01d000 rw-p 00000000 00:00 0
7f904a01d000-7f904a0d9000 r-xp 00000000 09:7f 2265646                    /usr/lib64/libnetsnmp.so.30.0.3
7f904a0d9000-7f904a2d8000 ---p 000bc000 09:7f 2265646                    /usr/lib64/libnetsnmp.so.30.0.3
7f904a2d8000-7f904a2d9000 r--p 000bb000 09:7f 2265646                    /usr/lib64/libnetsnmp.so.30.0.3
7f904a2d9000-7f904a2dd000 rw-p 000bc000 09:7f 2265646                    /usr/lib64/libnetsnmp.so.30.0.3
7f904a2dd000-7f904a313000 rw-p 00000000 00:00 0
7f904a313000-7f904a3ef000 r-xp 00000000 09:7f 2002357                    /usr/lib64/libfreetype.so.6.11.4
7f904a3ef000-7f904a5ef000 ---p 000dc000 09:7f 2002357                    /usr/lib64/libfreetype.so.6.11.4
7f904a5ef000-7f904a5f5000 r--p 000dc000 09:7f 2002357                    /usr/lib64/libfreetype.so.6.11.4
7f904a5f5000-7f904a5f6000 rw-p 000e2000 09:7f 2002357                    /usr/lib64/libfreetype.so.6.11.4
7f904a5f6000-7f904a7b0000 r-xp 00000000 09:7f 2502454                    /usr/lib64/libxml2.so.2.9.2
7f904a7b0000-7f904a9b0000 ---p 001ba000 09:7f 2502454                    /usr/lib64/libxml2.so.2.9.2
7f904a9b0000-7f904a9b7000 r--p 001ba000 09:7f 2502454                    /usr/lib64/libxml2.so.2.9.2
7f904a9b7000-7f904a9ba000 rw-p 001c1000 09:7f 2502454                    /usr/lib64/libxml2.so.2.9.2
7f904a9ba000-7f904a9bb000 rw-p 00000000 00:00 0
7f904a9bb000-7f904a9cf000 r-xp 00000000 09:7f 2142497                    /lib64/libnsl-2.20.so
7f904a9cf000-7f904abce000 ---p 00014000 09:7f 2142497                    /lib64/libnsl-2.20.so
7f904abce000-7f904abcf000 r--p 00013000 09:7f 2142497                    /lib64/libnsl-2.20.so
7f904abcf000-7f904abd0000 rw-p 00014000 09:7f 2142497                    /lib64/libnsl-2.20.so
7f904abd0000-7f904abd2000 rw-p 00000000 00:00 0
7f904abd2000-7f904abe9000 r-xp 00000000 09:7f 8680008                    /lib64/libbz2.so.1.0.6
7f904abe9000-7f904ade8000 ---p 00017000 09:7f 8680008                    /lib64/libbz2.so.1.0.6
7f904ade8000-7f904ade9000 r--p 00016000 09:7f 8680008                    /lib64/libbz2.so.1.0.6
7f904ade9000-7f904adea000 rw-p 00017000 09:7f 8680008                    /lib64/libbz2.so.1.0.6
7f904adea000-7f904ae7a000 r-xp 00000000 09:7f 2770534                    /usr/lib64/libcurl.so.4.3.0
7f904ae7a000-7f904b079000 ---p 00090000 09:7f 2770534                    /usr/lib64/libcurl.so.4.3.0
7f904b079000-7f904b07b000 r--p 0008f000 09:7f 2770534                    /usr/lib64/libcurl.so.4.3.0
7f904b07b000-7f904b07c000 rw-p 00091000 09:7f 2770534                    /usr/lib64/libcurl.so.4.3.0
7f904b07c000-7f904b0cd000 r-xp 00000000 09:7f 8827391                    /usr/lib64/libjpeg.so.8.4.0
7f904b0cd000-7f904b2cd000 ---p 00051000 09:7f 8827391                    /usr/lib64/libjpeg.so.8.4.0
7f904b2cd000-7f904b2ce000 r--p 00051000 09:7f 8827391                    /usr/lib64/libjpeg.so.8.4.0
7f904b2ce000-7f904b2cf000 rw-p 00052000 09:7f 8827391                    /usr/lib64/libjpeg.so.8.4.0
7f904b2cf000-7f904b310000 r-xp 00000000 09:7f 1985322                    /usr/lib64/libpng16.so.16.16.0
7f904b310000-7f904b50f000 ---p 00041000 09:7f 1985322                    /usr/lib64/libpng16.so.16.16.0
7f904b50f000-7f904b510000 r--p 00040000 09:7f 1985322                    /usr/lib64/libpng16.so.16.16.0
7f904b510000-7f904b511000 rw-p 00041000 09:7f 1985322                    /usr/lib64/libpng16.so.16.16.0
7f904b511000-7f904b56f000 r-xp 00000000 09:7f 2262191                    /usr/lib64/libt1.so.5.1.2
7f904b56f000-7f904b76e000 ---p 0005e000 09:7f 2262191                    /usr/lib64/libt1.so.5.1.2
7f904b76e000-7f904b76f000 r--p 0005d000 09:7f 2262191                    /usr/lib64/libt1.so.5.1.2
7f904b76f000-7f904b773000 rw-p 0005e000 09:7f 2262191                    /usr/lib64/libt1.so.5.1.2
7f904b773000-7f904b789000 rw-p 00000000 00:00 0
7f904b789000-7f904b798000 r-xp 00000000 09:7f 2659207                    /lib64/libpam.so.0.84.1
7f904b798000-7f904b997000 ---p 0000f000 09:7f 2659207                    /lib64/libpam.so.0.84.1
7f904b997000-7f904b998000 r--p 0000e000 09:7f 2659207                    /lib64/libpam.so.0.84.1
7f904b998000-7f904b999000 rw-p 0000f000 09:7f 2659207                    /lib64/libpam.so.0.84.1
7f904b999000-7f904ba0e000 r-xp 00000000 09:7f 1985592                    /usr/lib64/libonig.so.2.0.0
7f904ba0e000-7f904bc0d000 ---p 00075000 09:7f 1985592                    /usr/lib64/libonig.so.2.0.0
7f904bc0d000-7f904bc0e000 r--p 00074000 09:7f 1985592                    /usr/lib64/libonig.so.2.0.0
7f904bc0e000-7f904bc11000 rw-p 00075000 09:7f 1985592                    /usr/lib64/libonig.so.2.0.0
7f904bc11000-7f904bc1b000 r-xp 00000000 09:7f 1984466                    /usr/lib64/libltdl.so.7.3.1
7f904bc1b000-7f904be1a000 ---p 0000a000 09:7f 1984466                    /usr/lib64/libltdl.so.7.3.1
7f904be1a000-7f904be1b000 r--p 00009000 09:7f 1984466                    /usr/lib64/libltdl.so.7.3.1
7f904be1b000-7f904be1c000 rw-p 0000a000 09:7f 1984466                    /usr/lib64/libltdl.so.7.3.1
7f904be1c000-7f904be46000 r-xp 00000000 09:7f 3029427                    /usr/lib64/libmcrypt.so.4.4.8
7f904be46000-7f904c046000 ---p 0002a000 09:7f 3029427                    /usr/lib64/libmcrypt.so.4.4.8
7f904c046000-7f904c048000 r--p 0002a000 09:7f 3029427                    /usr/lib64/libmcrypt.so.4.4.8
7f904c048000-7f904c04a000 rw-p 0002c000 09:7f 3029427                    /usr/lib64/libmcrypt.so.4.4.8
7f904c04a000-7f904c04f000 rw-p 00000000 00:00 0
7f904c04f000-7f904c0a8000 r-xp 00000000 09:7f 2008562                    /lib64/libncurses.so.5.9
7f904c0a8000-7f904c2a8000 ---p 00059000 09:7f 2008562                    /lib64/libncurses.so.5.9
7f904c2a8000-7f904c2ac000 r--p 00059000 09:7f 2008562                    /lib64/libncurses.so.5.9
7f904c2ac000-7f904c2ad000 rw-p 0005d000 09:7f 2008562                    /lib64/libncurses.so.5.9
7f904c2ad000-7f904c2ae000 rw-p 00000000 00:00 0
7f904c2ae000-7f904c2eb000 r-xp 00000000 09:7f 2149939                    /lib64/libreadline.so.6.3
7f904c2eb000-7f904c4ea000 ---p 0003d000 09:7f 2149939                    /lib64/libreadline.so.6.3
7f904c4ea000-7f904c4ec000 r--p 0003c000 09:7f 2149939                    /lib64/libreadline.so.6.3
7f904c4ec000-7f904c4f2000 rw-p 0003e000 09:7f 2149939                    /lib64/libreadline.so.6.3
7f904c4f2000-7f904c4f4000 rw-p 00000000 00:00 0
7f904c4f4000-7f904c507000 r-xp 00000000 09:7f 2142975                    /lib64/libresolv-2.20.so
7f904c507000-7f904c707000 ---p 00013000 09:7f 2142975                    /lib64/libresolv-2.20.so
7f904c707000-7f904c708000 r--p 00013000 09:7f 2142975                    /lib64/libresolv-2.20.so
7f904c708000-7f904c709000 rw-p 00014000 09:7f 2142975                    /lib64/libresolv-2.20.so
7f904c709000-7f904c70b000 rw-p 00000000 00:00 0
7f904c70b000-7f904c724000 r-xp 00000000 09:7f 1988965                    /usr/lib64/libexslt.so.0.8.17
7f904c724000-7f904c923000 ---p 00019000 09:7f 1988965                    /usr/lib64/libexslt.so.0.8.17
7f904c923000-7f904c924000 r--p 00018000 09:7f 1988965                    /usr/lib64/libexslt.so.0.8.17
7f904c924000-7f904c925000 rw-p 00019000 09:7f 1988965                    /usr/lib64/libexslt.so.0.8.17
7f904c925000-7f904ca47000 r-xp 00000000 09:7f 3029250                    /usr/lib64/libc-client.so.1.0.0
7f904ca47000-7f904cc47000 ---p 00122000 09:7f 3029250                    /usr/lib64/libc-client.so.1.0.0
7f904cc47000-7f904cc49000 r--p 00122000 09:7f 3029250                    /usr/lib64/libc-client.so.1.0.0
7f904cc49000-7f904cc59000 rw-p 00124000 09:7f 3029250                    /usr/lib64/libc-client.so.1.0.0
7f904cc59000-7f904cc5a000 rw-p 00000000 00:00 0
7f904cc5a000-7f904d8b8000 r-xp 00000000 09:7f 2413284                    /usr/lib64/php5.6/apache2/libphp5.so
7f904d8b8000-7f904dab7000 ---p 00c5e000 09:7f 2413284                    /usr/lib64/php5.6/apache2/libphp5.so
7f904dab7000-7f904db72000 r--p 00c5d000 09:7f 2413284                    /usr/lib64/php5.6/apache2/libphp5.so
7f904db72000-7f904db88000 rw-p 00d18000 09:7f 2413284                    /usr/lib64/php5.6/apache2/libphp5.so
7f904db88000-7f904dbab000 rw-p 00000000 00:00 0
7f904dbab000-7f904dbaf000 r-xp 00000000 09:7f 2647290                    /usr/lib64/apache2/modules/mod_authz_svn.so
7f904dbaf000-7f904ddaf000 ---p 00004000 09:7f 2647290                    /usr/lib64/apache2/modules/mod_authz_svn.so
7f904ddaf000-7f904ddb0000 r--p 00004000 09:7f 2647290                    /usr/lib64/apache2/modules/mod_authz_svn.so
7f904ddb0000-7f904ddb1000 rw-p 00005000 09:7f 2647290                    /usr/lib64/apache2/modules/mod_authz_svn.so
7f904ddb1000-7f904ddd6000 r-xp 00000000 09:7f 2123705                    /usr/lib64/libmagic.so.1.0.0
7f904ddd6000-7f904dfd6000 ---p 00025000 09:7f 2123705                    /usr/lib64/libmagic.so.1.0.0
7f904dfd6000-7f904dfd7000 r--p 00025000 09:7f 2123705                    /usr/lib64/libmagic.so.1.0.0
7f904dfd7000-7f904dfd8000 rw-p 00026000 09:7f 2123705                    /usr/lib64/libmagic.so.1.0.0
7f904dfd8000-7f904e0c7000 r-xp 00000000 09:7f 2412748                    /usr/lib64/libsqlite3.so.0.8.6
7f904e0c7000-7f904e2c7000 ---p 000ef000 09:7f 2412748                    /usr/lib64/libsqlite3.so.0.8.6
7f904e2c7000-7f904e2c9000 r--p 000ef000 09:7f 2412748                    /usr/lib64/libsqlite3.so.0.8.6
7f904e2c9000-7f904e2cc000 rw-p 000f1000 09:7f 2412748                    /usr/lib64/libsqlite3.so.0.8.6
7f904e2cc000-7f904e350000 r-xp 00000000 09:7f 2646869                    /usr/lib64/libsvn_subr-1.so.0.0.0
7f904e350000-7f904e550000 ---p 00084000 09:7f 2646869                    /usr/lib64/libsvn_subr-1.so.0.0.0
7f904e550000-7f904e553000 r--p 00084000 09:7f 2646869                    /usr/lib64/libsvn_subr-1.so.0.0.0
7f904e553000-7f904e555000 rw-p 00087000 09:7f 2646869                    /usr/lib64/libsvn_subr-1.so.0.0.0
7f904e555000-7f904e56a000 r-xp 00000000 09:7f 2646845                    /usr/lib64/libsvn_delta-1.so.0.0.0
7f904e56a000-7f904e769000 ---p 00015000 09:7f 2646845                    /usr/lib64/libsvn_delta-1.so.0.0.0
7f904e769000-7f904e76a000 r--p 00014000 09:7f 2646845                    /usr/lib64/libsvn_delta-1.so.0.0.0
7f904e76a000-7f904e76b000 rw-p 00015000 09:7f 2646845                    /usr/lib64/libsvn_delta-1.so.0.0.0
7f904e76b000-7f904e76d000 r-xp 00000000 09:7f 2646823                    /usr/lib64/libsvn_fs_util-1.so.0.0.0
7f904e76d000-7f904e96c000 ---p 00002000 09:7f 2646823                    /usr/lib64/libsvn_fs_util-1.so.0.0.0
7f904e96c000-7f904e96d000 r--p 00001000 09:7f 2646823                    /usr/lib64/libsvn_fs_util-1.so.0.0.0
7f904e96d000-7f904e96e000 rw-p 00002000 09:7f 2646823                    /usr/lib64/libsvn_fs_util-1.so.0.0.0
7f904e96e000-7f904e979000 r-xp 00000000 09:7f 2646698                    /usr/lib64/libsvn_fs-1.so.0.0.0
7f904e979000-7f904eb78000 ---p 0000b000 09:7f 2646698                    /usr/lib64/libsvn_fs-1.so.0.0.0
7f904eb78000-7f904eb79000 r--p 0000a000 09:7f 2646698                    /usr/lib64/libsvn_fs-1.so.0.0.0
7f904eb79000-7f904eb7a000 rw-p 0000b000 09:7f 2646698                    /usr/lib64/libsvn_fs-1.so.0.0.0
7f904eb7a000-7f904ebb5000 r-xp 00000000 09:7f 2646888                    /usr/lib64/libsvn_repos-1.so.0.0.0
7f904ebb5000-7f904edb4000 ---p 0003b000 09:7f 2646888                    /usr/lib64/libsvn_repos-1.so.0.0.0
7f904edb4000-7f904edb5000 r--p 0003a000 09:7f 2646888                    /usr/lib64/libsvn_repos-1.so.0.0.0
7f904edb5000-7f904edb6000 rw-p 0003b000 09:7f 2646888                    /usr/lib64/libsvn_repos-1.so.0.0.0
7f904edb6000-7f904eded000 r-xp 00000000 09:7f 2647273                    /usr/lib64/apache2/modules/mod_dav_svn.so
7f904eded000-7f904efed000 ---p 00037000 09:7f 2647273                    /usr/lib64/apache2/modules/mod_dav_svn.so
7f904efed000-7f904efef000 r--p 00037000 09:7f 2647273                    /usr/lib64/apache2/modules/mod_dav_svn.so
7f904efef000-7f904eff0000 rw-p 00039000 09:7f 2647273                    /usr/lib64/apache2/modules/mod_dav_svn.so
7f904eff0000-7f904eff2000 r-xp 00000000 09:7f 2768864                    /usr/lib64/apache2/modules/mod_vhost_alias.so
7f904eff2000-7f904f1f1000 ---p 00002000 09:7f 2768864                    /usr/lib64/apache2/modules/mod_vhost_alias.so
7f904f1f1000-7f904f1f2000 r--p 00001000 09:7f 2768864                    /usr/lib64/apache2/modules/mod_vhost_alias.so
7f904f1f2000-7f904f1f3000 rw-p 00002000 09:7f 2768864                    /usr/lib64/apache2/modules/mod_vhost_alias.so
7f904f1f3000-7f904f1f6000 r-xp 00000000 09:7f 2768851                    /usr/lib64/apache2/modules/mod_usertrack.so
7f904f1f6000-7f904f3f5000 ---p 00003000 09:7f 2768851                    /usr/lib64/apache2/modules/mod_usertrack.so
7f904f3f5000-7f904f3f6000 r--p 00002000 09:7f 2768851                    /usr/lib64/apache2/modules/mod_usertrack.so
7f904f3f6000-7f904f3f7000 rw-p 00003000 09:7f 2768851                    /usr/lib64/apache2/modules/mod_usertrack.so
7f904f3f7000-7f904f3f9000 r-xp 00000000 09:7f 2768869                    /usr/lib64/apache2/modules/mod_userdir.so
7f904f3f9000-7f904f5f8000 ---p 00002000 09:7f 2768869                    /usr/lib64/apache2/modules/mod_userdir.so
7f904f5f8000-7f904f5f9000 r--p 00001000 09:7f 2768869                    /usr/lib64/apache2/modules/mod_userdir.so
7f904f5f9000-7f904f5fa000 rw-p 00002000 09:7f 2768869                    /usr/lib64/apache2/modules/mod_userdir.so
7f904f5fa000-7f904f5fc000 r-xp 00000000 09:7f 2768852                    /usr/lib64/apache2/modules/mod_unique_id.so
7f904f5fc000-7f904f7fb000 ---p 00002000 09:7f 2768852                    /usr/lib64/apache2/modules/mod_unique_id.so
7f904f7fb000-7f904f7fc000 r--p 00001000 09:7f 2768852                    /usr/lib64/apache2/modules/mod_unique_id.so
7f904f7fc000-7f904f7fd000 rw-p 00002000 09:7f 2768852                    /usr/lib64/apache2/modules/mod_unique_id.so
7f904f7fd000-7f904f9ed000 r-xp 00000000 09:7f 2772841                    /usr/lib64/libcrypto.so.1.0.0
7f904f9ed000-7f904fbed000 ---p 001f0000 09:7f 2772841                    /usr/lib64/libcrypto.so.1.0.0
7f904fbed000-7f904fc07000 r--p 001f0000 09:7f 2772841                    /usr/lib64/libcrypto.so.1.0.0
7f904fc07000-7f904fc13000 rw-p 0020a000 09:7f 2772841                    /usr/lib64/libcrypto.so.1.0.0
7f904fc13000-7f904fc16000 rw-p 00000000 00:00 0
7f904fc16000-7f904fc8d000 r-xp 00000000 09:7f 2772845                    /usr/lib64/libssl.so.1.0.0
7f904fc8d000-7f904fe8d000 ---p 00077000 09:7f 2772845                    /usr/lib64/libssl.so.1.0.0
7f904fe8d000-7f904fe91000 r--p 00077000 09:7f 2772845                    /usr/lib64/libssl.so.1.0.0
7f904fe91000-7f904fe98000 rw-p 0007b000 09:7f 2772845                    /usr/lib64/libssl.so.1.0.0
7f904fe98000-7f904fec9000 r-xp 00000000 09:7f 2768854                    /usr/lib64/apache2/modules/mod_ssl.so
7f904fec9000-7f90500c8000 ---p 00031000 09:7f 2768854                    /usr/lib64/apache2/modules/mod_ssl.so
7f90500c8000-7f90500c9000 r--p 00030000 09:7f 2768854                    /usr/lib64/apache2/modules/mod_ssl.so
7f90500c9000-7f90500cb000 rw-p 00031000 09:7f 2768854                    /usr/lib64/apache2/modules/mod_ssl.so
7f90500cb000-7f90500cd000 rw-p 00000000 00:00 0
7f90500cd000-7f90500d0000 r-xp 00000000 09:7f 2768868                    /usr/lib64/apache2/modules/mod_speling.so
7f90500d0000-7f90502cf000 ---p 00003000 09:7f 2768868                    /usr/lib64/apache2/modules/mod_speling.so
7f90502cf000-7f90502d0000 r--p 00002000 09:7f 2768868                    /usr/lib64/apache2/modules/mod_speling.so
7f90502d0000-7f90502d1000 rw-p 00003000 09:7f 2768868                    /usr/lib64/apache2/modules/mod_speling.so
7f90502d1000-7f90502d4000 r-xp 00000000 09:7f 2768853                    /usr/lib64/apache2/modules/mod_setenvif.so
7f90502d4000-7f90504d3000 ---p 00003000 09:7f 2768853                    /usr/lib64/apache2/modules/mod_setenvif.so
7f90504d3000-7f90504d4000 r--p 00002000 09:7f 2768853                    /usr/lib64/apache2/modules/mod_setenvif.so
7f90504d4000-7f90504d5000 rw-p 00003000 09:7f 2768853                    /usr/lib64/apache2/modules/mod_setenvif.so
7f90504d5000-7f90504e6000 r-xp 00000000 09:7f 2768871                    /usr/lib64/apache2/modules/mod_rewrite.so
7f90504e6000-7f90506e5000 ---p 00011000 09:7f 2768871                    /usr/lib64/apache2/modules/mod_rewrite.so
7f90506e5000-7f90506e6000 r--p 00010000 09:7f 2768871                    /usr/lib64/apache2/modules/mod_rewrite.so
7f90506e6000-7f90506e7000 rw-p 00011000 09:7f 2768871                    /usr/lib64/apache2/modules/mod_rewrite.so
7f90506e7000-7f90506f0000 r-xp 00000000 09:7f 2768865                    /usr/lib64/apache2/modules/mod_negotiation.so
7f90506f0000-7f90508ef000 ---p 00009000 09:7f 2768865                    /usr/lib64/apache2/modules/mod_negotiation.so
7f90508ef000-7f90508f0000 r--p 00008000 09:7f 2768865                    /usr/lib64/apache2/modules/mod_negotiation.so
7f90508f0000-7f90508f1000 rw-p 00009000 09:7f 2768865                    /usr/lib64/apache2/modules/mod_negotiation.so
7f90508f1000-7f90508f7000 r-xp 00000000 09:7f 2768848                    /usr/lib64/apache2/modules/mod_mime_magic.so
7f90508f7000-7f9050af7000 ---p 00006000 09:7f 2768848                    /usr/lib64/apache2/modules/mod_mime_magic.so
7f9050af7000-7f9050af8000 r--p 00006000 09:7f 2768848                    /usr/lib64/apache2/modules/mod_mime_magic.so
7f9050af8000-7f9050af9000 rw-p 00007000 09:7f 2768848                    /usr/lib64/apache2/modules/mod_mime_magic.so
7f9050af9000-7f9050afd000 r-xp 00000000 09:7f 2768855                    /usr/lib64/apache2/modules/mod_mime.so
7f9050afd000-7f9050cfd000 ---p 00004000 09:7f 2768855                    /usr/lib64/apache2/modules/mod_mime.so
7f9050cfd000-7f9050cfe000 r--p 00004000 09:7f 2768855                    /usr/lib64/apache2/modules/mod_mime.so
7f9050cfe000-7f9050cff000 rw-p 00005000 09:7f 2768855                    /usr/lib64/apache2/modules/mod_mime.so
7f9050cff000-7f9050d01000 r-xp 00000000 09:7f 2768846                    /usr/lib64/apache2/modules/mod_logio.so
7f9050d01000-7f9050f00000 ---p 00002000 09:7f 2768846                    /usr/lib64/apache2/modules/mod_logio.so
7f9050f00000-7f9050f01000 r--p 00001000 09:7f 2768846                    /usr/lib64/apache2/modules/mod_logio.so
7f9050f01000-7f9050f02000 rw-p 00002000 09:7f 2768846                    /usr/lib64/apache2/modules/mod_logio.so
7f9050f02000-7f9050f08000 r-xp 00000000 09:7f 2768845                    /usr/lib64/apache2/modules/mod_log_config.so
7f9050f08000-7f9051107000 ---p 00006000 09:7f 2768845                    /usr/lib64/apache2/modules/mod_log_config.so
7f9051107000-7f9051108000 r--p 00005000 09:7f 2768845                    /usr/lib64/apache2/modules/mod_log_config.so
7f9051108000-7f9051109000 rw-p 00006000 09:7f 2768845                    /usr/lib64/apache2/modules/mod_log_config.so
7f9051109000-7f905110e000 r-xp 00000000 09:7f 2768859                    /usr/lib64/apache2/modules/mod_info.so
7f905110e000-7f905130d000 ---p 00005000 09:7f 2768859                    /usr/lib64/apache2/modules/mod_info.so
7f905130d000-7f905130e000 r--p 00004000 09:7f 2768859                    /usr/lib64/apache2/modules/mod_info.so
7f905130e000-7f905130f000 rw-p 00005000 09:7f 2768859                    /usr/lib64/apache2/modules/mod_info.so
7f905130f000-7f905131c000 r-xp 00000000 09:7f 2768842                    /usr/lib64/apache2/modules/mod_include.so
7f905131c000-7f905151b000 ---p 0000d000 09:7f 2768842                    /usr/lib64/apache2/modules/mod_include.so
7f905151b000-7f905151c000 r--p 0000c000 09:7f 2768842                    /usr/lib64/apache2/modules/mod_include.so
7f905151c000-7f905151d000 rw-p 0000d000 09:7f 2768842                    /usr/lib64/apache2/modules/mod_include.so
7f905151d000-7f9051521000 r-xp 00000000 09:7f 2768850                    /usr/lib64/apache2/modules/mod_headers.so
7f9051521000-7f9051720000 ---p 00004000 09:7f 2768850                    /usr/lib64/apache2/modules/mod_headers.so
7f9051720000-7f9051721000 r--p 00003000 09:7f 2768850                    /usr/lib64/apache2/modules/mod_headers.so
7f9051721000-7f9051722000 rw-p 00004000 09:7f 2768850                    /usr/lib64/apache2/modules/mod_headers.so
7f9051722000-7f9051726000 r-xp 00000000 09:7f 2768843                    /usr/lib64/apache2/modules/mod_filter.so
7f9051726000-7f9051925000 ---p 00004000 09:7f 2768843                    /usr/lib64/apache2/modules/mod_filter.so
7f9051925000-7f9051926000 r--p 00003000 09:7f 2768843                    /usr/lib64/apache2/modules/mod_filter.so
7f9051926000-7f9051927000 rw-p 00004000 09:7f 2768843                    /usr/lib64/apache2/modules/mod_filter.so
7f9051927000-7f905192c000 r-xp 00000000 09:7f 2768841                    /usr/lib64/apache2/modules/mod_ext_filter.so
7f905192c000-7f9051b2b000 ---p 00005000 09:7f 2768841                    /usr/lib64/apache2/modules/mod_ext_filter.so
7f9051b2b000-7f9051b2c000 r--p 00004000 09:7f 2768841                    /usr/lib64/apache2/modules/mod_ext_filter.so
7f9051b2c000-7f9051b2d000 rw-p 00005000 09:7f 2768841                    /usr/lib64/apache2/modules/mod_ext_filter.so
7f9051b2d000-7f9051b30000 r-xp 00000000 09:7f 2768849                    /usr/lib64/apache2/modules/mod_expires.so
7f9051b30000-7f9051d2f000 ---p 00003000 09:7f 2768849                    /usr/lib64/apache2/modules/mod_expires.so
7f9051d2f000-7f9051d30000 r--p 00002000 09:7f 2768849                    /usr/lib64/apache2/modules/mod_expires.so
7f9051d30000-7f9051d31000 rw-p 00003000 09:7f 2768849                    /usr/lib64/apache2/modules/mod_expires.so
7f9051d31000-7f9051d33000 r-xp 00000000 09:7f 2768847                    /usr/lib64/apache2/modules/mod_env.so
7f9051d33000-7f9051f32000 ---p 00002000 09:7f 2768847                    /usr/lib64/apache2/modules/mod_env.so
7f9051f32000-7f9051f33000 r--p 00001000 09:7f 2768847                    /usr/lib64/apache2/modules/mod_env.so
7f9051f33000-7f9051f34000 rw-p 00002000 09:7f 2768847                    /usr/lib64/apache2/modules/mod_env.so
7f9051f34000-7f9051f36000 r-xp 00000000 09:7f 2768866                    /usr/lib64/apache2/modules/mod_dir.so
7f9051f36000-7f9052135000 ---p 00002000 09:7f 2768866                    /usr/lib64/apache2/modules/mod_dir.so
7f9052135000-7f9052136000 r--p 00001000 09:7f 2768866                    /usr/lib64/apache2/modules/mod_dir.so
7f9052136000-7f9052137000 rw-p 00002000 09:7f 2768866                    /usr/lib64/apache2/modules/mod_dir.so
7f9052137000-7f9052153000 r-xp 00000000 09:7f 1048719                    /lib64/libz.so.1.2.8
7f9052153000-7f9052352000 ---p 0001c000 09:7f 1048719                    /lib64/libz.so.1.2.8
7f9052352000-7f9052353000 r--p 0001b000 09:7f 1048719                    /lib64/libz.so.1.2.8
7f9052353000-7f9052354000 rw-p 0001c000 09:7f 1048719                    /lib64/libz.so.1.2.8
7f9052354000-7f905235c000 r-xp 00000000 09:7f 2768844                    /usr/lib64/apache2/modules/mod_deflate.so
7f905235c000-7f905255b000 ---p 00008000 09:7f 2768844                    /usr/lib64/apache2/modules/mod_deflate.so
7f905255b000-7f905255c000 r--p 00007000 09:7f 2768844                    /usr/lib64/apache2/modules/mod_deflate.so
7f905255c000-7f905255d000 rw-p 00008000 09:7f 2768844                    /usr/lib64/apache2/modules/mod_deflate.so
7f905255d000-7f9052561000 r-xp 00000000 09:7f 2768863                    /usr/lib64/apache2/modules/mod_dav_lock.so
7f9052561000-7f9052760000 ---p 00004000 09:7f 2768863                    /usr/lib64/apache2/modules/mod_dav_lock.so
7f9052760000-7f9052761000 r--p 00003000 09:7f 2768863                    /usr/lib64/apache2/modules/mod_dav_lock.so
7f9052761000-7f9052762000 rw-p 00004000 09:7f 2768863                    /usr/lib64/apache2/modules/mod_dav_lock.so
7f9052762000-7f905276f000 r-xp 00000000 09:7f 2768862                    /usr/lib64/apache2/modules/mod_dav_fs.so
7f905276f000-7f905296e000 ---p 0000d000 09:7f 2768862                    /usr/lib64/apache2/modules/mod_dav_fs.so
7f905296e000-7f905296f000 r--p 0000c000 09:7f 2768862                    /usr/lib64/apache2/modules/mod_dav_fs.so
7f905296f000-7f9052970000 rw-p 0000d000 09:7f 2768862                    /usr/lib64/apache2/modules/mod_dav_fs.so
7f9052970000-7f905298a000 r-xp 00000000 09:7f 2768856                    /usr/lib64/apache2/modules/mod_dav.so
7f905298a000-7f9052b89000 ---p 0001a000 09:7f 2768856                    /usr/lib64/apache2/modules/mod_dav.so
7f9052b89000-7f9052b8a000 r--p 00019000 09:7f 2768856                    /usr/lib64/apache2/modules/mod_dav.so
7f9052b8a000-7f9052b8b000 rw-p 0001a000 09:7f 2768856                    /usr/lib64/apache2/modules/mod_dav.so
7f9052b8b000-7f9052b94000 r-xp 00000000 09:7f 2768861                    /usr/lib64/apache2/modules/mod_cgid.so
7f9052b94000-7f9052d93000 ---p 00009000 09:7f 2768861                    /usr/lib64/apache2/modules/mod_cgid.so
7f9052d93000-7f9052d94000 r--p 00008000 09:7f 2768861                    /usr/lib64/apache2/modules/mod_cgid.so
7f9052d94000-7f9052d95000 rw-p 00009000 09:7f 2768861                    /usr/lib64/apache2/modules/mod_cgid.so
7f9052d95000-7f9052d9b000 r-xp 00000000 09:7f 2768860                    /usr/lib64/apache2/modules/mod_cgi.so
7f9052d9b000-7f9052f9b000 ---p 00006000 09:7f 2768860                    /usr/lib64/apache2/modules/mod_cgi.so
7f9052f9b000-7f9052f9c000 r--p 00006000 09:7f 2768860                    /usr/lib64/apache2/modules/mod_cgi.so
7f9052f9c000-7f9052f9d000 rw-p 00007000 09:7f 2768860                    /usr/lib64/apache2/modules/mod_cgi.so
7f9052f9d000-7f9052fa6000 r-xp 00000000 09:7f 2768858                    /usr/lib64/apache2/modules/mod_autoindex.so
7f9052fa6000-7f90531a5000 ---p 00009000 09:7f 2768858                    /usr/lib64/apache2/modules/mod_autoindex.so
7f90531a5000-7f90531a6000 r--p 00008000 09:7f 2768858                    /usr/lib64/apache2/modules/mod_autoindex.so
7f90531a6000-7f90531a7000 rw-p 00009000 09:7f 2768858                    /usr/lib64/apache2/modules/mod_autoindex.so
7f90531a7000-7f90531a8000 r-xp 00000000 09:7f 2768832                    /usr/lib64/apache2/modules/mod_authz_user.so
7f90531a8000-7f90533a8000 ---p 00001000 09:7f 2768832                    /usr/lib64/apache2/modules/mod_authz_user.so
7f90533a8000-7f90533a9000 r--p 00001000 09:7f 2768832                    /usr/lib64/apache2/modules/mod_authz_user.so
7f90533a9000-7f90533aa000 rw-p 00002000 09:7f 2768832                    /usr/lib64/apache2/modules/mod_authz_user.so
7f90533aa000-7f90533ac000 r-xp 00000000 09:7f 2768834                    /usr/lib64/apache2/modules/mod_authz_owner.so
7f90533ac000-7f90535ab000 ---p 00002000 09:7f 2768834                    /usr/lib64/apache2/modules/mod_authz_owner.so
7f90535ab000-7f90535ac000 r--p 00001000 09:7f 2768834                    /usr/lib64/apache2/modules/mod_authz_owner.so
7f90535ac000-7f90535ad000 rw-p 00002000 09:7f 2768834                    /usr/lib64/apache2/modules/mod_authz_owner.so
7f90535ad000-7f90535af000 r-xp 00000000 09:7f 2768830                    /usr/lib64/apache2/modules/mod_authz_host.so
7f90535af000-7f90537ae000 ---p 00002000 09:7f 2768830                    /usr/lib64/apache2/modules/mod_authz_host.so
7f90537ae000-7f90537af000 r--p 00001000 09:7f 2768830                    /usr/lib64/apache2/modules/mod_authz_host.so
7f90537af000-7f90537b0000 rw-p 00002000 09:7f 2768830                    /usr/lib64/apache2/modules/mod_authz_host.so
7f90537b0000-7f90537b2000 r-xp 00000000 09:7f 2768831                    /usr/lib64/apache2/modules/mod_authz_groupfile.so
7f90537b2000-7f90539b1000 ---p 00002000 09:7f 2768831                    /usr/lib64/apache2/modules/mod_authz_groupfile.so
7f90539b1000-7f90539b2000 r--p 00001000 09:7f 2768831                    /usr/lib64/apache2/modules/mod_authz_groupfile.so
7f90539b2000-7f90539b3000 rw-p 00002000 09:7f 2768831                    /usr/lib64/apache2/modules/mod_authz_groupfile.so
7f90539b3000-7f90539b4000 r-xp 00000000 09:7f 2768835                    /usr/lib64/apache2/modules/mod_authz_default.so
7f90539b4000-7f9053bb3000 ---p 00001000 09:7f 2768835                    /usr/lib64/apache2/modules/mod_authz_default.so
7f9053bb3000-7f9053bb4000 r--p 00000000 09:7f 2768835                    /usr/lib64/apache2/modules/mod_authz_default.so
7f9053bb4000-7f9053bb5000 rw-p 00001000 09:7f 2768835                    /usr/lib64/apache2/modules/mod_authz_default.so
7f9053bb5000-7f9053bb7000 r-xp 00000000 09:7f 2768833                    /usr/lib64/apache2/modules/mod_authz_dbm.so
7f9053bb7000-7f9053db6000 ---p 00002000 09:7f 2768833                    /usr/lib64/apache2/modules/mod_authz_dbm.so
7f9053db6000-7f9053db7000 r--p 00001000 09:7f 2768833                    /usr/lib64/apache2/modules/mod_authz_dbm.so
7f9053db7000-7f9053db8000 rw-p 00002000 09:7f 2768833                    /usr/lib64/apache2/modules/mod_authz_dbm.so
7f9053db8000-7f9053dba000 r-xp 00000000 09:7f 2782221                    /usr/lib64/apache2/modules/mod_authn_file.so
7f9053dba000-7f9053fb9000 ---p 00002000 09:7f 2782221                    /usr/lib64/apache2/modules/mod_authn_file.so
7f9053fb9000-7f9053fba000 r--p 00001000 09:7f 2782221                    /usr/lib64/apache2/modules/mod_authn_file.so
7f9053fba000-7f9053fbb000 rw-p 00002000 09:7f 2782221                    /usr/lib64/apache2/modules/mod_authn_file.so
7f9053fbb000-7f9053fbc000 r-xp 00000000 09:7f 2768828                    /usr/lib64/apache2/modules/mod_authn_default.so
7f9053fbc000-7f90541bb000 ---p 00001000 09:7f 2768828                    /usr/lib64/apache2/modules/mod_authn_default.so
7f90541bb000-7f90541bc000 r--p 00000000 09:7f 2768828                    /usr/lib64/apache2/modules/mod_authn_default.so
7f90541bc000-7f90541bd000 rw-p 00001000 09:7f 2768828                    /usr/lib64/apache2/modules/mod_authn_default.so
7f90541bd000-7f90541bf000 r-xp 00000000 09:7f 2782222                    /usr/lib64/apache2/modules/mod_authn_dbm.so
7f90541bf000-7f90543be000 ---p 00002000 09:7f 2782222                    /usr/lib64/apache2/modules/mod_authn_dbm.so
7f90543be000-7f90543bf000 r--p 00001000 09:7f 2782222                    /usr/lib64/apache2/modules/mod_authn_dbm.so
7f90543bf000-7f90543c0000 rw-p 00002000 09:7f 2782222                    /usr/lib64/apache2/modules/mod_authn_dbm.so
7f90543c0000-7f90543c2000 r-xp 00000000 09:7f 2782223                    /usr/lib64/apache2/modules/mod_authn_anon.so
7f90543c2000-7f90545c1000 ---p 00002000 09:7f 2782223                    /usr/lib64/apache2/modules/mod_authn_anon.so
7f90545c1000-7f90545c2000 r--p 00001000 09:7f 2782223                    /usr/lib64/apache2/modules/mod_authn_anon.so
7f90545c2000-7f90545c3000 rw-p 00002000 09:7f 2782223                    /usr/lib64/apache2/modules/mod_authn_anon.so
7f90545c3000-7f90545c5000 r-xp 00000000 09:7f 2768829                    /usr/lib64/apache2/modules/mod_authn_alias.so
7f90545c5000-7f90547c4000 ---p 00002000 09:7f 2768829                    /usr/lib64/apache2/modules/mod_authn_alias.so
7f90547c4000-7f90547c5000 r--p 00001000 09:7f 2768829                    /usr/lib64/apache2/modules/mod_authn_alias.so
7f90547c5000-7f90547c6000 rw-p 00002000 09:7f 2768829                    /usr/lib64/apache2/modules/mod_authn_alias.so
7f90547c6000-7f90547c8000 r-xp 00000000 09:7f 2768836                    /usr/lib64/apache2/modules/mod_auth_basic.so
7f90547c8000-7f90549c7000 ---p 00002000 09:7f 2768836                    /usr/lib64/apache2/modules/mod_auth_basic.so
7f90549c7000-7f90549c8000 r--p 00001000 09:7f 2768836                    /usr/lib64/apache2/modules/mod_auth_basic.so
7f90549c8000-7f90549c9000 rw-p 00002000 09:7f 2768836                    /usr/lib64/apache2/modules/mod_auth_basic.so
7f90549c9000-7f90549cc000 r-xp 00000000 09:7f 2768870                    /usr/lib64/apache2/modules/mod_alias.so
7f90549cc000-7f9054bcb000 ---p 00003000 09:7f 2768870                    /usr/lib64/apache2/modules/mod_alias.so
7f9054bcb000-7f9054bcc000 r--p 00002000 09:7f 2768870                    /usr/lib64/apache2/modules/mod_alias.so
7f9054bcc000-7f9054bcd000 rw-p 00003000 09:7f 2768870                    /usr/lib64/apache2/modules/mod_alias.so
7f9054bcd000-7f9054bcf000 r-xp 00000000 09:7f 2768867                    /usr/lib64/apache2/modules/mod_actions.so
7f9054bcf000-7f9054dce000 ---p 00002000 09:7f 2768867                    /usr/lib64/apache2/modules/mod_actions.so
7f9054dce000-7f9054dcf000 r--p 00001000 09:7f 2768867                    /usr/lib64/apache2/modules/mod_actions.so
7f9054dcf000-7f9054dd0000 rw-p 00002000 09:7f 2768867                    /usr/lib64/apache2/modules/mod_actions.so
7f9054dd0000-7f9054de5000 r-xp 00000000 09:7f 2413419                    /usr/lib64/gcc/x86_64-pc-linux-gnu/4.8.4/libgcc_s.so.1
7f9054de5000-7f9054fe4000 ---p 00015000 09:7f 2413419                    /usr/lib64/gcc/x86_64-pc-linux-gnu/4.8.4/libgcc_s.so.1
7f9054fe4000-7f9054fe5000 r--p 00014000 09:7f 2413419                    /usr/lib64/gcc/x86_64-pc-linux-gnu/4.8.4/libgcc_s.so.1
7f9054fe5000-7f9054fe6000 rw-p 00015000 09:7f 2413419                    /usr/lib64/gcc/x86_64-pc-linux-gnu/4.8.4/libgcc_s.so.1
7f9054fe6000-7f9054fea000 r-xp 00000000 09:7f 2491044                    /lib64/libattr.so.1.1.0
7f9054fea000-7f90551e9000 ---p 00004000 09:7f 2491044                    /lib64/libattr.so.1.1.0
7f90551e9000-7f90551ea000 r--p 00003000 09:7f 2491044                    /lib64/libattr.so.1.1.0
7f90551ea000-7f90551eb000 rw-p 00004000 09:7f 2491044                    /lib64/libattr.so.1.1.0
7f90551eb000-7f9055379000 r-xp 00000000 09:7f 2143011                    /lib64/libc-2.20.so
7f9055379000-7f9055578000 ---p 0018e000 09:7f 2143011                    /lib64/libc-2.20.so
7f9055578000-7f905557c000 r--p 0018d000 09:7f 2143011                    /lib64/libc-2.20.so
7f905557c000-7f905557e000 rw-p 00191000 09:7f 2143011                    /lib64/libc-2.20.so
7f905557e000-7f9055582000 rw-p 00000000 00:00 0
7f9055582000-7f9055584000 r-xp 00000000 09:7f 2144293                    /lib64/libdl-2.20.so
7f9055584000-7f9055784000 ---p 00002000 09:7f 2144293                    /lib64/libdl-2.20.so
7f9055784000-7f9055785000 r--p 00002000 09:7f 2144293                    /lib64/libdl-2.20.so
7f9055785000-7f9055786000 rw-p 00003000 09:7f 2144293                    /lib64/libdl-2.20.so
7f9055786000-7f905579c000 r-xp 00000000 09:7f 2142954                    /lib64/libpthread-2.20.so
7f905579c000-7f905599b000 ---p 00016000 09:7f 2142954                    /lib64/libpthread-2.20.so
7f905599b000-7f905599c000 r--p 00015000 09:7f 2142954                    /lib64/libpthread-2.20.so
7f905599c000-7f905599d000 rw-p 00016000 09:7f 2142954                    /lib64/libpthread-2.20.so
7f905599d000-7f90559a1000 rw-p 00000000 00:00 0
7f90559a1000-7f90559a9000 r-xp 00000000 09:7f 2142949                    /lib64/libcrypt-2.20.so
7f90559a9000-7f9055ba8000 ---p 00008000 09:7f 2142949                    /lib64/libcrypt-2.20.so
7f9055ba8000-7f9055ba9000 r--p 00007000 09:7f 2142949                    /lib64/libcrypt-2.20.so
7f9055ba9000-7f9055baa000 rw-p 00008000 09:7f 2142949                    /lib64/libcrypt-2.20.so
7f9055baa000-7f9055bd8000 rw-p 00000000 00:00 0
7f9055bd8000-7f9055bdf000 r-xp 00000000 09:7f 2142958                    /lib64/librt-2.20.so
7f9055bdf000-7f9055dde000 ---p 00007000 09:7f 2142958                    /lib64/librt-2.20.so
7f9055dde000-7f9055ddf000 r--p 00006000 09:7f 2142958                    /lib64/librt-2.20.so
7f9055ddf000-7f9055de0000 rw-p 00007000 09:7f 2142958                    /lib64/librt-2.20.so
7f9055de0000-7f9055de4000 r-xp 00000000 09:7f 1992093                    /lib64/libuuid.so.1.3.0
7f9055de4000-7f9055fe4000 ---p 00004000 09:7f 1992093                    /lib64/libuuid.so.1.3.0
7f9055fe4000-7f9055fe5000 r--p 00004000 09:7f 1992093                    /lib64/libuuid.so.1.3.0
7f9055fe5000-7f9055fe6000 rw-p 00005000 09:7f 1992093                    /lib64/libuuid.so.1.3.0
7f9055fe6000-7f9056021000 r-xp 00000000 09:7f 2130139                    /usr/lib64/libapr-1.so.0.5.2
7f9056021000-7f9056220000 ---p 0003b000 09:7f 2130139                    /usr/lib64/libapr-1.so.0.5.2
7f9056220000-7f9056221000 r--p 0003a000 09:7f 2130139                    /usr/lib64/libapr-1.so.0.5.2
7f9056221000-7f9056222000 rw-p 0003b000 09:7f 2130139                    /usr/lib64/libapr-1.so.0.5.2
7f9056222000-7f9056253000 r-xp 00000000 09:7f 1984121                    /usr/lib64/libexpat.so.1.6.0
7f9056253000-7f9056453000 ---p 00031000 09:7f 1984121                    /usr/lib64/libexpat.so.1.6.0
7f9056453000-7f9056455000 r--p 00031000 09:7f 1984121                    /usr/lib64/libexpat.so.1.6.0
7f9056455000-7f9056456000 rw-p 00033000 09:7f 1984121                    /usr/lib64/libexpat.so.1.6.0
7f9056456000-7f9056463000 r-xp 00000000 09:7f 1629813                    /usr/lib64/libgdbm.so.4.0.0
7f9056463000-7f9056663000 ---p 0000d000 09:7f 1629813                    /usr/lib64/libgdbm.so.4.0.0
7f9056663000-7f9056664000 r--p 0000d000 09:7f 1629813                    /usr/lib64/libgdbm.so.4.0.0
7f9056664000-7f9056665000 rw-p 0000e000 09:7f 1629813                    /usr/lib64/libgdbm.so.4.0.0
7f9056665000-7f9056849000 r-xp 00000000 09:7f 1994615                    /usr/lib64/libdb-4.8.so
7f9056849000-7f9056a49000 ---p 001e4000 09:7f 1994615                    /usr/lib64/libdb-4.8.so
7f9056a49000-7f9056a4b000 r--p 001e4000 09:7f 1994615                    /usr/lib64/libdb-4.8.so
7f9056a4b000-7f9056a4e000 rw-p 001e6000 09:7f 1994615                    /usr/lib64/libdb-4.8.so
7f9056a4e000-7f9056a7c000 r-xp 00000000 09:7f 2138855                    /usr/lib64/libaprutil-1.so.0.5.4
7f9056a7c000-7f9056c7b000 ---p 0002e000 09:7f 2138855                    /usr/lib64/libaprutil-1.so.0.5.4
7f9056c7b000-7f9056c7c000 r--p 0002d000 09:7f 2138855                    /usr/lib64/libaprutil-1.so.0.5.4
7f9056c7c000-7f9056c7e000 rw-p 0002e000 09:7f 2138855                    /usr/lib64/libaprutil-1.so.0.5.4
7f9056c7e000-7f9056d01000 r-xp 00000000 09:7f 2117757                    /lib64/libpcre.so.1.2.4
7f9056d01000-7f9056f00000 ---p 00083000 09:7f 2117757                    /lib64/libpcre.so.1.2.4
7f9056f00000-7f9056f01000 r--p 00082000 09:7f 2117757                    /lib64/libpcre.so.1.2.4
7f9056f01000-7f9056f02000 rw-p 00083000 09:7f 2117757                    /lib64/libpcre.so.1.2.4
7f9056f02000-7f9057002000 r-xp 00000000 09:7f 2143986                    /lib64/libm-2.20.so
7f9057002000-7f9057201000 ---p 00100000 09:7f 2143986                    /lib64/libm-2.20.so
7f9057201000-7f9057202000 r--p 000ff000 09:7f 2143986                    /lib64/libm-2.20.so
7f9057202000-7f9057203000 rw-p 00100000 09:7f 2143986                    /lib64/libm-2.20.so
7f9057203000-7f9057208000 r-xp 00000000 09:7f 1614046                    /lib64/libcap.so.2.22
7f9057208000-7f9057408000 ---p 00005000 09:7f 1614046                    /lib64/libcap.so.2.22
7f9057408000-7f9057409000 r--p 00005000 09:7f 1614046                    /lib64/libcap.so.2.22
7f9057409000-7f905740a000 rw-p 00006000 09:7f 1614046                    /lib64/libcap.so.2.22
7f905740a000-7f905742b000 r-xp 00000000 09:7f 2142532                    /lib64/ld-2.20.so
7f9057445000-7f905744c000 r--s 00000000 09:7f 3147094                    /usr/lib64/gconv/gconv-modules.cache
7f905744c000-7f905758e000 rw-p 00000000 00:00 0
7f905758e000-7f905760c000 rw-s 00000000 00:05 6931815                    /dev/zero (deleted)
7f905760c000-7f9057615000 rw-p 00000000 00:00 0
7f9057616000-7f9057617000 rw-p 00000000 00:00 0
7f9057617000-7f9057629000 rw-s 00000000 00:05 6931817                    /dev/zero (deleted)
7f9057629000-7f905762a000 rw-p 00000000 00:00 0
7f905762a000-7f905762b000 r--p 00020000 09:7f 2142532                    /lib64/ld-2.20.so
7f905762b000-7f905762c000 rw-p 00021000 09:7f 2142532                    /lib64/ld-2.20.so
7f905762c000-7f905762d000 rw-p 00000000 00:00 0
7ffdc9208000-7ffdc9229000 rw-p 00000000 00:00 0                          [stack]
7ffdc92bc000-7ffdc92be000 r--p 00000000 00:00 0                          [vvar]
7ffdc92be000-7ffdc92c0000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
[Sat Jul 25 19:39:07 2015] [error] child died with signal 6

Gathering clues

To begin with, the interesting parts are the first and last lines.

  • *** Error in `/usr/sbin/apache2': free(): invalid next size (fast): 0x00007f9041de1280 ***
  • [Sat Jul 25 19:39:07 2015] [error] child died with signal 6

Anytime a free() fails, that’s a strong indication there is something seriously wrong.

In this case, the probable culprit is a heap or stack corruption bug in some modules running within Apache. It is almost never a bug within Apache itself.

The stack trace gives us some clues about what might be happening.

======= Backtrace: =========
/lib64/libc.so.6(+0x74893)[0x7f905525f893]
/lib64/libc.so.6(+0x79f56)[0x7f9055264f56]
/lib64/libc.so.6(+0x7b1e2)[0x7f90552661e2]
/usr/lib64/libmagic.so.1(+0x445d)[0x7f904ddb545d]
/usr/lib64/libmagic.so.1(magic_close+0x1f)[0x7f904ddb4510]
/usr/lib64/apache2/modules/libphp5.so(finfo_resource_destructor+0x33)[0x7f904ce87fef]
/usr/lib64/apache2/modules/libphp5.so(list_entry_destructor+0x8a)[0x7f904d2580f8]
/usr/lib64/apache2/modules/libphp5.so(zend_hash_del_key_or_index+0x205)[0x7f904d2554bc]
/usr/lib64/apache2/modules/libphp5.so(_zend_list_delete+0x6d)[0x7f904d257c3c]
/usr/lib64/apache2/modules/libphp5.so(zif_finfo_close+0xc1)[0x7f904ce888ec]
/usr/lib64/apache2/modules/libphp5.so(+0x62fc29)[0x7f904d289c29]
/usr/lib64/apache2/modules/libphp5.so(+0x630a68)[0x7f904d28aa68]
/usr/lib64/apache2/modules/libphp5.so(execute_ex+0x9ed)[0x7f904d2880f2]
/usr/lib64/apache2/modules/libphp5.so(zend_execute+0x9af)[0x7f904d288af8]
/usr/lib64/apache2/modules/libphp5.so(zend_call_function+0x115e)[0x7f904d22f119]
/usr/lib64/apache2/modules/libphp5.so(zif_call_user_func_array+0x9a)[0x7f904d0bc61d]
/usr/lib64/apache2/modules/libphp5.so(+0x62fc29)[0x7f904d289c29]
/usr/lib64/apache2/modules/libphp5.so(+0x63784c)[0x7f904d29184c]
/usr/lib64/apache2/modules/libphp5.so(execute_ex+0x9ed)[0x7f904d2880f2]
/usr/lib64/apache2/modules/libphp5.so(zend_execute+0x9af)[0x7f904d288af8]
/usr/lib64/apache2/modules/libphp5.so(zend_execute_scripts+0x269)[0x7f904d2448a0]
/usr/lib64/apache2/modules/libphp5.so(php_execute_script+0x43f)[0x7f904d1b5d9d]
/usr/lib64/apache2/modules/libphp5.so(+0x70c0f0)[0x7f904d3660f0]
/usr/sbin/apache2(ap_run_handler+0x61)[0x43f6f7]
/usr/sbin/apache2(ap_invoke_handler+0x179)[0x44005a]
/usr/sbin/apache2(ap_internal_redirect+0x69)[0x45171d]
/usr/lib64/apache2/modules/mod_rewrite.so(+0xdfac)[0x7f90504e2fac]
/usr/sbin/apache2(ap_run_handler+0x61)[0x43f6f7]
/usr/sbin/apache2(ap_invoke_handler+0x179)[0x44005a]
/usr/sbin/apache2(ap_process_request+0x6d)[0x450b6a]
/usr/sbin/apache2[0x44da53]
/usr/sbin/apache2(ap_run_process_connection+0x61)[0x448c9d]
/usr/sbin/apache2(ap_process_connection+0x67)[0x449133]
/usr/sbin/apache2[0x4580cc]
/usr/sbin/apache2[0x4583f0]
/usr/sbin/apache2[0x458673]
/usr/sbin/apache2(ap_mpm_run+0x4bb)[0x458b7c]
/usr/sbin/apache2(main+0xae1)[0x42565c]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f905520ca95]
/usr/sbin/apache2[0x424279]

The stacktrace (backtrace) show us the steps leading up to the point of crash. There is no doubt the crash happened in libc, but as with Apache, there is almost never something wrong with system libraries. Those are used throughout the system and an error here would occurred somewhere else too, and not only in Apache.

The likely reason why libc is failing is because there was a failing assert in libc.

We can investigate this with gdb. Start GDB with the .so file as argument, like this gdb /lib64/libc.so.6. By using the offset from the backtrace, we can look up the closest symbol in the .so file.

Type info symbol 0x7b1e2 to find what method resides under the first frame in libc.so.

# gdb /lib64/libc.so.6

(gdb) info symbol 0x74893
__libc_message + 723 in section .text of /lib64/libc-2.20.so
(gdb) info symbol 0x79f56
malloc_printerr + 182 in section .text of /lib64/libc-2.20.so
(gdb) info symbol 0x7b1e2
_int_free + 1826 in section .text of /lib64/libc-2.20.so

Indeed, the first error message about there being something fishy with free() was not mistaken. However, we must dig deeper to find something more useful.

And about that digging deeper. I was curious on how PressBooks works when exporting a sample book, not that curious into debugging Apache and PHP.

The issue turned out to be some conflict with mod_svn and mod_php. All I had to do was disable mod_svn, as I’ve moved on to Mercurial and Git a long time ago.

In the /etc/conf.d/apache2 file, remove -D SVN from APACHE2_OPTS.

Restart Apache2 with /etc/init.d/apache2 restart and then everything was OK.