Into.the.Void.

3 days ago, after a Debian(squeeze/sid) upgrade on my laptop some programs started not to open. Specifically, pidgin and google-chrome were crashing while trying to open them. When I started them from a terminal the output was this:

kargig@laptop:~%pidgin
[1]    3853 abort      pidgin
kargig@laptop:~%google-chrome
[1]    3882 abort      google-chrome

The first thing I checked was the updated packages, whether there was some culprit.
The upgraded packages included among others:

[UPGRADE] libk5crypto3 1.8.3+dfsg-1 -> 1.8.3+dfsg-2
[UPGRADE] libkrb5-3 1.8.3+dfsg-1 -> 1.8.3+dfsg-2
[UPGRADE] libkrb5support0 1.8.3+dfsg-1 -> 1.8.3+dfsg-2
[UPGRADE] libnspr4-0d 4.8.4-2 -> 4.8.6-1
[UPGRADE] libnss3-1d 3.12.6-3 -> 3.12.8-1
[UPGRADE] linux-base 2.6.32-23 -> 2.6.32-25
[UPGRADE] linux-headers-2.6.32-5-686-bigmem 2.6.32-23 -> 2.6.32-25
[UPGRADE] linux-headers-2.6.32-5-common 2.6.32-23 -> 2.6.32-25
[UPGRADE] linux-image-2.6.32-5-686-bigmem 2.6.32-23 -> 2.6.32-25
[UPGRADE] linux-libc-dev 2.6.32-23 -> 2.6.32-25
[UPGRADE] xserver-xorg-video-intel 2:2.9.1-4 -> 2:2.12.0+shadow-2

My first point of checking was the xserver-xorg-video package. I started searching the Debian bug tracking system for references of crashes with abort. Nothing. Then I tried to check the other “suspicious” packages with abort crash reports on the bug tracker…still nothing.
It was time for strace.

kargig@laptop:~%strace pidgin
...
[snip]
...
open("/usr/lib/nss/libfreebl3.so", O_RDONLY) = 14
read(14, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0\30\0\0004\0\0\0"..., 512) = 512
fstat64(14, {st_mode=S_IFREG|0644, st_size=253328, ...}) = 0
mmap2(NULL, 268988, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 14, 0) = 0xb58e3000
mmap2(0xb5920000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 14, 0x3d) = 0xb5920000
mmap2(0xb5921000, 15036, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb5921000
close(14)                               = 0
open("/etc/ld.so.cache", O_RDONLY)      = 14
fstat64(14, {st_mode=S_IFREG|0644, st_size=79200, ...}) = 0
mmap2(NULL, 79200, PROT_READ, MAP_PRIVATE, 14, 0) = 0xb5ab7000
close(14)                               = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/swiftfox/libnspr4.so", O_RDONLY) = 14
read(14, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0P\227\0\0004\0\0\0"..., 512) = 512
fstat64(14, {st_mode=S_IFREG|0755, st_size=251136, ...}) = 0
close(14)                               = 0
munmap(0xb5ab7000, 79200)               = 0
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
tgkill(4027, 4027, SIGABRT)             = 0
--- SIGABRT (Aborted) @ 0 (0) ---
+++ killed by SIGABRT +++
[1]    4026 abort      strace pidgin

What immediately caught my eye was this line inside the output:
open("/usr/lib/swiftfox/libnspr4.so", O_RDONLY) = 14
I have a 3rd party package called swiftfox installed, but why was pidgin trying to use this package’s library instead of the system one ?

# ldconfig -p | grep nspr4
ldconfig -p | grep nspr4
  libnspr4.so.0d (libc6) => /usr/lib/libnspr4.so.0d
  libnspr4.so (libc6) => /usr/lib/swiftfox/libnspr4.so
  libnspr4.so (libc6) => /usr/lib/libnspr4.so

So the system package libnspr4-0d has installed its files in /usr/lib/libnspr4.so.0d and has also placed a symlink from /usr/lib/libnspr4.so to /usr/lib/libnspr4.so.0d. For some reason though the /usr/lib/swiftfox/libnspr4.so appears before /usr/lib/libnspr4.so in the cache output for the libnspr4.so library.
Checking out /etc/ld.so.conf.d/ directory, there was a moz.conf file containing the path “/usr/lib/swiftfox/“.
An “ldconfig -v” confirmed the finding:

/usr/local/lib:
/usr/lib/swiftfox:
        libnssckbi.so -> libnssckbi.so
        libssl3.so -> libssl3.so
        libsqlite3.so -> libsqlite3.so
        libnssutil3.so -> libnssutil3.so
        libnss3.so -> libnss3.so
        libnspr4.so -> libnspr4.so
        libsmime3.so -> libsmime3.so
        libmozjs.so -> libmozjs.so
        libsoftokn3.so -> libsoftokn3.so
        libplc4.so -> libplc4.so
        libxul.so -> libxul.so
        libplds4.so -> libplds4.so
        libxpcom.so -> libxpcom.so
        libnssdbm3.so -> libnssdbm3.so
        libfreebl3.so -> libfreebl3.so
/lib:
        libnss_compat.so.2 -> libnss_compat-2.11.2.so
        libselinux.so.1 -> libselinux.so.1
...
[snip]
...

Moving /usr/lib/swiftfox/libnspr4.so to some other location allowed applications like pidgin and google-chrome to start normally (and swiftfox still runs properly).

I guess that was my punishment for using 3rd party packages on Debian…

*UPDATE 23/11/2010*
Google chrome was crashing with some https:// sites with SIGABRT. After further investigation I had to delete /usr/lib/swiftfox/libnssutil3.so as well.

Filed by kargig at 19:39 under Linux
Tags: abort, crash, debian, google-chrome, ldconfig, libraries, Linux, pidgin, SIGABRT, strace, swiftfox
2 Comments | 9,642 views

Plesk is surely not ready for IPv6. Despite that fact, many people – me included, have the DNS records of their favorite domains managed by Plesk and still want to be able to add some IPv6 records to those.

Some time ago I had posted on my twitter account a link to another blog that had a “hackish way” to add AAAA records to Plesk. I have written a slightly more elegant shell script (to be run by root only) than the one provided by experimentalworks.

First of all you _need_ to alter dns_recs table of the psa database to allow AAAA records:

# mysql -u admin -p psa 
mysql> alter table dns_recs modify column type enum('NS','A','AAAA','CNAME','MX','PTR','TXT','SRV','master','none') NOT NULL default 'A'; 

Then download my plesk-AAAA.sh script and use it like the following example.

To add www.foobar.gr to point to 2001:db8:1001::1

Usage: ./plesk-AAAA.sh [zone serial]
#./plesk-AAAA.sh foobar.gr www 2001:db8:1001::1
#./plesk-AAAA.sh foobar.gr ipv6 2001:db8:1001::1 12

Known bug/feature:
If you add a record without adding a serial, for the soa record, at the end, it will add the serial of the domain in the form:

YYYYMMDD10

So if you add two ipv6 hosts in the same day for the same domain you _have_ to manually add a serial >10 for the second host (and so forth).

For the ones who don’t like downloading but would like to see the script source, here it is:

  1 #!/bin/sh
  2 
  3 usage () {
  4         echo "Usage: $0 <domain> <hostname> <v6 IP> [zone serial]"
  5         echo "Usage: $0 foobar.gr www 2001:db8:1001::1"
  6         exit 1
  7 }
  8 
  9 if [ $# -lt 3 ]; then
 10         usage
 11 fi
 12 DOMAIN=$1
 13 HOSTNAME=$2
 14 v6IP=$3
 15 INPUT_SERIAL=${4:-10}
 16 FULLHOST="$2.$1."
 17 
 18 ADMIN_PASS=`cat /etc/psa/.psa.shadow`
 19 MYSQL_BIN_D=`grep MYSQL_BIN_D /etc/psa/psa.conf | awk '{print $2}'`
 20 PRODUCT_ROOT_D=`grep PRODUCT_ROOT_D /etc/psa/psa.conf | awk '{print $2}'`
 21 SERIAL=`date +%Y%m%d${INPUT_SERIAL}`
 22 mysql="${MYSQL_BIN_D}/mysql -N -uadmin -p${ADMIN_PASS} psa"
 23 
 24 query1="SELECT dns_zone_id FROM dns_recs where host like \"$DOMAIN%\" LIMIT 0,1"
 25 ZONE_ID=`echo "$query1" | $mysql`
 26 echo "ZONE_ID=$ZONE_ID"
 27 query2="INSERT INTO dns_recs (displayHost, host, displayVal, val, type, dns_zone_id) VALUES ('$FULLHOST', '$FULLHOST', '$v6IP', '$v6IP', 'AAAA',$ZONE_ID)"
 28 echo "$query2" | $mysql
 29 
 30 query3="UPDATE dns_zone SET serial=\"$SERIAL\" WHERE id=$ZONE_ID LIMIT 1;"
 31 echo "$query3" | $mysql
 32 
 33 echo "REBUILDING zone file for $DOMAIN"
 34 $PRODUCT_ROOT_D/admin/sbin/dnsmng update $DOMAIN

The script has been tested with bash and zsh. I have no idea whether it works under any other shells.
The script probably won’t delete your databases, but…use it at your own risk 🙂 I hope someone finds it useful.

Filed by kargig at 20:10 under Internet,Linux,Networking
Tags: aaaa, aaaa record, bash, dns, ipv6, Linux, Networking, plesk, script, shell, zsh
2 Comments | 8,177 views

Lately I was given a task of moving some websites/webservices from real boxes to some VMs. Most of the sites were Joomla! applications so moving the installation was quite easy, tar files, check configuration.php for db username/pass/etc and dump the database on the old server and then copy these to the VM. Restore files, import database, minor path changes to configuration.php… that’s about it.

But then it was time to move an “eclass” application. Specifically it was an installation of Open eClass, a web based e-learning software created by Greek Academic Network. So I copied the files, found the configuration file with database credentials, dumped the db and moved it to the VM. The site came up but it was not functioning properly. Course material was missing from the website, but I could certainly see the files on the file system. I dumped the database again and re-imported it. Nothing, the site refused to work as expected. I went back to the original machine and shut down mysql to start it with “–skip-grant-tables” since I didn’t have the root mysql password. MySQL came up, I logged in as root and I typed: “show databases;”

Oh the horror!!!!
I couldn’t believe my eyes…in front of me there were more than 200 databases with the names of courses of the e-elearning platform! I shut down mysqld and restarted it normally. Then I logged in as the “eclass” user and issued the following:
show grants for eclass@localhost;
The output:

| GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, CREATE TEMPORARY TABLES, LOCK TABLES ON *.* TO 'eclass'@'localhost' IDENTIFIED BY PASSWORD 'XX' | 
| GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, CREATE TEMPORARY TABLES, LOCK TABLES ON `eclassdb`.* TO 'eclass'@'localhost'  |

I immediately started thinking that someone had _really_ fucked up the installation. I went to Open eClass website and tried to search for documentation on installation instructions. I downloaded a pdf and I read between the installation instructions:

A “username” and a “password” for MySQL with database creation rights.

.
Okie..let’s translate that to simple english, it needs a ‘root’ mysql account renamed to something else.

I am not a web developer, I do not even consider myself a developer, but this setup makes no sense for me. Who and why decided that it would be a good idea to have a web application’s mysql user being able to create new databases ? Is this application only to be installed on a machine of its own ? If so, it’s such a waste of resources. I can understand the complexity and the extra time that a well designed and correctly normalized database requires, but this isn’t an excuse when creating software to be distributed and widely used by lots of people, especially universities. I can’t judge the application, it actually looks quite useful, but it’s setup certainly has design problems that need to be solved.

And finally, what “if” there is some security hole in the application (sql injections anyone?) and a malicious user starts dropping databases other than the ones belonging to eclass ? Who’s to blame for that ?

My advice to anyone running this application is to have it as isolated as possible from the rest of his infrastructure. Possibly in a virtual machine of its own. And there should be a warning about it on the website.

P.S. Looking at the credits, it seems that I know in person some of its developers, and that makes it ever harder to blog about what I faced. I’ll certainly ask them about this web application the next time I meet them though.

Filed by kargig at 11:05 under Internet,Linux
Tags: database, design, eclass, mysql, openeclass, security, sql injection, virtual machine, web application, website
7 Comments | 15,844 views