{"id":1187,"date":"2011-07-24T11:09:42","date_gmt":"2011-07-24T08:09:42","guid":{"rendered":"http:\/\/www.void.gr\/kargig\/blog\/?p=1187"},"modified":"2011-10-18T10:15:42","modified_gmt":"2011-10-18T07:15:42","slug":"resolving-ossec-active-response-iptables-issues","status":"publish","type":"post","link":"https:\/\/www.void.gr\/kargig\/blog\/2011\/07\/24\/resolving-ossec-active-response-iptables-issues\/","title":{"rendered":"Resolving OSSEC active response iptables issues"},"content":{"rendered":"

The past few days some of my servers are having difficult times due to the increase of spam by some botnet(s). From around 600-700 emails per day for unknown addresses\/recipients on local domains, this number reached a peak of 8.000 emails 2 days ago. In order to reduce further botnet attempts I’m having ossec<\/a> to engage, which in turn tries to firewall hosts.<\/p>\n

That worked quite ok for a while but then I’ve started seeing errors in the active-response.log like the ones below:<\/p>\n

Unable to run (iptables returning != 3): 1 – \/var\/ossec\/active-response\/bin\/firewall-drop.sh delete – 91.121.21.8 1310919172.51029 31106
\nUnable to run (iptables returning != 1): 1 – \/var\/ossec\/active-response\/bin\/firewall-drop.sh delete – 79.149.198.149 1310919524.52191 3302
\nUnable to run (iptables returning != 1): 2 – \/var\/ossec\/active-response\/bin\/firewall-drop.sh delete – 79.149.198.149 1310919524.52191 3302
\nUnable to run (iptables returning != 1): 3 – \/var\/ossec\/active-response\/bin\/firewall-drop.sh delete – 79.149.198.149 1310919524.52191 3302
\nUnable to run (iptables returning != 1): 4 – \/var\/ossec\/active-response\/bin\/firewall-drop.sh delete – 79.149.198.149 1310919524.52191 3302
\nUnable to run (iptables returning != 1): 5 – \/var\/ossec\/active-response\/bin\/firewall-drop.sh delete – 79.149.198.149 1310919524.52191 3302
\nUnable to run (iptables returning != 4): 1 – \/var\/ossec\/active-response\/bin\/firewall-drop.sh add – 115.242.188.157 1310969220.1045522 3302\n<\/p><\/blockquote>\n

Obviously iptables is busy doing something else at the time, adding or deleting some other rule, so the loop inside firewall-drop.sh sometimes fails. That was a bit worrying, I had to fix ossec so one way or another so that iptables rules would eventually be applied. I’ve faced the same issue with iptables in the past, trying to simultaneously add multiple (>5) iptables rules at exactly the same time is very error prone, there’s no way to tell which of those rules will be applied. In order to circumvent the issue, I added locking to the active response script.<\/p>\n

Whenever it comes to locking with shell scripts I am using a set of four functions inside a file that I source when I need to. I place this file usually inside \/usr\/local\/bin\/ under the lock.sh filename.<\/p>\n

lockme () {\n    if [ -z "$1" ];then\n        echo " o Use an argument to lock"\n        return 1\n    fi\n    if [ -z "$2" ];then\n        PID=$$\n    else\n        PID=$2\n    fi\n    LOCK_PID_FILE=\/var\/lock\/$1\n    if [ -f $LOCK_PID_FILE ];then\n        sleep 1\n        echo " o Lock file found"\n        if [ ! -d \/proc\/`cat $LOCK_PID_FILE 2>\/dev\/null` ];then\n            echo " o Stale lock file ignoring..."\n            rm -f $LOCK_PID_FILE\n        else\n            return 1\n        fi  \n    fi  \n    #temp file\n    echo -n $PID > $LOCK_PID_FILE.$PID\n    ln -s $LOCK_PID_FILE.$PID $LOCK_PID_FILE && return 0\n    rm -f $LOCK_PID_FILE.$PID\n    return 1\n}\n\nlockme_wait () {\n    if [ -z "$1" ];then\n        echo " o Use an argument to lock"\n        return 1\n    fi  \n    if [ -z "$2" ];then\n        PID=$$\n    else\n        PID=$2\n    fi  \n    while [ 1 ];do\n        lockme $1 $PID && break\n        sleep 4\n    done\n    return 0\n}\n\nunlockme () {\n    if [ -z "$1" ];then\n        echo " o Use an argument to unlock"\n        return 1\n    fi\n    #remove pid file\n    rm -f \/var\/lock\/$1.`cat \/var\/lock\/$1 2>\/dev\/null`\n    rm -f \/var\/lock\/$1\n    return 0\n}   \n\nkill_locked () {\n    if [ -z "$1" ];then\n        echo " o Use an argument to kill_locked"\n        return 1\n    fi\n    if [ -e \/var\/lock\/$1 ]; then\n        kill `cat \/var\/lock\/$1 2>\/dev\/null`\n    fi\n    rm -f \/var\/lock\/$1.`cat \/var\/lock\/$1 2>\/dev\/null`\n    rm -f \/var\/lock\/$1\n}\n<\/code2><\/pre><\/p>\n
You can also use %s\/var\\\/lock\/tmp\/g<\/code> if you prefer having the locks on the \/tmp which is usually ramfs, partition.<\/p>\n
Afterwards I edited \/var\/ossec\/active-response\/bin\/firewall-drop.sh to just add 3 lines. (I only edited the relevant Linux section of the script, since I haven’t tested, or don’t even know if it’s needed on the BSD, SunOS sections, I left those unedited):<\/p>\n
  • Add . \/usr\/bin\/lock.sh<\/code> right after the “# Checking for an IP” section (around line 45)<\/li>\n
  • Right after “# Executing and exiting” add lockme_wait active-response<\/code> (around line 75)<\/li>\n
  • Right after the second while loop finishes, after “done” and before “exit 0” add unlockme active-response<\/code> (around line 110)<\/li>\n
    That’s it…just 3 lines added and the errors have completely stopped since then.<\/p>\n
    P.S. Yes, I could have used lockfile-progs<\/a> to achieve the same result, but I (also) use lock.sh file in embedded systems when needed, and it’s far more portable and easy.<\/p>\n","protected":false},"excerpt":{"rendered":"
    The past few days some of my servers are having difficult times due to the increase of spam by some botnet(s). From around 600-700 emails per day for unknown addresses\/recipients on local domains, this number reached a peak of 8.000 emails 2 days ago. In order to reduce further botnet attempts I’m having ossec to […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ep_exclude_from_search":false,"footnotes":""},"categories":[5,3],"tags":[389,385,328,595,387,388,217,386,206,304,238],"class_list":["post-1187","post","type-post","status-publish","format-standard","hentry","category-internet","category-linux","tag-active-response","tag-attacks","tag-iptables","tag-linux","tag-lock","tag-locking","tag-ossec","tag-scripting","tag-security","tag-shell","tag-spam"],"aioseo_notices":[],"views":15237,"_links":{"self":[{"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/posts\/1187","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/comments?post=1187"}],"version-history":[{"count":8,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/posts\/1187\/revisions"}],"predecessor-version":[{"id":1238,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/posts\/1187\/revisions\/1238"}],"wp:attachment":[{"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/media?parent=1187"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/categories?post=1187"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.void.gr\/kargig\/blog\/wp-json\/wp\/v2\/tags?post=1187"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}