I’ll go through a false positive example I found on my blog. False positives are inevitable, so it’s not a bad idea to run mod_security for a few weeks on detect only.

Here’s the url that was raising the alarm. The trigger was in the url.

http://chesterton.id.au/blog/2007/11/20/blue-tongue-harmonica-talk-cd/

Here’s what modsec_audit.log looked like

--f8a03521-A--
[21/Mar/2011:07:54:47 +1100] TYZpl0LcAZkAAHCLBPIAAAAA ::ffff:124.169.31.22 34169 127.0.0.1 81
--f8a03521-B--
GET /blog/2007/11/20/blue-tongue-harmonica-talk-cd/ HTTP/1.0
Host: chesterton.id.au
X-Real-IP: ::ffff:124.169.31.22
X-Forwarded-For: ::ffff:124.169.31.22
Connection: close
Cache-Control: no-cache
Pragma: no-cache
User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.26 (KHTML, like Gecko) Ubuntu/10.10 Chromium/12.0.709.0 Chrome/12.0.709.0 Safari/534.26
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3

--f8a03521-Z--


Section B –f8a03521-B– contains the request headers, and you can see the GET is /blog/2007/11/20/blue-tongue-harmonica-talk-cd/.

Section H is the audit log, why it was blocked in this case.
[msg "System Command Injection"] [data "cd/"] so it didn’t like cd/, I’m not sure that cd/ (without a space between the cd and the directory) would do much on a unix system, but perhaps it does on windows, where apache also runs.

It also tells you the pattern that matched, the file where the rule is located and the rule id.
Pattern match “\bcd\b\W*?[\/]” at REQUEST_FILENAME. [file "/etc/apache2/modsecurity_crs/base_rules/modsecurity_crs_40_generic_attacks.conf"] [line "396"] [id "958821"]

What we can do, is when REQUEST_FILENAME = /blog/2007/11/20/blue-tongue-harmonica-talk-cd/ remove rule id 958821.

SecRule REQUEST_FILENAME "^/blog/2007/11/20/blue-tongue-harmonica-talk-cd/(index.php)?$" "nolog,pass,ctl:RuleRemoveById=958821"


Add this rule to your local rules, the reason we don’t modify the base_rules is they get overwritten when there’s an update to the rules.

“pass” above means if the rule matches, keep processing other rules, so it will still catch xss for example:

http://chesterton.id.au/blog/2007/11/20/blue-tongue-harmonica-talk-cd/?a=%3Cscript%3Ealert(1)%3C/script%3E

I’d like to come back to this later and lock it down a little.

In my previous post I didn’t mention how to turn on the audit log, so without explanation, you put the following in your config.


SecAuditEngine RelevantOnly
SecAuditLogType serial
SecAuditLogParts ABCFHZ


We already set SecAuditLog to /var/log/modsecurity/SecAuditLog/modsec_audit.log in the previous post, which has to be writable by the web server user, www-data for ubuntu/debian. So after a reload, you should start seeing entries when mod_security denies a request.

I’m going to modify SecAuditLogType setting in a future post so we can enable the console.

Some time ago, I changed my setup from nginx + spawn-fcgi to nginx + apache so I could play with mod_security. Well, one of my clients got pwned by some sql injection vulnerability, so I finally got to put it in practice.

In a nut shell, mod_security is an apache module that scans http(s) traffic for suspicious strings. It can catch suspicious strings entering apache or leaving apache. It’s a firewall for http(s), it blocks bad traffic, and lets good traffic pass through.

mod_security is one of those applications that doesn’t fit the package model well. I guess it comes in two parts, the apache module and the rules. Packages don’t seem to even attempt to manage the rules, you download them separately and they come with a util to update them. Then you have your local rules that add to, disable, or override the upstream rules to fit your environment.

In ubuntu, the mod_security package is a little outdated, it always will be, especially on LTS, and it’s one of those packages that needs constant updates. The clever and motivated security researchers eventually find weakness in mod_security which are addressed in updates. So you either use the packages, and hope fixes are backported in a timely manner, which hasn’t appeared to of happened in ubuntu, or you install by source.

Installing by source increases your workload and moves the burden of keeping it up to date to yourself. But it’s a pretty straightforward process, and if you have only one or two machines to admin, it’s no issue.

It’s basically a matter of downloading the source, checking the signature, running apt-get build-dep libapache-mod-security, untaring, ./configure; make;make test;make install, then enable the module in apache. It’s pretty well covered in the docs and blogs found in google.

That takes care of the module, but it does nothing without the rules. I downloaded the core rules from OWASP. There some documentation in the package, but there’s a few gotchas, so I’ll try and document step by step the process I went through.

1. unpack the rules in your home directory
2. sudo mkdir /etc/apache2/modsecurity_crs/
3. cd modsecurity-crs_2.1.2
   sudo cp -a *_rules *.example /etc/apache2/modsecurity_crs/
4. cd /etc/apache2/modsecurity_crs/
   sudo mv modsecurity_crs_10_config.conf{.example,}
5. You have to edit modesecurity_crs_10_config.conf and uncomment a few lines, I think the main one being SecRuleEngine.
6. I created two files in /etc/apache2/mods-available
   0modsecurity.conf
   <IfModule security2_module>
Include modsecurity_crs/*.conf
Include modsecurity_crs/base_rules/*.conf
</IfModule>
   0modsecurity.load
   LoadFile /usr/lib/libxml2.so
LoadFile /usr/lib/liblua5.1.so
LoadModule security2_module /usr/lib/apache2/modules/mod_security2.so
7. sudo a2enmod 0modsecurity
8. create a file called /etc/apache2/modsecurity_crs/modsecurity_localconfig.conf with the following
   SecUploadDir /var/log/modsecurity/SecUploadDir
SecAuditLog /var/log/modsecurity/SecAuditLog/modsec_audit.log
SecAuditLogStorageDir /var/log/modsecurity/SecAuditLogStorageDir
SecDebugLog /var/log/modsecurity/SecDebugLog/modsec_debug.log
SecDataDir /var/log/modsecurity/SecDataDir
SecTmpDir /var/log/modsecurity/SecTmpDir

9. Make all the directories in /var/log/modsecurity, which have to be owned by www-data
   sudo mkdir -p /var/log/modsecurity/{SecUploadDir,SecAuditLog,SecAuditLogStorageDir,SecDebugLog,SecDataDir,SecTmpDir}
sudo chown -R www-data: /var/log/modsecurity

That will do for now, I think. To modify the rules I feel it’s best to slow down and buy a book for learning, I’m not sure docs and blogs are the best way to learn this, but it’s up to your learning style. I keep updated by following @ModSecurity on twitter.

I’ve reinstalled apache and I’m in the process of cutting over my sites. Not completely, just for non static files, mainly php. Nginx is infront of apache serving static content and is a reverse proxy to apache which runs on 127.0.0.1:81

The reason? I want to give mod_security a spin. It’s loaded now, but it doesn’t seem to do much atm. Will read up on it in due season ;)

Basically, you google for nginx apache proxy wordpress wp-super-cache and you hit the right sort of pages to set it up. It wasn’t smooth sailing though, I used http://tech.nocr.at/tech/how-to-speed-up-wordpress-with-nginx-and-wp-super-cache/ which has a flaw.

It contains the lines at the end on the nginx config sample.
# all other requests go to WordPress
if (!-e $request_filename) {
rewrite . /index.php last;
}
and funnily enough, every url opened the front page of my blog. It took me a bit of head scratching before I commented the lines out, enabled mod_rewrite in apache, and toggled a wordpress permalink setting to get .htaccess written.

After I got it working, I read the comments and someone had already discovered the flaw. Plus another page I read is basically the same, except it’s missing those lines (it wasn’t as good looking, so I didn’t use it).

So I’m pretty happy that I understand the config now, and I’ll continue to cut over the remaining sites, then turn off php5-cgi. I’m also happy apache is back, I went a bit anti apache there for a while, but it has its uses, like mod_security for example.

Sheesh, I write one post on security, mention a security expert in it, and he tries to exploit my server. It’s not an original idea, but if you want to test your server for vulnerabilities, come out and say you’re a security expert.

Anyway, I finished Cyber War, feeling very satisfied since it’s been a long time since I had the will to read a book. Twitter’s 140 character limit has taken a toll.

I think the conclusion the author made was regulation, which will erode privacy, and treaties. Also a new protocol where there are no anonymous users (except for the bad guys, i guess).

The author has no doubt that our critical systems already have backdoors and logic bombs installed, waiting for the day. One thing that struck me is no matter how secure my own network is (which it isn’t very, no need to test), if someone else is vulnerable, it could affect me and millions of others.

On to the next book. This one is lower level I think.

I’ve recently become more interested in security. I always had an interest in it, locking down linux servers and firewalls, trying to avoid common sql injection and XSS bugs while coding. But I guess that’s more of a passive interest or approach, I’m now interested in a more active way, deep packet inspection, etc.

In the past, I locked down a server and assumed it would take care of itself, be crack proof, as crack proof as anything connected to the internet can be. I wasn’t fooling myself, I knew it could still be cracked.

Now I’m looking at it more from position where I believe the hosts are vulnerable, and I want to examine the traffic coming in and out. Trying to secure the network rather than (or as well as) the hosts. Because, well, some hosts can’t be secured, there’s always going to be 0day exploits, old unpatched exploits, and social hacking. Not just trying to stop attacks, but trying to detect owned boxes. Some attacks happen out of band, like from infected usb sticks. Want to sneak some worm onto a target? Do it in plain sight, stand outside their premises and give away usb sticks in a fake promotion, or do it in some conference.

I guess I’ve got to send props to David Bl (Security Expert) for sparking the interest.

I’m half way through Cyber War, the first book I’ve read in a very long time. It’s not a technical manual on security, it’s more of a look at how vulnerable the USA and other countries are from attacks coming in through the internet. It talks about political and military tactics and strategy. It’s more about securing a nation, rather than a network.

It seems the only reason we don’t see a lot of countries launching attacks on other countries through the internet is the countries who have the ability to attack a potential adversary, also have an interest, be it in trade, or bonds or whatever, in the potential adversary. It’s not in their best interest to attack at the moment. Things are going to change though.

You don’t need to be a country to take on someone like the USA, a single person, or a small group with the right know how can do a lot of damage to the military or public infrastructure. Remotely damaging electric generators that take months to replace, crippling banks, whatever. Anything that is connected in someway to the internet is a target. Australia and USA are looking at smart grids, the electric grid is already connected to the internet and it’s going to get a lot more connected and complicated with the added bonus of more attack vectors.

What can be done by the government to protect its citizens in an attack? Nothing, really. Except for China, who have a big firewall they can use to shut off attacks. Anything a government might do to strengthen its internet defence is going to be an attack on citizens privacy. Who in their right mind would trust a government to do the right thing with all the power they would hold over their citizens, if they had the control they needed to defend themselves and their citizens from internet attacks?

Which brings me to the question of Australia’s proposed filter. Filter bad, Conroy bad, and all that, but the filter is a firewall between Australian citizens and the internet, it could be used as a defensive measure against attacks. Not necessarily the technology Conroy is proposing, but a suitable firewall could.

But it’s like the government being able to listen in on every phone call, every conversation. It’s not going to happen, but something needs to happen.

I dare say the font on the security tag on this blog is going to get large in the tag cloud. These are young thoughts, they may change and I’ll be talking about security a lot more.

I won’t go into too many details, I got everything I learned from google and my customisations aren’t very ground breaking.

Jdub tweeted about the wordpress apc object cache being updated. So I (re)installed that. I had it installed a while ago, but a wordpress upgrade broke it. http://wordpress.org/extend/plugins/apc/

Everyone knows about wp-super-cache, and there’s some good nginx recipes and discussions on the internet already about how to run the two together.

My blog isn’t very big, or updated very often, so I tried setting wp-super-cache to preload cache files once a day. Except after the first day, it seemed to get stuck and wouldn’t regenerate them. So now, it just generates (and compresses) them after the first visit and I’ve configured it so spiders can generate them, which isn’t the default. The cached files are set to expire after a day.

I’ve noticed recently that spiders are sometimes using the short url form ?p=100 rather than the long SEO style urls. With my nginx config, it won’t serve a super-cached file using the short url form. It won’t serve a super-cached file if the url has a ? in it.

This was new to me, a wordpress plugin called wp-minify. It’s an awesome idea. It combines multiple css (and, separately, js) links into one, minified version, and compresses and caches the result. So the browser only has to make one css get and one js get. Just helps pages load that little bit quicker. Good for the search engines, good for the users.

I set nginx to gzip text output, as well as look for a pre gzipped file first. So I basically ran:

for a in `find /www -name "*.js" -o -name "*.css" -o -name "*.html"`;do cat $a |gzip -c > $a.gz ;done

wp-recaptcha is a pain the bum when it comes to page load times, it slows the page display to one second. But it does a good job of stopping boring spam comments. auxesis suggested moving the <script src=…> to the end to the dom, which I’ve done, and that does help with the feel, it doesn’t reduce overall load time though. It’s just that I couldn’t find a good way to do it, and I’m going to have to merge my changes with every upgrade.

That’s about it I think, the name of the game was to decrease page load times.

Many years ago, I was a rabid (vapid?) PostgreSQL fanboi. I took the time to study how to tune postgres, it was very rewarding to see queries go from taking 30 seconds, to completing in under a second. This involved tuning knobs in the prostres config, to adding indexes. All done manually.

If some software would only work on mysql, I wouldn’t bother looking at it.

A few years ago I wanted to run WordPress on my Linode, I gave up my extremist software bias, and uninstalled postgres, and installed mysql. To tune it, I picked one of the three or four my.cnf files provided. And that was that. Although I gave up postgres for mysql, I showed it no love.

During my recent webserver admin gorge, I came across a shell script that would help you tune mysql’s knobs. Most knobs weren’t even mentioned in my my.cnf file, so were at the default. I found much delight in it telling me to raise this, and lower that. In the end, I ended up with mysql using about the same amount of memory, but much more efficiently, by using more memory here where required, and using less memory there where not needed.

The script is called tuning-primer.sh and can be found at http://www.day32.com/MySQL/.

It took me many days to tune my mysql databases, run the script, adjust knobs, restart mysql, let it run for a day or two, and repeat. It turns out some knobs don’t need adjusting until other knobs are adjusted first, so you can’t get it right the first go. Plus I was quite cautious in how much I would adjust a knob.

There’s also another script I haven’t yet looked at called mysqltuner.pl, google will hook you up.

And here’s some music I’ve been enjoying lately from Sia, (an adelaide girl) her live performances are better than her albums. Sia – Soon We’ll Be Found (Letterman)

I recently swapped over from lighty to nginx, and man, am I glad I did. I have no idea about the internals, which is faster, lighter, better, etc, but nginx is way more intuitive to configure. Which in the end means my server is set up better.

That kick started my latest obsession, speeding up web servers.

I might go into details in a later post, but first, a question. I’ve got my wordpress blog loading nice and quick, except for pages where you can leave a comment, the recaptcha pluggin slows down the display of the page to 1 second. Obviously this is unacceptable. 1 SECOND!!!!!

What I would like is a “leave a comment” link to be displayed at the end of the post or comments, and clicking it would load the comment box with the recaptcha.

Any ideas how I would go about it?

I’ve tried css tricks to hide divs, but the javascript and images still load and hold up the display of the page, even though they aren’t (immediately) displayed. Annoying.

I left a few things unsaid in my previous post basic hostname set up, and after talking with some people about /etc/hosts, I thought a quick follow up is in order.

Firstly, my preference is to use the ethernet’s ip address to tie the host name to the FQDN, but others like to use 127.0.1.1. There’s a case where you really should use 127.0.1.1 and that’s in laptops where the ethernet interface might disappear when it’s disconnected, probably desktops with dynamic ip addresses, too. But for servers with a static ip address, take your pick, I chose to use eth0.

An example /etc/hosts for a laptop
127.0.0.1 localhost
127.0.1.1 mymailname.com myhostname


Where myhostname is the host name configured in the kernel.

An example /etc/hosts for a server
127.0.0.1 localhost
192.168.1.1 mymailname.com myhostname.mymailname.com someothername myhostname


When the kernel first boots, the host name is configured. How depends on the distribution. Ubuntu seems to use /etc/hostname. In my example /etc/hostname contains myhostname .

When you run the command hostname it does a gethostname() to get the configured host name. It doesn’t reread /etc/hostname every time, it’s stored in the kernel, presumably.

When you want the FQDN, you run hostname -f, it does a gethostname() to get the configured host name, then does a getaddrinfo("myhostname",...) to get the FQDN. Using the last /etc/hosts example, the FQDN is mymailname.com because it’s the first string after the ip address in the line where myhostname appears.

Short names such as the host name myhostname should go at the end of the line, what you want the FQDN to be should be the first string after the ip address, and any other names goes in the middle.

In my examples, I made the FQDN the mail name, because some distributions use the FQDN by default as the mail name. Ubuntu doesn’t, it appears to use /etc/mailname.

Anyway, that’s my understanding based on messing around and reading a man page. Hopefully I’ve explained how the host name and the FQDN is tied together in /etc/hosts.