Archive for the 'Linux' Category


Building Scalable Web Architectures

I just read a very interesting presentation done by Aaron Bannert for ApacheCon 2005. Presentation is on “Building Scalable Web Architectures” and it is a very good reading for anyone interested in high scale web environments. Here is the link to the presentation.


Easy way to read MBR?

10$ question. Sometime ago you have created backup of your systems Master Boot Record (MBR). Now, after some change, you noticed you did a fatal mistake and your partition table is corrupted and you need to recover it from the backup you created, but you are not sure if it is the correct version. The question is, what is the easiest way to read partition table from the backup of your MBR? No, Hex editor is not the easiest way to do it (and it is bad for your eyes :)). I wonder how many of you said ‘file’ command? Yes, magical file command is able to read the data from the mbr dump and prints you the actual partition table. Here is an example from my laptop.

# file mbr.bin
mbr.bin: x86 boot sector;
partition 1: ID=0×83, active, starthead 1, startsector 63, 40949622 sectors;
partition 2: ID=0×82, starthead 254, startsector 40949685, 2088450 sectors;
partition 3: ID=0x8e, starthead 254, startsector 43038135, 74172105 sectors, code offset 0×48

As you can see I have only 3 partitions on the disk. First one has type 0×83, which is HEX id for ext3 type of partition and it is my / partition (you don’t see it here, but I know it :)). It is also active partition, it means that it is used for booting the system. You can also see the size of the partition in sectors. Knowing that one sector has length of 512 bytes you can easily find out the size of the partition.

# echo $(((40949622/2)/1024))
# df -k /
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 19833488 6504676 12305072 35% /

That’s it, correct. :)

Next partition is 0×82 which is swap partition. And last partition is 0x8e which is id for Linux LVM partition.

While I am here, I could also explain what is MBR and how it is used.

MBRMain Boot Record resides in first 512 bytes of your bootable disk. Besides partition table it also holds bootloader and something called a magic number. As you can see on the picture, bootloader takes the biggest part of MBR, whole 446 bytes. During the boot process BIOS search for a bootable devices attached to your system and once it finds it it looks at the MBR and loads the bootloader, also called primary bootloader. Primary bootloader looks at the partition table inside MBR (next 64 bytes after the bootloader) and searches for an active partition. When it finds active partition it loads the secondary boot loader from that partitions boot record which, in turn, loads the kernel, and so on.

Magic number is used for sanity check of your MBR. It holds only 2 bytes and should be 0xAA55.

So, in short words, MBR is used to easily locate and load kernel from the correct device. (It is also used by your operating system to find the layout of the disk, but that is another story.)

PS: You can create a dump of your MBR by issuing next command:

# dd if=/dev/sda of=mbr.bin bs=512 count=1

Replace /dev/sda with the correct address to your disk.

PS 2: Sorry for bad quality of the MBR scheme, but I didn’t have much time to work on it and I am not a graphic designer. :D


Apache proxy for Lighttpd

Everyone who has experience with hosting of popular web sites will agree with the next statement: “Apache is a memory eater!” Yes, Apache can be optimized but optimization goes more into direction of making it perform faster instead of making it resource friendly. One of the ways to overcome this is to use Apache to proxy requests for virtual hosts which generates high load to another “lightweight” web server, while leaving others on Apache.

My choice of lightweight web server is lighttpd. Developed by Jan Kneschke (he also stands behind MySQL-Proxy), lighttpd (pronounced lighty) was created with exactly this in mind – lightweight, high-performance web server with very small memory footprint. And all that he really is. Unlike Apache’s multi-threaded design, lighttpd has only one thread which has two bonuses

  • less consumption of memory – threads are using shared memory, but still every thread needs to reserve some amount of memory for itself
  • saving CPU time – creation of new threads can be very CPU costly

Apache provides modules for various server side technologies (PHP, Python, Ruby, etc). Lighttpd does not provide the same feature, but the same functionality is possible through FastCGI. And this, again, improves it’s performance and memory footprint.

In this small how-to I will describe how lighttpd and Apache are configured to work side by side.

Coming from old “Slackware school” of sys admins, I still sometimes prefer to compile software from source insted of using already built packages. Maybe it is a bad habit but it is hard to quit. :). So, we first need to install everything we need, and that is Apache with proxy support, PHP with FastCGI support and lighttpd with Lua support (needed for various rewrite support).

Building software from source is fairly simple and straight-froward process if you have all the libraries needed. We will start with Apache. Except all standard compile options (–enable-so, –enable-mods-shared=all, –enable-ssl=shared, etc) we will also use options which will include Apache proxy modules. After make install Apache is installed in /usr/local/httpd-2.0.59 folder.

# ./configure < standard apache compile options > \
> –prefix=/usr/local/httpd-2.0.59 \
> –enable-proxy \
> –enable-proxy-connect \
> –enable-proxy-http
# make
# make install

Next PHP. Story is the same as for Apache – besides standard options we are also building support for FastCGI. PHP is installed in /usr/local/php-5.2.3.

# ./configure < standard php compile options > \
> –prefix=/usr/local/php-5.2.3 \
> –enable-fastcgi \
> –enable-discard-path \
> –enable-force-redirect
# make
# make install

And on the end the star of this how-to, lighttpd. Lua support is necessary for mod_magnet which is used for certain rewrite capabilities (I will cover that in some other how-to). PKG_CONFIG_PATH needs to be adjusted if lua.pc is not already in the right location. It can be found in etc folder in Lua source tree. Lighty is installed in /usr/local/lighttpd-1.4.16.

> –prefix=/usr/local/lighttpd-1.4.16 \
> –with-lua \
> –with-memcache
# make
# make install

By now you are probably wondering why I use such insane paths for software installation. This method actually provides easy way to maintain and upgrade software. Once everything is installed you just go to /usr/local and create sym links from these folders to more generic ones and when you perform some upgrade all you need to change is the symlink and you are ready to use the new version. In case you want to revert back to the old version procedure is the same – change the destination of the sym link and you are back. Very simple.

# cd /usr/local/
# ln -sf httpd-2.0.59 httpd
# ln -sf php-5.2.3 php5
# ln -sf lighttpd-1.4.16 lighttpd

Now, let’s configure lighttpd.

Lighttpd comes with an example config file which you can customize to create configuration that will suit your needs. As a matter of fact there are only few things you should change to make lighttpd works. Here is my starting configuration

server.modules = ( “mod_rewrite”,
                   ”mod_magnet” )
server.document-root = “/var/www/html”
server.errorlog = “/var/log/lighttpd/error.log”
accesslog.filename = “/var/log/lighttpd/access.log” = “/var/run/”
url.access-deny = ( “~”, “.inc” )
index-file.names = ( “index.php”,
                     ”default.htm” )
static-file.exclude-extensions = ( “.php”,
                                   ”.fcgi” )
$HTTP["url"] =~ “\.pdf$” {
    server.range-requests = “disable”
server.username = “nobody”
server.groupname = “nogroup”
dir-listing.activate = “enable”

As you can see, I included some additional modules – lighttpd has only mod_access and mod_accesslog activated by default. I also changed paths to main document root, access and error log files and pid file.

Now let’s do some real changes. By default lighty listens on port 80, but that port is already taken by Apache so we will move lighttpd to port 8080 and make it listens only on a loopback interface, no need for outside world connections.

server.bind = “″
server.port = 8080

Path to virtual hosts doc root is in /home/<domain>/public_html so we need to tell lighty where to find them

evhost.path-pattern = “/home/%0/public_html/”

Now when lighttpd receives connection it will look at “Host” part of the HTTP header and check if /home/<host>/public_html exists and serve the proper page from this folder. If the folder does not exists it will try to serve a page from a default document root (we defined this above with server.document-root option).

And now the final part – FastCGI for PHP support.

Lighttpd can use remote FastCGI servers to easen the server load even more, but in this case we don’t need this high-end feature as we will be running FastCGI on the same server. With next options we instruct lighttpd to execute all files with .php extension through the FastCGI server on localhost and we also define the PHP binary what will be used for parsing these files.

fastcgi.server = ( “.php” =>
    ( “localhost” =>
        ”socket” => “/tmp/php-fastcgi.socket”,
        ”bin-path” => “/usr/local/php5/bin/php-cgi”

We can now start lighty.

/etc/init.d/rc.lighttpd start

And that was all for lighttpd, we can now move to Apache configuration.

Configuring Apache is even simplier. We first need to load proxy support.

LoadModule proxy_module modules/
LoadModule proxy_connect_module modules/
LoadModule proxy_http_module modules/

And than in the definition of the virtual host we want to transfer to lighttpd we add these proxy options and simple rewrite rule which will actually do all the proxying necessary.

<VirtualHost *:80>
    ProxyRequests Off
    ProxyPreserveHost On
    RewriteEngine On
    RewriteRule ^/(.*)$1 [P,L]

And that’s it. You web site is now running under lighttpd. And this is very easy to check too.

# telnet 80
Connected to
Escape character is ‘^]’.
GET / HTTP/1.0
HTTP/1.1 301 Moved Permanently
Date: Fri, 24 Aug 2007 17:49:12 GMT
Server: Apache
Content-Length: 235
Content-Type: text/html; charset=iso-8859-1
Via: 1.1 Application and Content Networking System Software 5.3.5
Connection: Close

# telnet 80
Connected to
Escape character is ‘^]’.
GET / HTTP/1.0
HTTP/1.1 302 Found
Date: Fri, 24 Aug 2007 17:51:44 GMT
Server: lighttpd/1.4.16
X-Powered-By: PHP/5.2.3
Content-type: text/html
Content-Length: 0
Via: 1.1 Application and Content Networking System Software 5.3.5
Connection: Close

Note:Only shortcuming of this method is that your PHP scripts will not see the correct IP address of your visitors but rather the IP address of the Apache server. To overcome this you can use $_SERVER['HTTP_X_FORWARDED_FOR'] instead of $_SERVER['REMOTE_ADDR'] and gethostbyaddr($_SERVER['HTTP_X_FORWARDED_FOR']) instead of $_SERVER['REMOTE_HOST']. Also, lighttpd logs will be almost useless for any traffic analysis, you will need to use Apache logs for that.


Logging SSH sessions

One of the main rules when working on production servers is to keep trace of your actions. That is, so called, “cover your ass” policy. :) As most of my colleagues in IBM are using Windows on their workstations (strange isn’t it?!) they are using putty which provides logging options for their SSH sessions. But I am using Linux and OpenSSH client does not provide this luxury so I had to create this short script to save my SSH logs. It will start SSH client with all the parameters you pass over the command line but at the same time it will also start script command and log everything in right log file. Very neat. :)

DATE=$(date +’%Y-%m-%d_%H:%M’)
case “$1″ in
[ ! -d ${LOG_FOLDER} ] && mkdir -p ${LOG_FOLDER}
script -c “ssh ${HOST} $*” ${LOG_FILE}


Linux LDOM

Linux kernel developers announced that they have successfully ported Logical Domain technology from Solaris kernel. Even more, it is already in the kernel! Good job guys! Another big step for the Linux virtualization community.

Speaking about Solaris, Sun opened source of it’s cluster suite. Can we expect some improvements in Linux cluster software?


DIA from SQL

Today I needed a fast way to generate DIA diagram from an MySQL database structure and I found this great little project on SourceForge called, very convenient, sql2dia. This tool is actually a PERL script which generates XML file (yes, if you ever wondered, DIA file format is pure XML. Nice, a? :)) sql2dia does not provide support for relations between tables, but this feature is planed for the future releases. Anyway, it does exactly what I needed it to do – create a diagram of tables in my PostfixAdmin database (don’t ask why… yet :)). Here is an example how it works and the result.

[ home sql2dia ] # ./mysql2dia -D -d maildb -h localhost \
> -u username -p password -o mail.dia
Debug mode activated.
Creating header…
Creating object admin…
Creating object domain…
Creating object vacation…
Creating object mailbox…
Creating object log…
Creating object alias…
Creating object domain_admins…
Creating footer…



Hanging yum

Yesterday I noticed a very strange problem with yum on my Fedora 6 notebook. It just plays dead after “Excluding Packages in global exclude list” message. This was not my first problem with yum but this one is the weirdest so far.
Without a clue what could cause such problem I used strace (tool I usually use when I have no clue why some software fails :)) to see if the process is really dead or it was actually doing something. Here is what I got:

[ home ~ ] # strace -p 12085
Process 12085 attached – interrupt to quit
futex(0x9fcc7b8, FUTEX_WAIT, 1, NULL …>

OK, so it was really stuck. And it was stuck on a futex system call which probaly means that yum didn’t exit cleanly on previous run, although I don’t remember it failed recently. As a matter of fact, two weeks ago I rebooted my notebook for the first time in 70 days (what I am trying to say is that it works perfectly :)

Anyway, simple google search on futex and yum reveals more users with similar problem. It seems that the source of the problem is corruption of RPM databases in /var/lib/rpm folder. Problem can be resolved in two ways. Either remove all ‘__db.*’ files from /var/lib/rpm folder or run /usr/lib/rpm/rpmdb_recover.

I somehow prefer the first solution – it gives more satisfaction. :)


Linux NAT with one network interface?

This morning I encountered a problem trying to setup IP masquerading with my ASUS notebook. Notebook has only one network interface and it is connected to my cable modem. No problem, I thought, I will connect my notebook, cable modem and my wife’s notebook to a small switch I have, and than create an interface alias for my primary network interface, and setup a simple IP masquerading so both notebooks have internet access. Simple as that. Network would look like:

Home Lan

First, I added interface alias so I can communicate with other notebook:

# ifconfig eth0:1 netmask
# ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=128 time=0.177 ms
64 bytes from icmp_seq=2 ttl=128 time=0.159 ms
64 bytes from icmp_seq=3 ttl=128 time=0.148 ms
64 bytes from icmp_seq=4 ttl=128 time=0.198 ms
— ping statistics —
4 packets transmitted, 4 received, 0% packet loss, time 2998ms
rtt min/avg/max/mdev = 0.148/0.170/0.198/0.023 ms

That seems to work. On to the next step. We have to setup NAT between those two computers. The exact process should be, for my configuration:

# echo “1″ > /proc/sys/net/ipv4/ip_forward
# iptables -A FORWARD -i eth0 -o eth0:1 -m state \
# iptables -A FORWARD -i eth0:1 -o eth0 -j ACCEPT
# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

So lets do it:

# echo “1″ > /proc/sys/net/ipv4/ip_forward

No error so far. Next:

# iptables -A FORWARD -i eth0 -o eth0:1 -m state \
Warning: wierd character in interface `eth0:1′
(No aliases, :, ! or *).

What was that?! “Warning: wierd character in interface `eth0:1′ (No aliases, :, ! or *).” Hmm… maybe I made some typo? I don’t think so (although there is a typo in error message, weird is not spelled like that :)). Seems that NetFilter is not able to do NAT with only one interface, which is pretty reasonable. But, cheap as I am, I had no intention of buying new PCMCIA network card, so I had to find a way of doing this with only one interface.

After hour spent searching the google for answer, I started getting some ideas. If we can not use interface alias maybe we can use IP address of the other node in the network insted of it. Seemed like a wild guess but I was getting pretty desperate. :)

# iptables -A FORWARD -i eth0 -d -m state \

No error. That’s a good sign. So, let’s try the rest.

# iptables -A FORWARD -s -o eth0 -j ACCEPT
# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Again, no error. Let’s test it from the other node.

# ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=241 time=142 ms
64 bytes from icmp_seq=2 ttl=241 time=141 ms
64 bytes from icmp_seq=3 ttl=241 time=143 ms
— ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 141.888/142.763/143.515/0.669 ms

Wow, it works. :) Sometimes being cheap pays off. :) So, one problem down. What about port forwarding. My wife is a huge e-mule fan, and if her incoming e-mule port is blocked she will get LowID which makes downloading painful. Again, first I tried iptables.

# iptables -t nat -A PREROUTING -p tcp -i eth0 \
> -d –dport 34567 -j DNAT –to
# iptables -A FORWARD -p tcp -i eth0 -d \
> –dport 34567 -j ACCEPT

And for some reason this was not working. At this point I gave up and tried something else. Something that works. :-p

I’m sure most of you know that one of the features of SSH is port forwarding.

# ssh -g -N -L 34567: localhost
root@localhost’s password:

And that’s it. With this we forwarded port 34567 on localhost to port 34567 on other notebook with address But there is still one problem. This way port forwarding can not be started at system boot since it demands user interaction (ssh asks for password when started). That problem is also easy to solve. We need to generate key that we will use instead of password.

# ssh-keygen -t dsa -b 1024 -f /root/ssh_key
Generating public/private dsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/ssh_key.
Your public key has been saved in /root/
The key fingerprint is:
c7:05:e7:e6:38:77:82:79:a1:d9:b5:ab:2f:3a:84:02 root@localhost

Now all we need to do is to copy /root/ into /root/.ssh/authorized_keys and next time we try to login using this key we will not be asked for password.

# cat /root/ >>/root/.ssh/authorized_keys

For the sake of security we can restrict login with this key only to localhost, so our /root/.ssh/authorized_keys will look like:

from=”″ ssh-dss AAAAB3NzaC1kc3MAAACBAKtVMMBIqjAmXKkk…

We can test it to see if it works.

# telnet 34567
telnet: connect to address Connection refused
telnet: Unable to connect to remote host: Connection refused
# ssh -g -N -L 34567: \
> -i /root/ssh_key
root@′s password: ^C
# ssh -g -N -L 34567: localhost \
> -i /root/ssh_key # telnet 34567
Connected to (
Escape character is ‘^]’.
Connection closed by foreign host.

That’s all folks. :)