30
Nov

PS1 for your Shell?

By miljan 2 Comments
Categories: General, Linux, Shell and UNIX
Few years ago I went on a quest to find a perfect shell prompt. I asked the mighty Internets for ideas, but it seemed futile. I tried many things, simple prompts, complex prompts, but nothing could satisfy my requirements (I don't even remember what were my requirements back then.) So I picked best of both worlds and got this little monster.

:) oscar:~#

Happy face! And in case of an error, it looks sad.

:( 2 oscar:~#

Cute, a? It even prints the exit code. Useful and cute at the same time! And here is definition of the prompt. As you can see it uses simple function to determine return code of executed command and adjust its feelings accordingly.

smiley() {
    RC=$?
    [[ ${RC} == 0 ]] && echo ':)' || echo ":( ${RC}"
}
export PS1="\$(smiley) \u@\h:\w\\$ "

I think I got the idea for the smiley thing somewhere, but unfortunately I don't remember anymore where from.
For more adventurous people, maybe this prompt would be more interesting.

export PS1="\[\033[0;36m\]\033(0l\033(B\[\033[0m\][\[\033[1;31m\]\u\[\033[0m\]]\[\033[0;36m\]\033(0q\033(B\[\033[0m\][\[\033[1;33m\]@\h\[\033[0m\]]\[\033[0;36m\]\033(0q\033(B\[\033[0m\][\[\033[0;37m\]\T\[\033[0m\]]\[\033[0;36m\]\033(0q\033(B\033(0q\033(B\033(0q\033(B\033(0q\033(B\033(0q\033(B\033(0q\033(B\033(0q\033(B\033(0q\033(B\[\033[0m\][\[\033[1;33m\]\w\[\033[0m\]]\n\[\033[0;36m\]\033(0m\033(B\[\033[0m\]\$ "

Magic! :)

What is your favorite prompt? Please leave boring \u@\h:\w for dinner with your parents. :P
23
Nov

Load Average > 680

By miljan 6 Comments
Categories: General, Linux and News

I just found this old screenshot from one of my previous jobs. It was taken on December 9th 2003, while one of the web hosting servers went woowoo due to badly optimized web site. Load average went sky high to 682! Anyone else had such a high load before or am I the absolute champion? :)

PS: I was using Fluxbox back than! Wow, crazy youth! :)

16
Nov

LVM Snapshots and XFS

By miljan 2 Comments
Categories: Linux
Small note for everyone planing to use Linux LVM snapshots and using XFS at the same time. XFS has UUIDs which are unique identifiers of the filesystem. Two file systems with same UUID can not be mounted on the same server. Now, if we know that a snapshot of a logical volume represents a point-in-time copy of the original logical volume it doesn't take much time to realize that the filesystem on the snapshot is also a copy, thus it will have the same UUID as the filesystem on the original logical volume. So here is what happens when you try to mount the snapshot:

[root@server ~]# df -hT /var/lib/mysql_backup/
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/sys-mysql_backup
              xfs    25G   19G  6.3G  76% /var/lib/mysql_backup
 
[root@server ~]# lvcreate -s -n bkp-snap -L1G /dev/sys/mysql_backup
  Logical volume "bkp-snap" created
 
[root@server ~]# mount /dev/sys/bkp-snap /mnt/misc/
mount: wrong fs type, bad option, bad superblock on /dev/sys/bkp-snap,
       missing codepage or other
       In some cases useful info is found in syslog - try
       dmesg | tail or so
 
[root@server ~]# dmesg | tail -1
XFS: Filesystem dm-9 has duplicate UUID - can't mount

It doesn't look very good. There are two solution for this problem. One is to use nouuid option for mount command.

[root@server ~]# mount -o nouuid /dev/sys/bkp-snap /mnt/misc/
 
[root@server ~]# dmesg | tail -3
XFS mounting filesystem dm-9
Starting XFS recovery on filesystem: dm-9 (logdev: internal)
Ending XFS recovery on filesystem: dm-9 (logdev: internal)

Another option would be to change UUID of the filesystem on the snapshot using xfs_admin command.

[root@server ~]# xfs_admin -U generate /dev/sys/bkp-snap
Clearing log and setting UUID
writing all SBs
new UUID = 1bdcf6e1-62fb-47f2-83e4-dc398bb7a1cd
 
[root@server ~]# dmesg | tail -2
XFS mounting filesystem dm-9
Ending clean XFS mount for filesystem: dm-9

I am in favour of the first option (mount -o nouuid) since it does not perform any modification on the filesystem. It just feels safer, that's all... :)
10
Nov

More information on Linux memory management

By miljan 2 Comments
Categories: Linux

I have just noticed that I missed to mention one very important thing in my previous post.

File /proc/meminfo contains a very useful field named Committed_AS. This field indicates TOTAL value of committed memory. If all applications would require all memory allocated to them your server would need this amount of memory.

If we look in the example from my previous post we would find following values:

loreto:/tmp # cat /proc/meminfo
MemTotal:     33274944 kB
…
Committed_AS: 49751960 kB
…

So my server has 32GB of RAM, but total amount of memory allocated is 48GB. That is 150%! If all this memory would be required at once server would crash pretty bad (or OOM killer would start butchering my Oracle databases to get some memory back!). :-)

10
Jul

Linux memory management

By miljan 0 Comments
Categories: Linux

Yesterday I had a request for memory usage report on Oracle servers in my company. As we are using Centreon, Nagios frontend which makes good use of performance data reported by Nagios plugins and makes nice graphs out of it, it was a matter of pasting the images into the mail and sending it. But than interesting question was raised: how come on a server with 32GB of RAM and with 30+ databases running, only 5GB of RAM is reported as used? Strange indeed.

I quickly logged in to server and checked memory usage:

loreto:/tmp # free
             total       used       free    shared   buffers     cached
Mem:      33274944   32931032     343912         0        20   27013200
-/+ buffers/cache:    5917812   27357132
Swap:     16779884    5603256   11176628

Really only 5GB, check_memory plugin was not wrong. Next this I checked were shared memory segments – Oracle uses shared memory in huge quantities, so this is also very important parameter.

loreto:/tmp # a=0; for i in $(ipcs -m|grep ^0x|awk ‘{print $5}’); do let a+=$i; done; echo $a
20443037885

Ugh, 20GB allocated for shared memory, while system reports only 5GB. Something is very wrong here. Confused, I took out the artillery.

loreto:/tmp # cat /proc/meminfo
MemTotal:     33274944 kB
MemFree:        198580 kB
Buffers:            20 kB
Cached:       27439580 kB
SwapCached:     223880 kB
Active:       16333936 kB
Inactive:     15428724 kB
HighTotal:    32634140 kB
HighFree:        33516 kB
LowTotal:       640804 kB
LowFree:        165064 kB
SwapTotal:    16779884 kB
SwapFree:     10856700 kB
Dirty:            1668 kB
Writeback:           0 kB
AnonPages:     4089416 kB
Mapped:       10222968 kB
Slab:           427584 kB
CommitLimit:  33417356 kB
Committed_AS: 49751960 kB
PageTables:     826016 kB
VmallocTotal:   112632 kB
VmallocUsed:     22228 kB
VmallocChunk:    90180 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Usually, used memory on Linux is calculated as (Total Memory – (Unused Memory + Buffers + Page Cache)). Why buffers and caches are not counted into memory usage? Simply because it contains data that is not really critical for operating system and applications running. It contains data that can be flushed and removed from the memory at any time.

So in my case that was:

33274944 – (198580 + 20 + 27439580) = 5636764

OK, this matches output from free command. But what about those 20GB of allocated shared memory?

Next few hours I spent in searching and reading Linux documentation on memory management and found few interesting things.

Linux uses principle of memory overcommitment. Basically, what this means is that when application requests memory to be allocated, kernel will always “give: the memory hoping that application will not really use it, or at least not the whole size allocated. Only when application tries to write the data into the memory, kernel will mark the memory as used. This can lead to situation where the size of allocated memory is actually higher than the size of physical memory inside the machine. But as long as there is no demand for allocated memory, system is running without problems.

And this is the core of my dilemma. Shared memory is allocated, but since there is no data in it, it is not counted into used memory.

Memory overcommit can be configured via two parameters:

loreto:/tmp # sysctl -a|grep overcommit
vm.overcommit_ratio = 50
vm.overcommit_memory = 0

From Red Hat manual:

  • overcommit_memory — Configures the conditions under which a large memory request is accepted or denied. The following three modes are available:
    • 0 — The kernel performs heuristic memory over commit handling by estimating the amount of memory available and failing requests that are blatantly invalid. Unfortunately, since memory is allocated using a heuristic rather than a precise algorithm, this setting can sometimes allow available memory on the system to be overloaded. This is the default setting.
    • 1 — The kernel performs no memory over commit handling. Under this setting, the potential for memory overload is increased, but so is performance for memory intensive tasks (such as those executed by some scientific software).
    • 2 — The kernel fails requests for memory that add up to all of swap plus the percent of physical RAM specified in /proc/sys/vm/overcommit_ratio. This setting is best for those who desire less risk of memory overcommitment.
      Note This setting is only recommended for systems with swap areas larger than physical memory.
  • overcommit_ratio — Specifies the percentage of physical RAM considered when /proc/sys/vm/overcommit_memory is set to 2. The default value is 50.
10
Oct

Pidgin status v2

By miljan 1 Comment
Categories: Linux, News, Pidgin and Python

As requested by bleketux, I made some modifications to pidgin_status.py script.

Main news is that now it is possible to change Pidgin status message periodically. Script will go to background (it is a real daemon now :P), change the status, and wait for the set time interval until it changes the message again, and then all over again, wait-change-wait-change.

To change status message every 5 minutes with a random line from file /home/miljan/quotes/dusko_radovic.txt:

pidgin_status.py -d -t 5 -f /home/miljan/quote/dusko_radovic.txt

And in Pidgin you would get something like this every five minutes:

To show the song you are listening to as status message:

pidgin_status.py -s “Mukeka di Rato – Kustapassaaessedrmobral”

And in Pidgin it would look like:

You can see all possible options by running script with -h argument for help.

bleketux, I hope you are still around to enjoy this. ;)

Download: pidgin_status_v2.py

25
Sep

American SPAM

By miljan 0 Comments
Categories: News and Web

This is so funny that I have to post it. :)

We all know about Nigerian SPAM, but how about American SPAM? :o)

25
Mar

Why I love strace

By miljan 2 Comments
Categories: Linux, Shell and UNIX

Strace is a tool that should be in a toolbox of every system administrator. Not only that it can help in troubleshooting simple problems (ie. missing libraries in newly created chroot, which ldd mysteriously misses to report) but it also helps in debugging very complex system problems and performance issues.

Recently I experienced a very strange problem with one of the RHEL 3 servers we’ve got. Problem manifested in a very strange way, SSH and su logins hanged, other daemons were also hanging during the startup, only way to reboot or shutdown the server was to physically press the restart/power off button, etc. All this could have been caused by problems on both software and hardware level. First suspicious was bad RAID controller, but after tests this proved to be a mislead. After more tests and brainstorms hardware problems were definitely excluded, so problem has to be on the software side. But what could be the problem?

After few more misleading steps I tried to trace system calls created by su command and found very interesting results.

$ strace -f -s 1024 -o /tmp/su.strace.out su -
[-- cut --]
3138 open(”/dev/audit”, O_RDWR) = 3
3138 fcntl64(3, F_GETFD) = 0
3138 fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
3138 ioctl(3, 0×801c406f

And this is where the strace output ends and su command hangs. Audit device file is opened (file descriptor 3) and as soon as the first request is dispatched to this device (ioctl system call to file descriptor 3) command freezes. According to this I should just disable audit on the server and the problem will be gone.
As a test, audit daemon was temporarily stopped and I tried to switch to another user and the problem was indeed gone.

After searching for similar problems with audit daemon I found an article in Red Hat knowledge base regarding the exactly same
issue (http://kbase.redhat.com/faq/FAQ_79_6169.shtm).
From the article:

When the free space in the filesystem holding the audit logs is less than 20%, the above notify command will error out and auditd will enter suspend mode. This causes all system calls to block.

So this behavior is not a bug but actual feature of the software. :o) From security point of view this is expected behaviour – attacker could fill up filesystem where audit logs are stored before the attack and audit will be disabled, meaning no logs of his activity, so better not to allow ANY activity on the system if audit is not able to write to its logs. But still, this kind of behaviour renders the system completely useless to legitimate users.

The topic of this post is not audit, so I will stop here. Important thing is that strace led us directly to the main source of the problem. Resolution of issues like this would be much more complex and time consuming without this great little tool. :)

09
Mar

Setting Pidgin Status with Python or How to Waste Perfectly Good Saturday

By miljan 19 Comments
Categories: Linux, Pidgin and Python

I was very bored today. Tired from working on Ratuus (don’t go there, site is under heavy construction :)) I needed something to help me take my mind off everything. And what better way to do it, than playing with Python, Pidgin and D-BUS. :D

To cut the long story short, I needed something that will update my Pidgin status message with the information about the current song I am listening. Till recently I was using Rhythmbox player and there is a perfect little Pidgin plugin called Current Track that worked with this player. Last week I discovered gmusicbrowser and fell in love immediately. It is fast, rich with functionalities but still simple to use. Exactly what I want from audio player. (Hm, I just noticed it is written in PERL. Now when Python is used for everything this comes as a big surprise.)

gmusicbrowser already has a plugin called NowPlaying. It will trigger some command whenever song is changed. I just needed to write this command that will inform Pidgin about the change. So, this seemed like a perfect exercise for slow Saturday. :)

Quick search on Pidgin and D-BUS showed extensive documentation about Pidgin API accessible through D-BUS. There is even a working example of how to change the status message! :)

But that was too simple, so I got another idea. Some time ago, I wrote a small daemon in C that will bind to a specific port and display random bofh-excuses fortune messages when someone would telnet to it. (Seems like I have a lot of spare time. I should really find some hobby!) Something similar to telnet bofh.jeffballard.us 666 (here for more information). So I was thinking about implementing the same for my Pidgin status. Random BOFH excuses in your status message! How cool geeky is that!

The result of all that is short (~60 lines of code) Python script that will set your Pidgin status message to:

a) you current song

pidgin_status.py -m The Real McKenzies – Outta Scotch

b) random line from a file

pidgin_status.py -f /usr/local/share/bofh-example

c) anything you give as the command line argument

pidgin_status.py Some very interesting and funny status message

Only difference between a) and c) is the type of the icon that will be shown. In example a) there will be a small musical note, while in example b) and c) nice arrow pointing to right side will be show.

In the middle of testing I noticed this strange message:

Being from Serbia myself, I find this extremely funny. Although, I didn’t know Serbian hackers are so notorious! :)

I hope someone will find it useful. In any case, I am accepting donations for some long and adventurous vacation. As you can see, I really need it! :D

pidgin_status.py
bofh-excuses

13
Sep

Building Scalable Web Architectures

By miljan 1 Comment
Categories: Apache, Linux, News and Web

I just read a very interesting presentation done by Aaron Bannert for ApacheCon 2005. Presentation is on “Building Scalable Web Architectures” and it is a very good reading for anyone interested in high scale web environments. Here is the link to the presentation.

« Previous Entries