32 * 2 = 16h

Last week I had an interesting assignment, upgrading one AIX 5.2 server from 32bit to 64bit kernel. Process should be pretty straight forward and is very nicely explained in AIX documentation, but as usual, all actions that require application stopping have to be done after working hours – in this case after 9pm. Considering that all changes, system reboot and application start/stop sequence should not take more than 45 minutes this is not a big problem. As many times before, I didn’t count on good ol’ friend of all system administrators – Murphy.

But, let’s start from the start. First thing I did was to check if the server supports 64bit environment and what version of the kernel is currently running.

# bootinfo -y
# bootinfo -K

So, the hardware on this server is 64bit (as expected) and active kernel is 32bit. Now, let’s stop applications. Only important application on this server is a production Oracle database. We have to stop it before reboot. (Important thing to note at this moment is the version of database, it is old release of Oracle.)

# su – oracle
% sqlplus /nolog
SQL*Plus: Release – Production on Wed Jul 4 21:01:20 2007
(c) Copyright 2000 Oracle Corporation. All rights reserved.
SQL> conn / as sysdba
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle8i Enterprise Edition Release – Production
JServer Release – Production

In order to be able to execute 64bit binaries we must edit /etc/inittab so the syscall64 kernel extension is loaded during the boot. This is need even with 64bit kernel.

# mkitab “load64bit:2:wait:/etc/methods/cfg64 >/dev/console 2>&1″

The switch to 64bit kernel is done by simply relinking paths to the kernel and libraries, and updating boot image on the boot device. Followed by a reboot. Simple as that.

# ln -sf /usr/lib/boot/unix_64 /unix
# ln -sf /usr/lib/boot/unix_64 /usr/lib/boot/unix
# bosboot -a
# shutdown -Fr

After the reboot, I checked the version of running kernel to see if the change actually took place.

# bootinfo -K

Perfect! so simple isn’t it. I just love when things go so smoothly. Now let’s start Oracle.

# su – oracle
% sqlplus /nolog
Could not load program sqlplus:
Symbol resolution failed for sqlplus because:
Symbol pw_post (number 272) is not exported from dependent module /unix.
Symbol pw_wait (number 273) is not exported from dependent module /unix.
Symbol pw_config (number 274) is not exported from dependent module /unix.
Symbol aix_ora_pw_version3_required (number 275) is not exported from dependent module /unix.
Examine .loader section symbols with the ‘dump -Tv’ command.

“Argh, this can’t be happening!” I was thinking, so I tried again. Surprisingly, that didn’t help. After the initial shock, I looked at the message more carefully and tried to figure out what the hell it meant. Kernel doesn’t support necessary Oracle symbols – so maybe the Oracle kernel extension is not loaded, let’s check.

# loadext -r
Oracle Kernel Extension Loader for AIX
Copyright (c) 1998,1999 Oracle Corporation
sh: /usr/sbin/crash: not found
No Kernel Extension is currently running.

I was on a right trail. But this is strange, Oracle kernel extension is loaded from /etc/inittab during the boot, it SHOULD be loaded. Maybe the inittab got corrupted.

# lsitab -a|grep ora
orapw:2:wait:/etc/loadext -l /etc/pw-syscall

It is there. In the agony I thought maybe syscall64 extension was not loaded so it failed (although it should not matter).

# genkex|grep syscall
4635e70 390 /usr/lib/drivers/syscalls64.ext

It is there. Let’s try to call it manually, maybe it will work now.

# loadext -l /etc/pw-syscall
Oracle Kernel Extension Loader for AIX
Copyright (c) 1998,1999 Oracle Corporation
Kernel Extension Version: 3
SYS_SINGLELOAD: Exec format error
kmid: 0 (0×0)
path: ‘/etc/pw-syscall’
libpath: ”

Maybe, this extension does not support 64bit environment?

# strings /etc/pw-syscall|head -3
Kernel Extension Version: 3
$Revision: 1.9 $
Supported Oracle Instances: 32-bit & 64-bit

Now I am puzzled even more.

At this point I felt stuck. Reverting back to 32bit kernel was not even an option as this was only one part of the big migration process on this server. But, on the other hand Oracle has to be up and running by morning – this is a very important production server. As I am not an Oracle guru and there was no one from DB team around to ask for advice, I asked Google for help. As many times before, it proved to be wise choice. People already had this problem and solved it by applying small patch for Oracle.

Important thing here is that Oracle version 8 does not support 64bit kernel on AIX. It requires patch number 2896876 in order to do so.

After applying this patch you get a new kernel extension which loads without complaining.

# genkex|grep syscall
466c850 1218 /etc/pw-syscall64
4641ec0 390 /usr/lib/drivers/syscalls64.ext

Now, let’s try to start Oracle.

# su – oracle
% sqlplus /nolog
SQL*Plus: Release – Production on Thu Jul 5 00:47:45 2007
(c) Copyright 2000 Oracle Corporation. All rights reserved.
SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area  178704276 bytes
Fixed Size                    73620 bytes
Variable Size             135630848 bytes
Database Buffers           41943040 bytes
Redo Buffers                1056768 bytes
Database mounted.
Database opened.
SQL> exit
% ^D

Nice. :) Next thing is to change inittab to load new Oracle kernel extension,

# chitab “orapw:2:wait:/etc/loadext -l /etc/pw-syscall64″

stop oracle and reboot server again to see how it will behave after the reboot. Luckily everything works fine so at 01am I can finally go home. It was about time since I was there for almost 16 hours (hence the subject of the post.) Ah, the pleasures of being a system administrators are flexible working hours, isn’t it? :)

1 Response to “32 * 2 = 16h”

  1. 1 Lucian Oct 26th, 2007 at 10:22 am

    great job man , they should pay you less

Comments are currently closed.