Often the server admin has little control over the applications which uses MySQL and it is hard to find the bottlenecks. This blog post can’t bring the peace in the world, or help NASA to finally land on the Mars. Instead those tasks, I’ll try to solve something else and present my own experiences with MySQL storage engines (at least for MyISAM and InnoDB as the most popular).

Keep in mind that I’m not an expert in this field. When I have problems with MySQL, I dig with my both hands (and head) to solve them without casualties. Every app has its own queries and you’ll need to track them down via slow query log.

Before you do anything, keep in mind that default MySQL installation usually works fine. Unfortunately maximum performances can’t be achieved without adjustments to increase performance and stability.

Before you can do anything, you need to enable slow query log.

Exec

# touch /var/log/slow-query.log

to create slow-query.log file. Then add

log-slow-queries = /var/log/slow-query.log
long_query_time = 4
log-queries-not-using-indexes

(for MySQL 5.0.x)

to /etc/my.cnf (inside [mysqld] section) and restart mysql. Wait at least 24-48 hours before you proceed to the next step.

(For MySQL 5.1.x check here: LINK)

High performance tuning scripts

Mysqltuner

MySQLTuner is a script written in Perl that will assist you with your MySQL configuration and make recommendations for increased performance and stability.

# wget mysqltuner.pl
# chmod +x mysqltuner.pl
# ./mysqltuner.pl

Read output and try to follow the recommendations.

Tuning-primer

Tuning-primer is another script who takes information from “SHOW STATUS LIKE…” and “SHOW VARIABLES LIKE…” to produce readable recommendations for tuning server variables.

# wget http://www.day32.com/MySQL/tuning-primer.sh
# ./tuning-primer.sh

Also, read output and try to see what you can do to fix problems. Keep in mind that you need to restart mysql after you add something to /etc/my.cnf. In case something goes wrong, check mysql log (usually /var/log/mysqld.log

MyTOP

Mytop is a console-based (non-gui) tool for monitoring the threads and overall performance of a MySQL

To install mytop,

# wget http://jeremy.zawodny.com/mysql/mytop/mytop-1.6.tar.gz
# tar -xvzf mytop-1.6.tar.gz
# cd mytop-1.6
# perl Makefile.PL
# make
# make install

In case some Perl libraries are missing, you should install them via cpan.

For example, Term::ReadKey is required and you can get it via cpan.

# cpan
(pres Enter several times until you get cpan shell)
cpan> install Term::ReadKey
cpan> quit

Usage: # mytop -d DATABASE -u USERNAME -p PASSWORD

(replace the DATABASE, USERNAME and PASSWORD with your parameters.

MySQLReport

mysqlreport makes a friendly report of important MySQL status values. mysqlreport transforms the values from SHOW STATUS into an easy-to-read report that provides an in-depth understanding of how well MySQL is running. mysqlreport is a better alternative (and practically the only alternative) to manually interpreting SHOW STATUS.

# wget http://hackmysql.com/mysqlreport
# chmod +x mysqlreport
# ./mysqlreport --user root --password

Enter password for root and check output. More info can be found here. LINK

In this post I will say something about FreeRadius config files, database connection, basic instruction how to insert user in database, etc. Before you step inside this post, I recommend reading part 1 and part 2.

I suppose you’re using RH based distros (Red Hat, CentOS, Fedora,..) and you already installed FreeRadius from source (config files are located in /usr/local/etc/raddb/). Now lets get back to FreeRadius source dir (the place where you extracted the tar.gz).

Inside redhat dir you can find freeradius-radiusd-init script which can be used for easy start/stop radiusd process. Copy this script to /etc/init.d/ dir

# cp freeradius-radiusd-init /etc/init.d/radiusd

Now open /etc/init.d/radiusd script and change the next lines

exec=${exec:=/usr/sbin/$prog}
config_dir=${config_dir:=/etc/raddb}
config=${config:=$config_dir/radiusd.conf}
pidfile=${pidfile:=/var/run/$prog/$prog.pid}
lockfile=${lockfile:=/var/lock/subsys/radiusd}

into

exec=${exec:=/usr/local/sbin/$prog}
config_dir=${config_dir:=/usr/local/etc/raddb}
config=${config:=$config_dir/radiusd.conf}
pidfile=${pidfile:=/usr/local/var/run/$prog/$prog.pid}
lockfile=${lockfile:=/var/lock/subsys/radiusd}

Save changes and exit from editor. (Notice above that we actually changed the path from / to /usr/local/)

Now you can easily start/stop radiusd process.

[root@ms /]# service radiusd
Usage: /etc/init.d/radiusd {start|stop|status|restart|condrestart|try-restart|reload|force-reload}

Also, you can exec chkconfig –level 235 radiusd on to start radiusd on boot.

Now lets get back to our setup.

I suppose you have at least one NAS (A Network Access Server (NAS) is a system that provides access to a network. In some cases also known as a Terminal Server or Remote Access Server (RAS).) NAS is a CLIENT for your radiusd server so please do not mess users and clients. Freeradius doesn’t interact with your users directly so “radius client” is another term for NAS.

The first step is to add your NAS to client list and to create a unique password. Inside clients.conf (/usr/local/etc/raddb/clients.conf) you can find the next lines

#client 192.168.0.0/24 {
#       secret          = testing123-1
#       shortname       = private-network-1
#}

Uncomment those lines and set up client IP address according to your addresses. In the example shown above, all IPs from 192.168.0.0/24 network will be able to use your radiusd server.

You can allow any IP with

client 0.0.0.0/0 {
       secret          = mysecret
       shortname       = myNAS
}

which means all IPs in the world can use my radius server (which is not recommended)…

To allow only one IP (in this case 192.168.0.15),

client 192.168.0.15 {
       secret          = mysecret
       shortname       = myNAS
}

Delete user Cleartext-Password := “password” line from users because we don’t need this any more.

Stop radiusd and start in debugging mode (radiusd -X).

You should see the similar lines

...............
radiusd: #### Loading Clients ####
 client localhost {
        ipaddr = 127.0.0.1
        require_message_authenticator = no
        secret = "testing123"
        nastype = "other"
 }
 client 192.168.0.15 {
        require_message_authenticator = no
        secret = "mysecret"
        shortname = "myNAS"
 }
...........

This means that radiusd will allow NAS with IP address 192.168.0.15 and secret mysecret. Ctrl+C to stop radiusd.

In case you want to use MySQL with freeradius, you should do the next steps. Before anything, you need to create a database for freeradius.

Connect as root to your mysql and exec next queries.

CREATE USER 'radius'@'localhost' IDENTIFIED BY  'radpass';
GRANT USAGE ON * . * TO  'radius'@'localhost' IDENTIFIED BY  'radpass';
CREATE DATABASE IF NOT EXISTS  `radius` ;
GRANT ALL PRIVILEGES ON  `radius` . * TO  'radius'@'localhost';

Another option is to use admin.sql script from raddb/sql/mysql dir.

CREATE USER 'radius'@'localhost';
SET PASSWORD FOR 'radius'@'localhost' = PASSWORD('radpass');
GRANT SELECT ON radius.* TO 'radius'@'localhost';
GRANT ALL ON radius.radacct TO 'radius'@'localhost';
GRANT ALL ON radius.radpostauth TO 'radius'@'localhost';

This script will set a little bit safer permissions where radius will be able only to write radacct and radpostauth tables. (Do not forget to change default username/pass shown above).

The next step is to import default Freeradius tables (the sql files can be found inside raddb/sql/mysql dir). You should import nas.sql and schema.sql. The nas.sql will create a table for your NASes. It is much easier to maintain the NAS list inside database then inside clients.conf. Also, you can add more fields to nas table so you can do other operations with your NAS.

After this operations you should have something like:

[root@ms mysql]# mysql -u radius -p
Enter password:
Welcome TO the MySQL monitor.  Commands END WITH ; OR \g.
Your MySQL connection id IS 23387
Server version: 5.0.77-log SOURCE distribution
 
TYPE 'help;' OR '\h' FOR help. TYPE '\c' TO clear the buffer.
 
mysql> USE radius;
Reading TABLE information FOR completion OF TABLE AND COLUMN names
You can turn off this feature TO GET a quicker startup WITH -A
 
DATABASE changed
mysql> SHOW TABLES;
+------------------+
| Tables_in_radius |
+------------------+
| nas              |
| radacct          |
| radcheck         |
| radgroupcheck    |
| radgroupreply    |
| radpostauth      |
| radreply         |
| radusergroup     |
+------------------+
8 ROWS IN SET (0.00 sec)
 
mysql>

Now we have a working database and we need to configure FreeRadius to use SQL.

radiusd.conf

Open radiusd.conf file (/usr/local/etc/raddb/radiusd.conf), and uncomment $INCLUDE sql.conf line inside modules section. Save changes and exit.

sql.conf

Open sql.conf and edit next lines

        # Connection info:
        server = "localhost"
        #port = 3306
        login = "radius"
        password = "radpass"
 
        # Database table configuration for everything except Oracle
        radius_db = "radius"

to fit your settings (database name, username and password).

dialup.conf

Then open /usr/local/etc/raddb/sql/mysql/dialup.conf and find the next lines (near the end)

 # Uncomment simul_count_query to enable simultaneous use checking
        simul_count_query = "SELECT COUNT(*) \
                             FROM ${acct_table1} \
                             WHERE username = '%{SQL-User-Name}' \
                             AND acctstoptime IS NULL"

Sometimes you will need to check users for simultaneous use and uncommenting sql in session section and uncommenting the query shown above will help you to do this.

default

Now open /usr/local/etc/raddb/sites-available/default and uncomment sql lines inside authorize, accounting and session sections. You can uncomment sql inside post-auth section too if you want to log login attempts (notice that this is not recommended for production servers. Your database can grow and eat up all free space in case someone tries to brute force your NAS.).

Then comment the next lines: files inside authorize section, detail, unix and radutmp inside accounting section and radutmp inside session section.

Please note that those lines we commented above are not important for now and commenting those lines can improve performance. Also, note that detail should remain uncommented in case you want to create ‘detail’ed log of the packets for accounting requests. You will need this in case you want to proxy accounting to another server.

Then save the file and check your config with radiusd -X (debugging mode).

After this you should see something like

rlm_sql (sql): Driver rlm_sql_mysql (module rlm_sql_mysql) loaded and linked
rlm_sql (sql): Attempting to connect to radius@localhost:/radius
rlm_sql (sql): starting 0
rlm_sql (sql): Attempting to connect rlm_sql_mysql #0
rlm_sql_mysql: Starting connect to MySQL server for #0
rlm_sql (sql): Connected new DB handle, #0
rlm_sql (sql): starting 1
rlm_sql (sql): Attempting to connect rlm_sql_mysql #1
rlm_sql_mysql: Starting connect to MySQL server for #1
rlm_sql (sql): Connected new DB handle, #1
rlm_sql (sql): starting 2
rlm_sql (sql): Attempting to connect rlm_sql_mysql #2
rlm_sql_mysql: Starting connect to MySQL server for #2
rlm_sql (sql): Connected new DB handle, #2
rlm_sql (sql): starting 3
rlm_sql (sql): Attempting to connect rlm_sql_mysql #3
rlm_sql_mysql: Starting connect to MySQL server for #3
rlm_sql (sql): Connected new DB handle, #3
rlm_sql (sql): starting 4
rlm_sql (sql): Attempting to connect rlm_sql_mysql #4
rlm_sql_mysql: Starting connect to MySQL server for #4
rlm_sql (sql): Connected new DB handle, #4

which means your freeradius server successfully connected to MySQL database.

There are hundreds of options inside the files shown above and it is impossible to explain all of them. Read comments inside config files and try to figure yourself about them. If you’re using another database scheme, you will need to set up sql.conf and dialup.conf according to your tables. All parameters are editable and it is very easy to understand them. For example if you have a large number on users (1000-xxxx) open sql.conf and increase num_sql_socks from 5 to 15 or 20.

You should not change/delete any other lines in the config file without reading and understanding the comments!

Populating tables and testing

This is the most important part. Before you continue, you need to know what actually do you want from FreeRadius. Which kind of connection do you expect, etc. Also, you need to know something about tables, attributes, operators, etc.

This is it for now…. Next time we will add some users inside database and see what we can do.

Stay tuned…

Few days ago one client called and asked about high CPU load on his Fedora server…
It was very easy to detect that CPU is consumed by kipmi0 process. Unfortunately classic commands kill, pkill, kill -9, … didn’t help.

The problem was in loaded modules (lsmod command)

ipmi_si 38349 0
ipmi_msghandler 32665 1 ipmi_si

I didn’t have enough time to investigate about this but simple rmmod ipmi_msghandler and rmmod ipmi_si was enough to solve this problem. Well I was wrong that this will solve the problem… After reboot it was all like before… The modules are still here and kipmi0 process was too aggressive…

Searching on Google about this problem didn’t help a lot. After some digging, I found out that the lm_sensors is responsible for loading these modules.

nano /etc/sysconfig/lm_sensors gave me the answer. At the end, there are few lines

MODULE_0=ipmi-si
MODULE_1=ipmisensors
MODULE_2=coretemp

Commenting those lines was enough to solve this problem. I suppose that this server is too old (Fedora 6) and some incompatibility exists between hardware platform and those modules…

If you have any info about this problem, please drop a comment so we can all learn something…

If you have a big network with the hundreds of hosts you can expect “Neighbour table overflow” error which occurs in large networks when there are two many ARP requests which the server is not able to reply. For example you’re using server as a DHCP server, cable modems provisioning, etc.

Nov 10 03:18:17 myhost Neighbour table overflow.
Nov 10 03:18:23 myhost printk: 12 messages suppressed.

Of curse, this can be fixed. The solution is to increase the threshhold values in /etc/sysctl.conf. Add following lines to /etc/sysctl.conf (RH based distros)

net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 8192
net.ipv4.neigh.default.gc_thresh3 = 8192
net.ipv4.neigh.default.base_reachable_time = 86400
net.ipv4.neigh.default.gc_stale_time = 86400

Save sysctl.conf and exec sysctl -p. You can also reboot but it isn’t necessary.

The default sysctl.conf file

net.ipv4.ip_forward=0
kernel.shmmax=68719476736
kernel.msgmax=65536
kernel.msgmnb=65536
net.ipv4.conf.default.rp_filter=1
kernel.sysrq=0
net.ipv4.conf.default.accept_source_route=0
kernel.shmall=4294967296
kernel.core_uses_pid=1
net.ipv4.tcp_syncookies=1

“Tuned” systctl.conf

net.ipv4.ip_forward=0
kernel.shmmax=4294967295
kernel.msgmax=65536
kernel.msgmnb=65536
net.ipv4.conf.default.rp_filter=1
kernel.sysrq=0
net.ipv4.conf.default.accept_source_route=0
kernel.shmall=268435456
kernel.core_uses_pid=1
net.ipv4.tcp_syncookies=1
net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 8192
net.ipv4.neigh.default.gc_thresh3 = 8192
net.ipv4.neigh.default.base_reachable_time = 86400
net.ipv4.neigh.default.gc_stale_time = 86400

Explanation…

The neighbour table is generally known as ARP table and the default value for gc_thresh1 is 128 (Adjust where the gc will leave arp table alone)

[root@myServer ~]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh1
128

which is not enough for large networks (more than 128 hosts). Thats why we need to tune this value. The gc_thresh2 is a soft limit (Tell the gc when to become aggressive with arp table cleaning.) and the gc_thresh3 is a hard limit (Don’t allow the arp table to become bigger than this).

To enlarge the ARP cache table on the live system run:

# sysctl -w net.ipv4.neigh.default.gc_thresh3=8192
# sysctl -w net.ipv4.neigh.default.gc_thresh2=8192
# sysctl -w net.ipv4.neigh.default.gc_thresh1=4096

It is possible that after distro update your systctl.conf will be replaced with the default values. Check this file periodically..

This morning I had a problem with apache. The httpd was stopped and the #service httpd restart didn’t work.

Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs

The port 80 was already in use.

# fuser -k -n tcp 80

was the solution…

FreeRADIUS is the most widely deployed RADIUS server in the world. It is the basis for multiple commercial offerings. It supplies the AAA needs of many Fortune-500 companies and Tier 1 ISPs. In this post I will try to describe basic installation and config options (at least some of them). The biggest problem for me was the lack of documentation and it was very hard to learn something about it when the latest book about Radius was published 8 years ago.

Where possible, I recommend using the packaging system that is used for your distro. The version that is supplied might be out of date, but it is likely to work “out of the box”.

RPM packages

FreeRADIUS is distributed on Fedora/RHEL/CentOS systems as a set of RPM packages. There is a main package called “freeradius” and several subpackages whose name is “freeradius-XXX” where XXX is optional functionality. For example the support needed for MySQL database backend will be found in the package “freeradius-mysql”.

On CentOS and Red Hat, “yum install freeradius” will install FreeRadius 1.1.3 which is a several years old version. Better option is to install FreeRadius 2.x with yum install freeradius2. Please see the notes above about optional packages. Also, keep in mind that all config files will be installed in /etc/raddb. More info can be found HERE.

More info about RPM versions can be found Here (Thanks J. Dennis).

[root@ms ~]# yum search freeradius
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * addons: mirror.centos.com.ba
 * base: mirror.centos.com.ba
 * extras: mirror.centos.com.ba
 * rpmforge: ftp-stud.fht-esslingen.de
 * updates: mirror.centos.com.ba
Excluding Packages in global exclude list
Finished
====================== Matched: freeradius ===================================
freeradius.x86_64 : High-performance and highly configurable free RADIUS server.
freeradius-mysql.x86_64 : MySQL bindings for freeradius
freeradius-postgresql.x86_64 : postgresql bindings for freeradius
freeradius-unixODBC.x86_64 : unixODBC bindings for freeradius
freeradius2.x86_64 : High-performance and highly configurable free RADIUS server
freeradius2-krb5.x86_64 : Kerberos 5 support for freeradius
freeradius2-ldap.x86_64 : LDAP support for freeradius
freeradius2-mysql.x86_64 : MySQL support for freeradius
freeradius2-perl.x86_64 : Perl support for freeradius
freeradius2-postgresql.x86_64 : Postgresql support for freeradius
freeradius2-python.x86_64 : Python support for freeradius
freeradius2-unixODBC.x86_64 : Unix ODBC support for freeradius
freeradius2-utils.x86_64 : FreeRADIUS utilities

More info about basic settings will be shown later.

Install from source

Download the latest FreeRadius from this link. (Current version is 2.1.10)

# wget ftp://ftp.freeradius.org/pub/freeradius/freeradius-server-2.1.10.tar.gz
# tar xvzf freeradius-server-2.1.10.tar.gz
# cd freeradius-server-2.1.10
# ./configure

It is very likely that config process will fail for some reasons. To fix this, search WARNINGS lines and install the missing rpms (yum install libtood-ltdl libtool-ltdl-devel is required).

# make
# make install

This is a default installation and all config files will be located in /usr/local/etc/raddb and you should find the next files inside

[root@ms raddb]# ls -la
total 220
drwxr-xr-x 7 root root  4096 Jan 27 15:54 .
drwxr-xr-x 4 root root  4096 Jan 27 15:53 ..
-rw-r----- 1 root root   671 Jan 27 15:54 acct_users
-rw-r----- 1 root root  4174 Jan 27 15:54 attrs
-rw-r----- 1 root root   513 Jan 27 15:54 attrs.access_challenge
-rw-r----- 1 root root   458 Jan 27 15:54 attrs.access_reject
-rw-r----- 1 root root   437 Jan 27 15:54 attrs.accounting_response
-rw-r----- 1 root root  2022 Jan 27 15:54 attrs.pre-proxy
drwxr-x--- 2 root root  4096 Jan 27 15:54 certs
-rw-r----- 1 root root  6703 Jan 27 15:54 clients.conf
-rw-r----- 1 root root   883 Jan 27 15:54 dictionary
-rw-r----- 1 root root 18063 Jan 27 15:54 eap.conf
-rwxr-xr-x 1 root root  4744 Jan 27 15:54 example.pl
-rw-r----- 1 root root 12722 Jan 27 15:54 experimental.conf
-rw-r----- 1 root root  2352 Jan 27 15:54 hints
-rw-r----- 1 root root  1604 Jan 27 15:54 huntgroups
-rw-r----- 1 root root  3218 Jan 27 15:54 ldap.attrmap
drwxr-x--- 2 root root  4096 Jan 27 15:54 modules
-rw-r----- 1 root root  2840 Jan 27 15:54 policy.conf
-rw-r----- 1 root root  4873 Jan 27 15:54 policy.txt
-rw-r----- 1 root root   984 Jan 27 15:54 preproxy_users
-rw-r----- 1 root root 26529 Jan 27 15:54 proxy.conf
-rw-r----- 1 root root 27238 Jan 27 15:54 radiusd.conf
drwxr-x--- 2 root root  4096 Jan 27 15:54 sites-available
drwxr-x--- 2 root root  4096 Jan 27 15:54 sites-enabled
drwxr-x--- 7 root root  4096 Jan 27 15:54 sql
-rw-r----- 1 root root  3042 Jan 27 15:54 sql.conf
-rw-r----- 1 root root  2475 Jan 27 15:54 sqlippool.conf
-rw-r----- 1 root root  3597 Jan 27 15:54 templates.conf
-rw-r----- 1 root root  6524 Jan 27 15:54 users

The default configuration is designed to work everywhere, and to provide nearly every authentication method. Do not edit the default configuration files until you understand what they do. This means reading the documentation contained in the comments of the configuration files.

When the server has been installed on a new machine, the first step is to start it in debugging mode, as user root:

# radiusd -X

This step demonstrates that the server is installed and configured properly. If you have installed Version 2 from source, this step will also create the default certificates used for EAP authentication. If everything went OK, you should see the lines

......
Listening on authentication address * port 1812
Listening on accounting address * port 1813
Listening on command file /usr/local/var/run/radiusd/radiusd.sock
Listening on authentication address 127.0.0.1 port 18120 as server inner-tunnel
Listening on proxy address * port 1814
Ready to process requests.

To stop freeradius press Ctrl+C.

Thats all for now… In next few days I will add more articles about FreeRadius.

Few servers I maintain totally confused me. The loadavg is steadily increasing every round hour. With top command I can’t see any relevant process which can produce high load.

top - 15:07:17 up 41 days,  3:52,  1 user,  load average: 4.22, 1.61, 0.76
Tasks: 147 total,   1 running, 146 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.7%sy,  0.0%ni, 85.5%id, 13.5%wa,  0.1%hi,  0.2%si,  0.0%st
Mem:   1025084k total,  1016732k used,     8352k free,    24472k buffers
Swap:  2064376k total,      116k used,  2064260k free,   133380k cached
 
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6082 root      15   0  126m 9632 5008 S  0.3  0.9   0:00.50 php
 7363 root      15   0 12736 1112  808 R  0.3  0.1   0:00.03 top
27418 root      15   0  347m 3860 1096 S  0.3  0.4   0:22.80 radiusd
    1 root      15   0 10344  680  568 S  0.0  0.1   0:01.88 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.54 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:15.33 ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:01.81 migration/1
    6 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/1
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/1
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:39.01 migration/2
...

The server is CentOS 5.5 64b, quad core Intel processor. After some digging I found out that 4 servers are affected and they are CentOS 5.x 64b. 32bit systems are not affected…

So, the first step is to check cron settings because it is obvious that something is triggered by cron (hourly). Here it is: mcelog.cron. After Googleing about this problem I found this LINK. Or here LINK.

The bug is “closed” but I wouldn’t say so… I had latest mcelog installed and it causes the same problem which is described above.

Few days ago I had a strange problem with yum and instead of clean install and update process I got Python errors and rpm message “rpmdb: Lock table is out of available locker entries”. After few minutes I found out that during installation or update process, rpm accesses the Berkeley database files and it makes temporary locker entries within the tables while it searches for data. Sometimes the locks are never cleared and we have a problem… Don’t worry… It can be fixed…

First here is the complete error:

[root@myserver ~]# yum install firefox
Loading "fastestmirror" plugin
rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: 3929
error: db4 error(22) from db->close: Invalid argument
error: cannot open Packages index using db3 - Cannot allocate memory (12)
error: cannot open Packages database in /var/lib/rpm
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum-cli/yummain.py", line 85, in main
    base.getOptionsConfig(args)
  File "/usr/share/yum-cli/cli.py", line 163, in getOptionsConfig
    disabled_plugins=self.optparser._splitArg(opts.disableplugins))
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 164, in _getConf
ig
    self._conf = config.readMainConfig(startupconf)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 685, in readMainCo
nfig
    yumvars['releasever'] = _getsysver(startupconf.installroot, startupconf.dist
roverpkg)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 755, in _getsysver
    idx = ts.dbMatch('provides', distroverpkg)
TypeError: rpmdb open failed

It looks scary

The first step is to backup /var/lib/rpm (in case that something goes wrong) with

# mkdir /backup
# tar cvzf /backup/rpm-backup.tar.gz /var/lib/rpm

You don’t need to backup but it is highly recommended.

Remove the Berkeley databases that rpm uses with

# rm /var/lib/rpm/__db.00*

Note: Probably there will be several files. Confirm all removal with “y”

Make rpm rebuild the databases from scratch (this may take a minute or two)

# rpm --rebuilddb

And that should be enough to fix this problem. Additionally you can list rpms to be sure that everything is OK.

# rpm -qa | sort

If you’re using RH based distros, you’ll probably notice that their habbit is to keep the same software versions in one release.  For example, if you need PHP on CentOS 5.x, # yum install php will install PHP 5.1.6  (Latest PHP version available on http://php.net is 5.3.x).  If you need MySQL, you can count on MySQL 5.0.xx branch and any other wishes will force you to use independent repos (like http://www.jasonlitka.com/yum-repository/). Another option is to use  “do-it-yourself” method.

In this post I’ll write about manual upgrade steps from rpm archives. Please keep in mind that this procedure works for me and please do not send me the private messages. I can’t help you on that way. Only options is to post your comments here and I will try to solve your problem(s).

First thing you need to do is to see what mysql packages do you have installed

# rpm -qa | grep -i ^mysql

You will get something like

mysql-connector-odbc-3.51.12-2.2
mysql-5.0.77-4.el5_5.3
MySQL-python-1.2.1-1
mysql-server-5.0.77-4.el5_5.3
mysql-devel-5.0.77-4.el5_5.3
mysql-bench-5.0.77-4.el5_5.3

Then backup all your databases, save them on the safe location, protect with alarms, guards, poison dogs, cobras, ninjas, etc…

Download rpms from http://www.mysql.com/downloads/mysql/ (MySQL-client-community-5.1.50-1.rhel5.i386.rpm, MySQL-devel-community-5.1.50-1.rhel5.i386.rpm, MySQL-server-community-5.1.50-1.rhel5.i386.rpm, MySQL-shared-community-5.1.50-1.rhel5.i386.rpm)

Note: if you have 32-bit OS, download i386 rpms, if you have 64bit download x86_64 rpms)

Stop mysql server with

# service mysqld stop

Then remove mysql rpms but with –nodeps option (in case you use yum remove mysql, you will need to reinstall a lot of apps because they will be deleted too)

# rpm -e --nodeps mysql

Repeat the same with other mysql packages (devel, bench, client,…). Then you need to install downloaded rpms with rpm -i mysql… and do it.

It is possible to receive the errors like

ls: /var/lib/mysql/*.err: No such file or directory
ls: /var/lib/mysql/*.err: No such file or directory
ERROR: 1136  Column count doesn't match value count at row 1
100910 10:24:00 [ERROR] Aborting
 
100910 10:24:00 [Note] /usr/sbin/mysqld: Shutdown complete
 
Installation of system tables failed!  Examine the logs in /var/lib/mysql for more information.

This will be fixed later…

Now you need to add .my.cnf file to your root dir so you can execute mysql_upgrade command (note that . before m means that file is hidden). Add next lines inside this file (and don’t forget to replace rootpass with your real mysql root pass)

[client]
user=root
password=rootpass

(chmod .my.cnf to 600 for security reasons)

Then exec next command

# service mysql start

# mysql_upgrade

This will produce the similar output

Looking for 'mysql' as: mysql
Looking for 'mysqlcheck' as: mysqlcheck
Running 'mysqlcheck with default connection arguments
Running 'mysqlcheck with default connection arguments
blabla.table1                            OK
blabla.table2                             OK
blabla.table3                                OK
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.event                                        OK
mysql.func                                         OK
mysql.general_log
Error    : You can't use locks with log tables.
status   : OK
mysql.help_category
error    : Table upgrade required. Please do "REPAIR TABLE `help_category`" or dump/reload to fix it!
mysql.help_keyword
error    : Table upgrade required. Please do "REPAIR TABLE `help_keyword`" or dump/reload to fix it!
mysql.help_relation                                OK
database2.cache
error    : Table upgrade required. Please do "REPAIR TABLE `cache`" or dump/reload to fix it!
database2.contacts                             OK
database2.identities                           OK
database2.messages
error    : Table upgrade required. Please do "REPAIR TABLE `messages`" or dump/reload to fix it!
database2.session
error    : Table upgrade required. Please do "REPAIR TABLE `session`" or dump/reload to fix it!
database2.users
error    : Table upgrade required. Please do "REPAIR TABLE `users`" or dump/reload to fix it!
 
Repairing tables
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_topic                                   OK
mysql.proc                                         OK
mysql.time_zone_name                               OK
database2.cache
note     : The storage engine for the table doesn't support repair
database2.messages
note     : The storage engine for the table doesn't support repair
database2.session
note     : The storage engine for the table doesn't support repair
database2.users
note     : The storage engine for the table doesn't support repair
Running 'mysql_fix_privilege_tables'...
OK

As you can see, the database database2 can’t be repaired and you should drop all tables inside this database and import your backup. After this, you can check is everything ok with mysql_upgrade –force

Once again, restart mysql with service mysql restart and check logs. Test is everything ok, try ti create a new database, optimize your installation, eat something…

Note:
I had a lot of problems with this upgrade. I had to upgrade PHP to 5.3.x, I had to recompile postfix with MySQL support, I had to download and recompile Dovecot because Dovecot from CentOS repos is compiled with mysql 5.0 branch. Some versions of RoundCube doesn’t work with php 5.3.x so you should download latest, etc etc… It can be done but please be careful with this. Who knows which nuclear reactor will explode after this

I already wrote about upgrading via Jason Litka repo on this PAGE but in case you have problems with this repo, you can add Remi Collet repo.

First, import Remi GPG key with

#rpm --import http://rpms.famillecollet.com/RPM-GPG-KEY-remi

# cd /etc/yum.repos.d
# wget http://rpms.famillecollet.com/enterprise/remi.repo

This file provides configuration for remi and remi-test repositories. Keep in mind that Remi repo is disabled by default so you can add –enablerepo=remi to yum command or you can edit line enabled=0 to enabled=1 inside remo.repo. For production servers I don’t recommend enabled=1 to remi-test repo.

Current PHP is 5.3.3 and MySQL 5.1.50