I already wrote about Sysbench and MySQL tunning scripts. Those articles can be found HERE and HERE.

Now lets do some tests and lets see the dependance from write cache enabled/disabled option on your disks. Keep in mind that RH based distros will probably disable write cache (I tried Ubuntu on this server and the same thing happen – write cache was disabled after installation).

The first step is to check the disk with

[root@s1 sysbench-0.4.12]# hdparm -i /dev/sda
 
/dev/sda:
 
 Model=GB0250EAFYK                             , FwRev=HPG1    , SerialNo=WCAT1E535427
 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq }
 RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50
 BuffType=unknown, BuffSize=16384kB, MaxMultSect=16, MultSect=?16?
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=268435455
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio3 pio4
 DMA modes:  mdma0 mdma1 mdma2
 UDMA modes: udma0 udma1 udma2
 AdvancedPM=no WriteCache=disabled
 Drive conforms to: unknown:  ATA/ATAPI-1 ATA/ATAPI-2 ATA/ATAPI-3 ATA/ATAPI-4 ATA/ATAPI-5 ATA/ATAPI-6 ATA/ATAPI-7

Note WriteCache=disabled line

I suppose you know how to create a database, assign a user and set password

Read-write test for InnoDB

# sysbench --num-threads=16 --max-requests=10000 --test=oltp --oltp-table-size=500000 --mysql-socket=/var/lib/mysql/mysql.sock --oltp-test-mode=complex --mysql-user=test_database --mysql-password=test_database_password run

gave me the next results

 
No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 16
 
Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 10000
Threads started!
Done.
 
OLTP test statistics:
    queries performed:
        read:                            140000
        write:                           50000
        other:                           20000
        total:                           210000
    transactions:                        10000  (42.63 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 190000 (809.95 per sec.)
    other operations:                    20000  (85.26 per sec.)
 
Test execution summary:
    total time:                          234.5821s
    total number of events:              10000
    total time taken by event execution: 3751.1329
    per-request statistics:
         min:                                 17.33ms
         avg:                                375.11ms
         max:                               3850.27ms
         approx.  95 percentile:             766.79ms
 
Threads fairness:
    events (avg/stddev):           625.0000/1.73
    execution time (avg/stddev):   234.4458/0.08

The total time is more than 234 seconds which is terrible result.

Now, delete sbtest table and recreate it again (please see the sysbench howto).

Enable write cache with

# hdparm -W1 /dev/sda

and start test again

The result is

No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 16
 
Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 10000
Threads started!
Done.
 
OLTP test statistics:
    queries performed:
        read:                            140000
        write:                           50000
        other:                           20000
        total:                           210000
    transactions:                        10000  (621.08 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 190000 (11800.43 per sec.)
    other operations:                    20000  (1242.15 per sec.)
 
Test execution summary:
    total time:                          16.1011s
    total number of events:              10000
    total time taken by event execution: 255.8624
    per-request statistics:
         min:                                  2.49ms
         avg:                                 25.59ms
         max:                                689.84ms
         approx.  95 percentile:              41.84ms
 
Threads fairness:
    events (avg/stddev):           625.0000/4.68
    execution time (avg/stddev):   15.9914/0.03

Now we have a 16.1011 seconds for a complete test. I repeated the test several time with the same results.

We have a 14.62 times better result.

It is obvious that write cache is very important for MySQL but keep in mind that something can be lost in case of power failure and you should think about battery backups.

Tuning Kernel parameters

It is good idea to tune a few kernel parameters too. For this purpose you can install ktune (yum install ktune). (more info https://fedorahosted.org/ktune/)

With “service ktune start” you will set up scheduler on deadline instead of cfq (which can be up to 20% slower)

[root@s1 bekap]# service ktune start
Applying ktune sysctl settings:
/etc/sysctl.ktune:                                         [  OK  ]
Applying sysctl settings from /etc/sysctl.conf:            [  OK  ]
Applying deadline elevator: sda                            [  OK  ]

After Ktune I have a little faster time (around 15 seconds)

More info about disk elevators can be found here http://www.redhat.com/magazine/008jun05/features/schedulers/

Optimizing the EXT3 file system on CentOS

noatime
This mount option tells the system not to update inode access times. This is a good option for web servers, news servers or other uses with high access file systems.

Open /etc/fstab and add noatime like shown below

/dev/VolGroup00/LogVol00 /                       ext3    defaults,noatime        1 1

If you want to find the newest file inside directory, you can use the ls & tail combination.

ls -t | head -1

Output

bash$ ls -t | head -1
id695.txt

Another option is with the next command

ls -lrth | tail -1

Output

bash$ ls -lrth | tail -1
-rw-r-----   1 user      ugroup         85 Nov 11 15:38 id695.txt

Note:
| is used to send the output of the first command as input to the second one.
tail — outputs the last files
-1 — denotes the number of lines u want to display (in case you don’t set -1, by default you will get the last 10 lines)

If you need to copy HTML tables into Excel, Libreoffice Calc or other datasheet applications, you can think about Firefox and Dafizilla Table2Clipboard firefox addon.

This addon will allow you to select rows and columns from a table simply pressing Control key and picking rows/columns with left mouse button. The selection can be copied to clipboard and pasted into datasheet applications but without ugly results.

Pasting in plain text editors is also supported as CSV file.

Here is the LINK for this great tool.

Do not forget to donate

Using scp, rsync and ssh requires the password unless you add the public key from src host to authorised_keys on destination host.

scp and rsync commands are used to transfer or backup files between known hosts or by the same user on both the hosts.

Lets say you want to copy between two hosts SOURCE and DESTINATION. SOURCE is the host where you would run the scp, ssh or rsync command.

On the SOURCE host, run

# ssh-keygen -t rsa

It will prompt for a passphrase but do not enter anything. Instead, just press the enter key. It’ll generate an identification (private key) and a public key. Do not ever share the private key with anyone!
The public key will be generated in ~/.ssh/id_rsa.pub.

For example in root/.ssh/id_rsa.pub

Copy id_rsa.pub file to DESTINATION inside the /root/.ssh/
On DESTINATION host, login as the remote user which you plan to use (in this case root) and copy the contents of id_rsa.pub to /root/.ssh/authorized_keys

# cat id_rsa.pub >> /root/.ssh/authorized_keys
# chmod 700 /root/.ssh/authorized_keys

If this file authorized_keys does not exists, the command above will create it. Make sure you remove permission for others to read this file.

On some distros, ssh by default does not allow root to log in. To enable root login, edit /etc/ssh/sshd_config and changing the option of PermitRootLogin from no to yes (on DESTINATION host). Restart sshd to apply changes and that is it.

In case you want to ssh, scp or rsync from DESTINATION to SOURCE host you will be asked for password. You can reverse the steps above (generate the public key on DESTINATION and copy it to the SOURCE host) and it will work in both directions.

In case that one server gets hacked, the other one will be too

Taking care about disk usage is a very good habit. There are several options for this and now I will only describe df and du commands.

The df utility displays the disk space usage on all mounted filesystems.

[root@abc ~]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      35772016  20297132  13628408  60% /
/dev/sda1               101086     17692     78175  19% /boot
tmpfs                   513044         0    513044   0% /dev/shm

Adding the -T option (type) will print the filesystem type. Also, the good idea is to add -h option because df measures the size in 1K blocks, which is difficult for a desktop user to recalculate.

Now we have

[root@abc ~]# df -h -T
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
              ext3     35G   20G   13G  60% /
/dev/sda1     ext3     99M   18M   77M  19% /boot
tmpfs        tmpfs    502M     0  502M   0% /dev/shm

As you can see, it is much easier to understand.

You can use the du command to determine which files or directories need to be deleted or reduced. A simple du will print usage for the present working directory and its subdirectories, along with the size of each directory.

du directory will print the size of an particular dir.

Avilable options:
-h – print the size of the podcasts directory in a more readable format
-c – prints the grand total size of the directory at the end.
-a – displays the file names along with directories
-s – display a summary, without showing all of the subdirectories.

For example, du -ch | grep total prints just one line with the total size of the directory.

Often the server admin has little control over the applications which uses MySQL and it is hard to find the bottlenecks. This blog post can’t bring the peace in the world, or help NASA to finally land on the Mars. Instead those tasks, I’ll try to solve something else and present my own experiences with MySQL storage engines (at least for MyISAM and InnoDB as the most popular).

Keep in mind that I’m not an expert in this field. When I have problems with MySQL, I dig with my both hands (and head) to solve them without casualties. Every app has its own queries and you’ll need to track them down via slow query log.

Before you do anything, keep in mind that default MySQL installation usually works fine. Unfortunately maximum performances can’t be achieved without adjustments to increase performance and stability.

Before you can do anything, you need to enable slow query log.

Exec

# touch /var/log/slow-query.log

to create slow-query.log file. Then add

log-slow-queries = /var/log/slow-query.log
long_query_time = 4
log-queries-not-using-indexes

(for MySQL 5.0.x)

to /etc/my.cnf (inside [mysqld] section) and restart mysql. Wait at least 24-48 hours before you proceed to the next step.

(For MySQL 5.1.x check here: LINK)

High performance tuning scripts

Mysqltuner

MySQLTuner is a script written in Perl that will assist you with your MySQL configuration and make recommendations for increased performance and stability.

# wget mysqltuner.pl
# chmod +x mysqltuner.pl
# ./mysqltuner.pl

Read output and try to follow the recommendations.

Tuning-primer

Tuning-primer is another script who takes information from “SHOW STATUS LIKE…” and “SHOW VARIABLES LIKE…” to produce readable recommendations for tuning server variables.

# wget http://www.day32.com/MySQL/tuning-primer.sh
# ./tuning-primer.sh

Also, read output and try to see what you can do to fix problems. Keep in mind that you need to restart mysql after you add something to /etc/my.cnf. In case something goes wrong, check mysql log (usually /var/log/mysqld.log

MyTOP

Mytop is a console-based (non-gui) tool for monitoring the threads and overall performance of a MySQL

To install mytop,

# wget http://jeremy.zawodny.com/mysql/mytop/mytop-1.6.tar.gz
# tar -xvzf mytop-1.6.tar.gz
# cd mytop-1.6
# perl Makefile.PL
# make
# make install

In case some Perl libraries are missing, you should install them via cpan.

For example, Term::ReadKey is required and you can get it via cpan.

# cpan
(pres Enter several times until you get cpan shell)
cpan> install Term::ReadKey
cpan> quit

Usage: # mytop -d DATABASE -u USERNAME -p PASSWORD

(replace the DATABASE, USERNAME and PASSWORD with your parameters.

MySQLReport

mysqlreport makes a friendly report of important MySQL status values. mysqlreport transforms the values from SHOW STATUS into an easy-to-read report that provides an in-depth understanding of how well MySQL is running. mysqlreport is a better alternative (and practically the only alternative) to manually interpreting SHOW STATUS.

# wget http://hackmysql.com/mysqlreport
# chmod +x mysqlreport
# ./mysqlreport --user root --password

Enter password for root and check output. More info can be found here. LINK

The first post for this year…

Simple FTP backup script

# nano backup_script

Add next lines and save.

#!/bin/sh
DM=$(date +"%b-%d-%Y")
tar -Pzcf /backup/backup_$DM.tar.gz /backup/some_dir/
cd /backup
ftp -nv <<EOF
open 192.168.1.2
user ftp_username ftp_password
bin
put backup_$DM.tar.gz
quit
EOF
echo -e  "FTP backup done!"

Make it executable with

# chmod +x backup_script

Of course, you need to replate ftp_username and ftp_password with valid username and password. Also, IP address should be replaced with your ftp server IP address.

Add this script to cron (for example once per day at 4:00AM)

One of the biggest mistakes you can make with MySQL is to leave its default configuration. It will work, but not as it should.

For example, default MySQL installation (5.0x) will keep all InnoDB data in one file – ibdata1. This file is usually located in /var/lib/mysql (at least on RH and Debian based distros) and after few months this file can became very big (in my case it was 20GB). This file has a initial size of 10Mb and it automatically extends but it can’t be reduced with DELETE, TRUNCATE or DROP. The file could reach the maximum size allowed by the filesystem if no limit is set in the my.cnf file (Debian/Ubuntu -/etc/mysql/my.cnf or RH based distros /etc/my.cnf). The best idea is to force MySQL to create an ibd file for each InnoDB table (add innodb_file_per_table in my.cnf and restart mysql). Unf. this won’t affect old tables which are already created and which are in use.

You can “FIX” this on several ways but no matter what option do you choose, BACKUP YOUR WHOLE MYSQL DATA DIRECTORY and stop services connected to your MySQL server (httpd, radius, postfix, dovecot, etc).

Keep in mind that converting InnoDB table to MyISAM will kill foreign keys so do not do it unless you know how to recreate your foreign keys.

The best option is to dump all your databases in one sql file.

Step 1

# /usr/bin/mysqldump ––extended-INSERT ––all-DATABASES ––add-drop-DATABASE ––disable-KEYS ––flush-privileges ––quick ––routines ––triggers > backup.SQL

Step 2

Stop mysql server with

# service mysqld stop

or

# /etc/init.d/mysqld stop

Step 3

Backup complete mysql data dir (/var/lib/mysql).

# cd /var/lib/
# mv mysql mysql_backup
# mkdir mysql
# chown mysql:mysql mysql

Then you should have something like

drwxr-xr-x  5 mysql     mysql     4096 2010-12-30 13:38 mysql

Step 4

Add innodb_file_per_table option in /etc/my.cnf file and save file

Step 5

Re-initialize the database with the following commands

# su mysql
$ mysql_install_db
$ exit

Step 6

Start mysql server with service mysqld start and get into mysql console with

# mysql -u root

Then exec next commands

SET FOREIGN_KEY_CHECKS=0;
SOURCE backup.SQL;
SET FOREIGN_KEY_CHECKS=1;

Step 7

Restart mysql server with service mysqld restart

That should be all.

In case that something goes wrong you still have mysql_backup dir which contains all you databases and files. Simple rename the new mysql dir to mysql_new and mysql_backup to mysql. Then restart mysql.

Keep in mind that this operation will kill all services who depends on mysql. So, be quick

Also, good idea is to execute command mysql_secure_installation which will “tight” your MySQL server.

Round robin works with a fixed amount of data, and a pointer to the current element. This can be described like a circle with N dots plotted on the edge of the circle. Those dots are the places where data can be stored. The pointer can be described like an arrow from the center of the circle to one of the dots. It shows to the current dot. When the current data is read or written, the pointer moves to the next element. As we are on a circle there is no beginning nor an end, you can go on and on. After a while, all the available places will be used and the process automatically reuses old locations. It is easy to understand that this database will not grow and the file size is always the same.

RRDtool is the OpenSource industry standard, high performance data logging and graphing system for time series data (it uses Round Robin Databases (RRDs)). RRDTool is created by Tobias Oetiker.

For traffic stats, RRD stores the difference in values (between the last value and current) in the database, rather than the value itself. This type of data source is called COUNTER. This type is for continuous incrementing counters like the ifInOctets counter in a router. (assumes that the counter never decreases, except when a counter overflows). It is easy to work with but you will have a problem when routers or modems are rebooted. The counters on the interfaces will be cleared and you will have spikes on your graphs (because the difference between the last value and current is abnormal – for example you will see 100Mbps on 1Mbps link).

To see what I’m walking about, check the next image

It is noticeable that the only thing we can see on this graph is the spike. Other details are not visible unless we remove this/those spike(s).

Here is the one solution.

The first thing you should do is to backup your rrd. The second step is to identify data sources used inside the router.rrd file. You can achieve this with next command:

# rrdtool info router.rrd | more

When you find your data sources, you will need to tune them. (I suppose those DSes are In and Out and we will limit them on 5000000 (5Mbps)

# rrdtool tune router.rrd -a Out:5000000
# rrdtool tune router.rrd -a In:5000000

Dump rrd data inside XML file with

# rrdtool dump router.rrd > fix.xml

And delete router.rrd file

# rm -fr router.rrd

Then import data from XML file in new router.rrd

# rrdtool restore fix.xml router.rrd -r

That’s it.

The graph is OK now.

As you can see, the spike is limited to 5Mbps. You can decrease this limit to less then 5000000 if you don’t expect values above 5Mbps.

I suppose you already know how to generate graphs. If you don’t know how check rrdtool tutorial or leave a comment and I’ll write a small article about it.

Few days ago I had a strange problem with yum and instead of clean install and update process I got Python errors and rpm message “rpmdb: Lock table is out of available locker entries”. After few minutes I found out that during installation or update process, rpm accesses the Berkeley database files and it makes temporary locker entries within the tables while it searches for data. Sometimes the locks are never cleared and we have a problem… Don’t worry… It can be fixed…

First here is the complete error:

[root@myserver ~]# yum install firefox
Loading "fastestmirror" plugin
rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: 3929
error: db4 error(22) from db->close: Invalid argument
error: cannot open Packages index using db3 - Cannot allocate memory (12)
error: cannot open Packages database in /var/lib/rpm
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum-cli/yummain.py", line 85, in main
    base.getOptionsConfig(args)
  File "/usr/share/yum-cli/cli.py", line 163, in getOptionsConfig
    disabled_plugins=self.optparser._splitArg(opts.disableplugins))
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 164, in _getConf
ig
    self._conf = config.readMainConfig(startupconf)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 685, in readMainCo
nfig
    yumvars['releasever'] = _getsysver(startupconf.installroot, startupconf.dist
roverpkg)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 755, in _getsysver
    idx = ts.dbMatch('provides', distroverpkg)
TypeError: rpmdb open failed

It looks scary

The first step is to backup /var/lib/rpm (in case that something goes wrong) with

# mkdir /backup
# tar cvzf /backup/rpm-backup.tar.gz /var/lib/rpm

You don’t need to backup but it is highly recommended.

Remove the Berkeley databases that rpm uses with

# rm /var/lib/rpm/__db.00*

Note: Probably there will be several files. Confirm all removal with “y”

Make rpm rebuild the databases from scratch (this may take a minute or two)

# rpm --rebuilddb

And that should be enough to fix this problem. Additionally you can list rpms to be sure that everything is OK.

# rpm -qa | sort