Category Archives: Red Hat/CentOS

Neighbour table overflow – sysctl.conf tunning

If you have a big network with the hundreds of hosts you can expect “Neighbour table overflow” error which occurs in large networks when there are two many ARP requests which the server is not able to reply. For example you’re using server as a DHCP server, cable modems provisioning, etc.

Nov 10 03:18:17 myhost Neighbour table overflow.
Nov 10 03:18:23 myhost printk: 12 messages suppressed.

Continue reading Neighbour table overflow – sysctl.conf tunning

Problem with apache – Address already in use… Unable to open logs

This morning I had a problem with apache. The httpd was stopped and the #service httpd restart didn’t work.

Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs

The port 80 was already in use.

# fuser -k -n tcp 80

was the solution…

FreeRadius install howto (1)

FreeRADIUS is the most widely deployed RADIUS server in the world. It is the basis for multiple commercial offerings. It supplies the AAA needs of many Fortune-500 companies and Tier 1 ISPs. In this post I will try to describe basic installation and config options (at least some of them). The biggest problem for me was the lack of documentation and it was very hard to learn something about it when the latest book about Radius was published 8 years ago.

Where possible, I recommend using the packaging system that is used for your distro. The version that is supplied might be out of date, but it is likely to work “out of the box”.

RPM packages

FreeRADIUS is distributed on Fedora/RHEL/CentOS systems as a set of RPM packages. There is a main package called “freeradius” and several subpackages whose name is “freeradius-XXX” where XXX is optional functionality. For example the support needed for MySQL database backend will be found in the package “freeradius-mysql”.

On CentOS and Red Hat, “yum install freeradius” will install FreeRadius 1.1.3 which is a several years old version. Better option is to install FreeRadius 2.x with yum install freeradius2. Please see the notes above about optional packages. Also, keep in mind that all config files will be installed in /etc/raddb. More info can be found HERE.

More info about RPM versions can be found Here (Thanks J. Dennis).

[root@ms ~]# yum search freeradius
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * addons: mirror.centos.com.ba
 * base: mirror.centos.com.ba
 * extras: mirror.centos.com.ba
 * rpmforge: ftp-stud.fht-esslingen.de
 * updates: mirror.centos.com.ba
Excluding Packages in global exclude list
Finished
====================== Matched: freeradius ===================================
freeradius.x86_64 : High-performance and highly configurable free RADIUS server.
freeradius-mysql.x86_64 : MySQL bindings for freeradius
freeradius-postgresql.x86_64 : postgresql bindings for freeradius
freeradius-unixODBC.x86_64 : unixODBC bindings for freeradius
freeradius2.x86_64 : High-performance and highly configurable free RADIUS server
freeradius2-krb5.x86_64 : Kerberos 5 support for freeradius
freeradius2-ldap.x86_64 : LDAP support for freeradius
freeradius2-mysql.x86_64 : MySQL support for freeradius
freeradius2-perl.x86_64 : Perl support for freeradius
freeradius2-postgresql.x86_64 : Postgresql support for freeradius
freeradius2-python.x86_64 : Python support for freeradius
freeradius2-unixODBC.x86_64 : Unix ODBC support for freeradius
freeradius2-utils.x86_64 : FreeRADIUS utilities

More info about basic settings will be shown later.

Install from source

Download the latest FreeRadius from this link. (Current version is 2.1.10)

# wget ftp://ftp.freeradius.org/pub/freeradius/freeradius-server-2.1.10.tar.gz
# tar xvzf freeradius-server-2.1.10.tar.gz
# cd freeradius-server-2.1.10
# ./configure

It is very likely that config process will fail for some reasons. To fix this, search WARNINGS lines and install the missing rpms (yum install libtood-ltdl libtool-ltdl-devel is required).

# make
# make install

This is a default installation and all config files will be located in /usr/local/etc/raddb and you should find the next files inside

[root@ms raddb]# ls -la
total 220
drwxr-xr-x 7 root root  4096 Jan 27 15:54 .
drwxr-xr-x 4 root root  4096 Jan 27 15:53 ..
-rw-r----- 1 root root   671 Jan 27 15:54 acct_users
-rw-r----- 1 root root  4174 Jan 27 15:54 attrs
-rw-r----- 1 root root   513 Jan 27 15:54 attrs.access_challenge
-rw-r----- 1 root root   458 Jan 27 15:54 attrs.access_reject
-rw-r----- 1 root root   437 Jan 27 15:54 attrs.accounting_response
-rw-r----- 1 root root  2022 Jan 27 15:54 attrs.pre-proxy
drwxr-x--- 2 root root  4096 Jan 27 15:54 certs
-rw-r----- 1 root root  6703 Jan 27 15:54 clients.conf
-rw-r----- 1 root root   883 Jan 27 15:54 dictionary
-rw-r----- 1 root root 18063 Jan 27 15:54 eap.conf
-rwxr-xr-x 1 root root  4744 Jan 27 15:54 example.pl
-rw-r----- 1 root root 12722 Jan 27 15:54 experimental.conf
-rw-r----- 1 root root  2352 Jan 27 15:54 hints
-rw-r----- 1 root root  1604 Jan 27 15:54 huntgroups
-rw-r----- 1 root root  3218 Jan 27 15:54 ldap.attrmap
drwxr-x--- 2 root root  4096 Jan 27 15:54 modules
-rw-r----- 1 root root  2840 Jan 27 15:54 policy.conf
-rw-r----- 1 root root  4873 Jan 27 15:54 policy.txt
-rw-r----- 1 root root   984 Jan 27 15:54 preproxy_users
-rw-r----- 1 root root 26529 Jan 27 15:54 proxy.conf
-rw-r----- 1 root root 27238 Jan 27 15:54 radiusd.conf
drwxr-x--- 2 root root  4096 Jan 27 15:54 sites-available
drwxr-x--- 2 root root  4096 Jan 27 15:54 sites-enabled
drwxr-x--- 7 root root  4096 Jan 27 15:54 sql
-rw-r----- 1 root root  3042 Jan 27 15:54 sql.conf
-rw-r----- 1 root root  2475 Jan 27 15:54 sqlippool.conf
-rw-r----- 1 root root  3597 Jan 27 15:54 templates.conf
-rw-r----- 1 root root  6524 Jan 27 15:54 users

The default configuration is designed to work everywhere, and to provide nearly every authentication method. Do not edit the default configuration files until you understand what they do. This means reading the documentation contained in the comments of the configuration files.

When the server has been installed on a new machine, the first step is to start it in debugging mode, as user root:

# radiusd -X

This step demonstrates that the server is installed and configured properly. If you have installed Version 2 from source, this step will also create the default certificates used for EAP authentication. If everything went OK, you should see the lines

......
Listening on authentication address * port 1812
Listening on accounting address * port 1813
Listening on command file /usr/local/var/run/radiusd/radiusd.sock
Listening on authentication address 127.0.0.1 port 18120 as server inner-tunnel
Listening on proxy address * port 1814
Ready to process requests.

To stop freeradius press Ctrl+C.

Thats all for now… In next few days I will add more articles about FreeRadius.

mcelog problem

Few servers I maintain totally confused me. The loadavg is steadily increasing every round hour. With top command I can’t see any relevant process which can produce high load.

top - 15:07:17 up 41 days,  3:52,  1 user,  load average: 4.22, 1.61, 0.76
Tasks: 147 total,   1 running, 146 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.7%sy,  0.0%ni, 85.5%id, 13.5%wa,  0.1%hi,  0.2%si,  0.0%st
Mem:   1025084k total,  1016732k used,     8352k free,    24472k buffers
Swap:  2064376k total,      116k used,  2064260k free,   133380k cached
 
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6082 root      15   0  126m 9632 5008 S  0.3  0.9   0:00.50 php
 7363 root      15   0 12736 1112  808 R  0.3  0.1   0:00.03 top
27418 root      15   0  347m 3860 1096 S  0.3  0.4   0:22.80 radiusd
    1 root      15   0 10344  680  568 S  0.0  0.1   0:01.88 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.54 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:15.33 ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:01.81 migration/1
    6 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/1
    7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/1
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:39.01 migration/2
...

The server is CentOS 5.5 64b, quad core Intel processor. After some digging I found out that 4 servers are affected and they are CentOS 5.x 64b. 32bit systems are not affected…

So, the first step is to check cron settings because it is obvious that something is triggered by cron (hourly). Here it is: mcelog.cron. After Googleing about this problem I found this LINK. Or here LINK.

The bug is “closed” but I wouldn’t say so… I had latest mcelog installed and it causes the same problem which is described above.

YUM problem: rpmdb: Lock table is out of available locker entries

Few days ago I had a strange problem with yum and instead of clean install and update process I got Python errors and rpm message “rpmdb: Lock table is out of available locker entries”. After few minutes I found out that during installation or update process, rpm accesses the Berkeley database files and it makes temporary locker entries within the tables while it searches for data. Sometimes the locks are never cleared and we have a problem… Don’t worry… It can be fixed…

First here is the complete error:

[root@myserver ~]# yum install firefox
Loading "fastestmirror" plugin
rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: 3929
error: db4 error(22) from db->close: Invalid argument
error: cannot open Packages index using db3 - Cannot allocate memory (12)
error: cannot open Packages database in /var/lib/rpm
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum-cli/yummain.py", line 85, in main
    base.getOptionsConfig(args)
  File "/usr/share/yum-cli/cli.py", line 163, in getOptionsConfig
    disabled_plugins=self.optparser._splitArg(opts.disableplugins))
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 164, in _getConf
ig
    self._conf = config.readMainConfig(startupconf)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 685, in readMainCo
nfig
    yumvars['releasever'] = _getsysver(startupconf.installroot, startupconf.dist
roverpkg)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 755, in _getsysver
    idx = ts.dbMatch('provides', distroverpkg)
TypeError: rpmdb open failed

It looks scary 🙂

The first step is to backup /var/lib/rpm (in case that something goes wrong) with

# mkdir /backup
# tar cvzf /backup/rpm-backup.tar.gz /var/lib/rpm

You don’t need to backup but it is highly recommended.

Remove the Berkeley databases that rpm uses with

# rm /var/lib/rpm/__db.00*

Note: Probably there will be several files. Confirm all removal with “y”

Make rpm rebuild the databases from scratch (this may take a minute or two)

# rpm --rebuilddb

And that should be enough to fix this problem. Additionally you can list rpms to be sure that everything is OK.

# rpm -qa | sort

Manual MySQL update on RHEL/CentOS

If you’re using RH based distros, you’ll probably notice that their habbit is to keep the same software versions in one release.  For example, if you need PHP on CentOS 5.x, # yum install php will install PHP 5.1.6  (Latest PHP version available on http://php.net is 5.3.x).  If you need MySQL, you can count on MySQL 5.0.xx branch and any other wishes will force you to use independent repos (like http://www.jasonlitka.com/yum-repository/). Another option is to use  “do-it-yourself” method.

In this post I’ll write about manual upgrade steps from rpm archives. Please keep in mind that this procedure works for me and please do not send me the private messages. I can’t help you on that way. Only options is to post your comments here and I will try to solve your problem(s).

First thing you need to do is to see what mysql packages do you have installed

# rpm -qa | grep -i ^mysql

You will get something like

mysql-connector-odbc-3.51.12-2.2
mysql-5.0.77-4.el5_5.3
MySQL-python-1.2.1-1
mysql-server-5.0.77-4.el5_5.3
mysql-devel-5.0.77-4.el5_5.3
mysql-bench-5.0.77-4.el5_5.3

Then backup all your databases, save them on the safe location, protect with alarms, guards, poison dogs, cobras, ninjas, etc…

Download rpms from http://www.mysql.com/downloads/mysql/ (MySQL-client-community-5.1.50-1.rhel5.i386.rpm, MySQL-devel-community-5.1.50-1.rhel5.i386.rpm, MySQL-server-community-5.1.50-1.rhel5.i386.rpm, MySQL-shared-community-5.1.50-1.rhel5.i386.rpm)

Note: if you have 32-bit OS, download i386 rpms, if you have 64bit download x86_64 rpms)

Stop mysql server with

# service mysqld stop

Then remove mysql rpms but with –nodeps option (in case you use yum remove mysql, you will need to reinstall a lot of apps because they will be deleted too)

# rpm -e --nodeps mysql

Repeat the same with other mysql packages (devel, bench, client,…). Then you need to install downloaded rpms with rpm -i mysql… and do it.

It is possible to receive the errors like

ls: /var/lib/mysql/*.err: No such file or directory
ls: /var/lib/mysql/*.err: No such file or directory
ERROR: 1136  Column count doesn't match value count at row 1
100910 10:24:00 [ERROR] Aborting
 
100910 10:24:00 [Note] /usr/sbin/mysqld: Shutdown complete
 
Installation of system tables failed!  Examine the logs in /var/lib/mysql for more information.

This will be fixed later…

Now you need to add .my.cnf file to your root dir so you can execute mysql_upgrade command (note that . before m means that file is hidden). Add next lines inside this file (and don’t forget to replace rootpass with your real mysql root pass)

[client]
user=root
password=rootpass

(chmod .my.cnf to 600 for security reasons)

Then exec next command

# service mysql start
# mysql_upgrade

This will produce the similar output

Looking for 'mysql' as: mysql
Looking for 'mysqlcheck' as: mysqlcheck
Running 'mysqlcheck with default connection arguments
Running 'mysqlcheck with default connection arguments
blabla.table1                            OK
blabla.table2                             OK
blabla.table3                                OK
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.event                                        OK
mysql.func                                         OK
mysql.general_log
Error    : You can't use locks with log tables.
status   : OK
mysql.help_category
error    : Table upgrade required. Please do "REPAIR TABLE `help_category`" or dump/reload to fix it!
mysql.help_keyword
error    : Table upgrade required. Please do "REPAIR TABLE `help_keyword`" or dump/reload to fix it!
mysql.help_relation                                OK
database2.cache
error    : Table upgrade required. Please do "REPAIR TABLE `cache`" or dump/reload to fix it!
database2.contacts                             OK
database2.identities                           OK
database2.messages
error    : Table upgrade required. Please do "REPAIR TABLE `messages`" or dump/reload to fix it!
database2.session
error    : Table upgrade required. Please do "REPAIR TABLE `session`" or dump/reload to fix it!
database2.users
error    : Table upgrade required. Please do "REPAIR TABLE `users`" or dump/reload to fix it!
 
Repairing tables
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_topic                                   OK
mysql.proc                                         OK
mysql.time_zone_name                               OK
database2.cache
note     : The storage engine for the table doesn't support repair
database2.messages
note     : The storage engine for the table doesn't support repair
database2.session
note     : The storage engine for the table doesn't support repair
database2.users
note     : The storage engine for the table doesn't support repair
Running 'mysql_fix_privilege_tables'...
OK

As you can see, the database database2 can’t be repaired and you should drop all tables inside this database and import your backup. After this, you can check is everything ok with mysql_upgrade –force

Once again, restart mysql with service mysql restart and check logs. Test is everything ok, try ti create a new database, optimize your installation, eat something…

Note:
I had a lot of problems with this upgrade. I had to upgrade PHP to 5.3.x, I had to recompile postfix with MySQL support, I had to download and recompile Dovecot because Dovecot from CentOS repos is compiled with mysql 5.0 branch. Some versions of RoundCube doesn’t work with php 5.3.x so you should download latest, etc etc… It can be done but please be careful with this. Who knows which nuclear reactor will explode after this 😛

Upgrading PHP and MySQL on CentOS or RHEL

I already wrote about upgrading via Jason Litka repo on this PAGE but in case you have problems with this repo, you can add Remi Collet repo.

First, import Remi GPG key with

#rpm --import http://rpms.famillecollet.com/RPM-GPG-KEY-remi
# cd /etc/yum.repos.d
# wget http://rpms.famillecollet.com/enterprise/remi.repo

This file provides configuration for remi and remi-test repositories. Keep in mind that Remi repo is disabled by default so you can add –enablerepo=remi to yum command or you can edit line enabled=0 to enabled=1 inside remo.repo. For production servers I don’t recommend enabled=1 to remi-test repo.

Current PHP is 5.3.3 and MySQL 5.1.50

fsck in CentOS 5.x howto

fsck is used to check and optionally repair one or more Linux file systems. File system can be a device name (e.g. /dev/sda2), a mount point (e.g. /, /usr,… ), or an ext2 label or UUID specifier. By default, the fsck will try to handle filesystems on different physical disk drives in parallel to reduce the total amount of time needed to check all of the filesystems.

Continue reading fsck in CentOS 5.x howto

Releasing a message from a quarantine with amavisd-relase

amavisd-new is a high-performance and reliable interface between mailer (MTA) and one or more content checkers: virus scanners, and/or Mail::SpamAssassin Perl module. It is written in Perl, ensuring high reliability, portability and maintainability. It talks to MTA via (E)SMTP or LMTP protocols, or by using helper programs. No timing gaps exist in the design, which could cause a mail loss.

In other words, amavisd-new will help you to fight against spam. In this post, I won’t write about installation (coming soon in you theaters)

This post is just a small trick which will help you to release specific message from quarantine (false positive or you simple want to read spam messages)

Fist you need to find message inside the messages log file (usually /var/log/messages)

May 10 10:06:56 ns1 amavis[12774]: (12774-13) Blocked SPAM, [207.46.22.35] [207.46.22.35] <cnfrmpro@microsoft.com> -> <mymail@domain.tld>, quarantine: spam-1lvc624m6MVB.gz, Message-ID: <BY2MSFTVSMTP03Dfn8e0003d305@by2msftvsmtp03.phx.gbl>, mail_id: 1lvc624m6MVB, Hits: 7.743, size: 3013, 4325 ms

As you can see above, it is spam-1lvc624m6MVB.gz

Now you can release specific message with

[root@s1 ~]# amavisd-release spam-1lvc624m6MVB.gz

And you will see something like

250 2.0.0 Ok, id=rel-1lvc624m6MVB, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 403206AF07CE

Now you just need to check your inbox and you should see the message.

Config mta – howto

Centos has a neat application for switching between alternative software packages, called alternatives.

Few days ago I noticed that one server doesn’t send logwatch email. I wanted to see what was the reason and here are the few tips you can check before you dig inside logwatch settings.

First, check /etc/aliases and root email inside

# nano /etc/aliases

at the end, check next lines:

# Person who should get root's mail
root:           blabla@domain.tld

After you save aliases, enter command

# newaliases

This command rebuilds the random access data base for the mail aliases file /etc/aliases. It must be run each time this file is changed in order for the change to take effect. This would be enough to receive all email directed to root, but in case you still don’t get root emails, check mta with:

# alternatives --display mta

This will show you something like

[root@s1 ~]# alternatives --display mta
mta - status is manual.
 link currently points to /usr/sbin/sendmail.sendmail
/usr/sbin/sendmail.sendmail - priority 90
 slave mta-pam: /etc/pam.d/smtp.sendmail
 slave mta-mailq: /usr/bin/mailq.sendmail
 slave mta-newaliases: /usr/bin/newaliases.sendmail
 slave mta-rmail: /usr/bin/rmail.sendmail
 slave mta-sendmail: /usr/lib/sendmail.sendmail
 slave mta-mailqman: /usr/share/man/man1/mailq.sendmail.1.gz
 slave mta-newaliasesman: /usr/share/man/man1/newaliases.sendmail.1.gz
 slave mta-aliasesman: /usr/share/man/man5/aliases.sendmail.5.gz
 slave mta-sendmailman: /usr/share/man/man8/sendmail.sendmail.8.gz
/usr/sbin/sendmail.postfix - priority 30
 slave mta-pam: /etc/pam.d/smtp.postfix
 slave mta-mailq: /usr/bin/mailq.postfix
 slave mta-newaliases: /usr/bin/newaliases.postfix
 slave mta-rmail: /usr/bin/rmail.postfix
 slave mta-sendmail: /usr/lib/sendmail.postfix
 slave mta-mailqman: /usr/share/man/man1/mailq.postfix.1.gz
 slave mta-newaliasesman: /usr/share/man/man1/newaliases.postfix.1.gz
 slave mta-aliasesman: /usr/share/man/man5/aliases.postfix.5.gz
 slave mta-sendmailman: /usr/share/man/man1/sendmail.postfix.1.gz
Current `best' version is /usr/sbin/sendmail.sendmail.

We use postfix so we should change this with

[root@s1 ~]# alternatives --config mta
 
There are 2 programs which provide 'mta'.
 
  Selection    Command
-----------------------------------------------
*+ 1           /usr/sbin/sendmail.sendmail
   2           /usr/sbin/sendmail.postfix
 
Enter to keep the current selection[+], or type selection number:

Enter 2 and press Enter. Then check your mta with

[root@s1 ~]# alternatives --display mta
mta - status is manual.
 link currently points to /usr/sbin/sendmail.postfix

That’s it… Now if your logwatch is configured properly you should receive root emails…