Skip to content



Fix OpenSSL bug – my way

Fix OpenSSL bug – my way

On 04/11/2014 06:02 PM, Marinko Tarlać via PayPal wrote:
> PayPal <https://www.paypal.com/us>
>
> Hello OpenSSL Software Foundation,
>
> This email confirms that you have received a donation of $xxxxx USD from
> Marinko Tarlać(mtarlac@xxxxx <mailto:mtarlac@xxxxx>). ...
 
We received your donation of US$xxxxx. Thank you for your support of
the OpenSSL project!
 
-Steve M.
 
-- Steve Marquess OpenSSL Software Foundation

Posted in Other.


Heart Bleed Bug – OpenSSL – part 2

I maintain more than 30 servers and several of them was affected with Heartbleed bug. CentOS released update for OpenSSL package(s) so there are no excuses not to update (yum update openssl, … ).

In the meantime, there are hundreds of sysadmins which still didn’t do anything to protect their servers and clients (https://gist.github.com/dberkholz/10169691).

Testing REMOVED.com for example:

boky@bojler ~/Downloads $ ./test.py REMOVED.com
Connecting...
Sending Client Hello...
Waiting for Server Hello...
 ... received message: type = 22, ver = 0302, length = 58
 ... received message: type = 22, ver = 0302, length = 4837
 ... received message: type = 22, ver = 0302, length = 4
Sending heartbeat request...
 ... received message: type = 24, ver = 0302, length = 16384
Received heartbeat response:
  0000: 02 40 00 D8 03 02 53 43 5B 90 9D 9B 72 0B BC 0C  .@....SC[...r...
  0010: BC 2B 92 A8 48 97 CF BD 39 04 CC 16 0A 85 03 90  .+..H...9.......
  0020: 9F 77 04 33 D4 DE 00 00 66 C0 14 C0 0A C0 22 C0  .w.3....f.....".
  0030: 21 00 39 00 38 00 88 00 87 C0 0F C0 05 00 35 00  !.9.8.........5.
  0040: 84 C0 12 C0 08 C0 1C C0 1B 00 16 00 13 C0 0D C0  ................
  0050: 03 00 0A C0 13 C0 09 C0 1F C0 1E 00 33 00 32 00  ............3.2.
  0060: 9A 00 99 00 45 00 44 C0 0E C0 04 00 2F 00 96 00  ....E.D...../...
  0070: 41 C0 11 C0 07 C0 0C C0 02 00 05 00 04 00 15 00  A...............
  0080: 12 00 09 00 14 00 11 00 08 00 06 00 03 00 FF 01  ................
  0090: 00 00 49 00 0B 00 04 03 00 01 02 00 0A 00 34 00  ..I...........4.
  00a0: 32 00 0E 00 0D 00 19 00 0B 00 0C 00 18 00 09 00  2...............
  00b0: 0A 00 16 00 17 00 08 00 06 00 07 00 14 00 15 00  ................
  00c0: 04 00 05 00 12 00 13 00 01 00 02 00 03 00 0F 00  ................
  00d0: 10 00 11 00 23 00 00 00 0F 00 01 01 67 3A 20 67  ....#.......g: g
  00e0: 7A 69 70 2C 20 64 65 66 6C 61 74 65 0D 0A 52 65  zip, deflate..Re
  00f0: 66 65 72 65 72 3A 20 68 74 74 70 73 3A 2F 2F 77  ferer: https://w
  0100: 77 77 2E 74 6F 73 68 69 62 61 2E 63 6F 6D 2F 74  ww.REMOVED.com/t
  0110: 69 63 2F 70 72 6F 64 75 63 74 2F 76 32 30 30 30  ic/product/v2000
  0120: 2D 73 65 72 69 65 73 2D 73 6D 61 6C 6C 2D 70 6C  -series-small-pl
  0130: 63 73 0D 0A 43 6F 6F 6B 69 65 3A 20 4A 53 45 53  cs..Cookie: JSES
  0140: 53 49 4F 4E 49 44 3D 44 39 37 36 34 38 30 32 30  SIONID=D97648020
  0150: 41 45 36 32 31 46 45 41 31 44 38 45 30 37 33 42  AE621FEA1D8E073B
  0160: 42 38 31 44 44 32 36 2E 74 61 3B 20 63 69 74 72  B81DD26.ta; citr
  0170: 69 78 5F 6E 73 5F 69 64 3D 62 35 53 33 58 6A 6B  ix_ns_id=b5S3Xjk
  0180: 4A 49 59 4B 53 31 6E 42 2F 31 45 73 4B 6C 58 46  JIYKS1nB/1EsKlXF
  0190: 6D 70 71 45 41 30 30 30 0D 0A 43 6F 6E 6E 65 63  mpqEA000..Connec
  01a0: 74 69 6F 6E 3A 20 6B 65 65 70 2D 61 6C 69 76 65  tion: keep-alive
  01b0: 0D 0A 49 66 2D 4D 6F 64 69 66 69 65 64 2D 53 69  ..If-Modified-Si
  01c0: 6E 63 65 3A 20 54 75 65 2C 20 30 35 20 4E 6F 76  nce: Tue, 05 Nov
  01d0: 20 32 30 31 33 20 31 34 3A 32 30 3A 33 34 20 47   2013 14:20:34 G
  01e0: 4D 54 0D 0A 0D 0A 69 65 1F 0E 88 65 6C 48 9C E1  MT....ie...elH..
  01f0: 7C 8F FD AC 1C 93 A1 A8 7E 9F 00 00 00 00 00 00  |.......~.......
  0200: 0D 0A 49 66 2D 4E 6F 6E 65 2D 4D 61 74 63 68 3A  ..If-None-Match:
  0210: 20 22 31 61 66 38 36 31 2D 37 34 2D 34 64 66 32   "1af861-74-4df2
  0220: 32 34 31 34 38 39 33 30 30 22 0D 0A 0D 0A 4E 1A  241489300"....N.
....
  3fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  3fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  3fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  3ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
 
WARNING: server returned more data than it should - server is vulnerable!

For security reasons, real domain which I tested is replaced with “REMOVED”

Some hosts from the list I posted above are already patched (which is good)

boky@bojler ~/Downloads $ ./test.py zoho.com
Connecting...
Sending Client Hello...
Waiting for Server Hello...
 ... received message: type = 22, ver = 0302, length = 66
 ... received message: type = 22, ver = 0302, length = 2399
 ... received message: type = 22, ver = 0302, length = 331
 ... received message: type = 22, ver = 0302, length = 4
Sending heartbeat request...
Unexpected EOF receiving record header - server closed connection
No heartbeat response received, server likely not vulnerable

Throwing rocks to OpenSSL developers is not the good idea. Donating money for paid developers is much better option…

Posted in Linux, Other.


Heart Bleed Bug – OpenSSL

A massive vulnerability has been found in OpenSSL, the open-source software package broadly used to encrypt Web communications. The flaw allows attackers to steal the information that is normally protected by SSL/TLS encryption (web applications, e-mail, instant messaging, VPNs, etc).

Essentially, that means a lot of Internet users are affected and passwords and credit card information could be available to hackers.

CentOS released the updated OpenSSL packages which should fix this issue.

# yum update openssl
# service httpd restart

For more information:
http://www.exploit-db.com/exploits/32745/
http://heartbleed.com/

Posted in CentOS, Server project, Tips & Tricks.


Quick search and replace inside file

sed -i 's/original/new/g' file.txt

Where

  • sed means Stream EDitor
  • -i – in-place (for example save back to the original file)
  • s – the substitute command
  • original – a regular expression describing the word to replace (or just the word itself)
  • new – the text to replace it with
  • g – global (replace all and not just the first occurrence)
  • myfile.txt – the file name

 

Posted in Tips & Tricks.


Postfix queue monitoring script

Here is a small Perl script which can be useful as a Postfix queue monitor (count number of emails in Postfix queue)

#!/usr/bin/env perl
 
# postfix queue/s size
# author:
# source: http://tech.groups.yahoo.com/group/postfix-users/message/255133
 
use strict;
use warnings;
use Symbol;
sub count {
	my ($dir) = @_;
        my $dh = gensym();
        my $c = 0;
        opendir($dh, $dir) or die "$0: opendir: $dir: $!\n";
        while (my $f = readdir($dh)) {
                if ($f =~ m{^[A-F0-9]{5,}$}) {
                        ++$c;
                } elsif ($f =~ m{^[A-F0-9]$}) {
                        $c += count("$dir/$f");
                }
        }
	closedir($dh) or die "closedir: $dir: $!\n";
        return $c;
}
my $qdir = `postconf -h queue_directory`;
chomp($qdir);
chdir($qdir) or die "$0: chdir: $qdir: $!\n";
printf "Incoming: %d\n", count("incoming");
printf "Active: %d\n", count("active");
printf "Deferred: %d\n", count("deferred");
printf "Bounced: %d\n", count("bounce");
printf "Hold: %d\n", count("hold");
printf "Corrupt: %d\n", count("corrupt");

Save script as queueStatus (for example), chmod +x to make it exec, and output

[root@s1 tmp]# ./queueStatus 
Incoming: 0
Active: 0
Deferred: 8
Bounced: 6
Hold: 0
Corrupt: 0

Posted in Postfix.


Protect wp-login.php with .htaccess

Brute force attack aims at being the simplest kind of method to gain access to a site (wordpress or not). It combines usernames and passwords, over and over again, until it gets in. That is the main reason why you should always use secure passwords and avoid common usernames (admin, siteadmin, etc…)

The simple way to protect your WordPress site from brute force is to lock the access to wp-login.php file with htaccess.

<Files wp-login.php>
Order Deny,Allow
Deny from all
Allow from x.x.x.x
Allow from y.y.y.y
</Files>

You can add as much as you want IPs inside the Files block and all other IPs will be blocked with Error 403 (Forbidden error).

Unfortunately this is not the nicest way because IPs you’re accessing from are not always static…

Posted in Apache, Web Hosting.


How to clone MySQL database

Here is a way to create a duplicate of one database, with all its tables and their data

Dump your source database into sql file

# mysqldump -uroot -p production -r production.sql

If you need only schema (database with empty tables)

# mysqldump -uroot -p production -r production.sql --no-data

Open up a MySQL shell and login as root

# mysql -uroot -p

Create a new database and populate it with the dumped data

CREATE DATABASE production_copy;
USE production_copy;
SOURCE production.SQL;

Now if you like, you can create a new user and give it permissions to the new database

CREATE USER new_user IDENTIFIED BY 'some_password';
GRANT ALL ON production_copy.* TO 'new_user'@'localhost' IDENTIFIED BY 'some_password';
FLUSH PRIVILEGES;

Note: this procedure works on Windows and Linux

Posted in MySQL, Tips & Tricks.


EoIP tunnel on Linux

Ethernet over IP (EoIP) Tunneling is a MikroTik RouterOS protocol (stateless and light ethernet point to point tunnel protocol with 28 bytes static overhead) that creates an Ethernet tunnel between two routers on top of an IP connection. The EoIP tunnel may run over IPIP tunnel, PPTP tunnel or any other connection capable of transporting IP.

To connect Linux with Mikrotik over EoIP tunnel, you’ll need THIS.

# wget --no-check-certificate https://linux-eoip.googlecode.com/files/linux-eoip-0.5.tgz
# tar zxvf linux-eoip-0.5.tgz
# cd linux-eoip-0.5
# ./configure
# make
# make install

Copy eoip.cfg to /etc dir, change settings inside according to your needs and save the file. If you use dynamic=1 option, take attention that there is no authorization,
and it is not secure. It is not good idea to use this feature with public ip or insecure(not completely under your control, each host) network.

For not lets suppose you need only one tunnel to remote IP address 1.1.1.1

[zeoip0]
id=1
dst=1.1.1.1

On Mikrotik create EoIP tunnel with the same ID (1) and set your server’s IP address as remote IP.  Run eoio with

# /usr/local/bin/eoip /etc/eoip.cfg

Add IP address to your eoip interface

# /sbin/ifconfig zeoip0 10.254.254.2 netmask 255.255.255.252 up

And optionally add routes (if you have any)

# route add -net 10.2.0.0/16 gw 10.254.254.1

Add the last few lines inside rc.local to enable tunnel after reboot. The eoip interface can be threaten just like any other interface.

# ifconfig
zeoip0    Link encap:Ethernet  HWaddr 5B:25:C9:44:6A:79  
          inet addr:10.254.254.2  Bcast:10.254.254.3  Mask:255.255.255.252
          inet6 addr: fe80::5425:d9ff:fe80:6b79/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:167397 errors:0 dropped:0 overruns:0 frame:0
          TX packets:138861 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:14934574 (14.2 MiB)  TX bytes:12520192 (11.9 MiB)
# ps ax|grep dhcp
5180 ?        Ss     0:02 /usr/sbin/dhcpd eth1 zeoip0
27356 pts/1    S+     0:00 grep dhcp

As you can see, you can run dhcp server on eoip interface. Just open /etc/sysconfig/dhcpd and add DHCPDARGS=”eth1 zeoip0″ inside. Save the file and restart dhcp server.

Posted in Linux, Server project, Tips & Tricks.


Red Hat and the CentOS join forces

This year started very nice for us…

It seems that Red Hat will try to fight vs Oracle Unbreakable Linux which is very similar to CentOS project. According to press release:

  • Red Hat Enterprise Linux will remain the same (for commercial development and deployment),
  • CentOS provides a base for community adoption and integration of open source technologies on a Red Hat-based platform (community integration beyond the operating system)
  • Fedora will continue to serve as the upstream project on which future Red Hat Enterprise Linux releases are based (mostly untested software for testing and home use).

More info can be found HERE

Posted in CentOS, RedHat.


Slow InnoDB insert/update

If you’re migrating from MyISAM to InnoDB or you’re using MySQL 5.5.x or newer (InnoDB default engine) you’ll probably be disappointed with INSERT/UPDATE queries (with InnoDB tables). InnoDB is a transaction-safe, ACID compliant MySQL storage engine and with default settings, the log buffer is written out to the log file at each transaction commit and the flush to disk operation is performed on the log file. This can be very slow (but very safe – every transaction is 100% written to the disk).

Since MyISAM is not an option, we need to tune up our server so it can be used with InnoDB correctly. According to MySQL site, the next couple things should be considered:

  • Use OPTIMIZE TABLE statement to reorganize the table and compact any wasted space. Of course this operation won’t help if your database is empty
  • Use AUTO_INCREMENT column as the primary key
  • If you’re storing variable-length strings or if the column may contain NULL values, use the VARCHAR data type instead of CHAR (smaller tables fit better in the buffer pool and reduce disk I/O)
  • Since InnoDB must flush the log to disk at each transaction commit (if that transaction made modifications to the database), attach several queries into a single transaction to reduce the number of flush operations
  • In case you’re not building a finance application which can’t afford data loss if a crash occurs, you can set the parameter innodb_flush_log_at_trx_commit parameter to 0. In this case, InnoDB tries to flush the log once per second and not after every transaction (default setting is 1 which mean flush the log after every transaction).
  • To reduce the amount of disk I/O used by queries to access InnoDB tables, you can increase the innodb_buffer_pool_size.
  • Big disk-bound operations are always expensive. Use DROP TABLE and CREATE TABLE to empty a table, not DELETE FROM….Also TRUNCATE TABLE is much faster then DELETE * FROM…
  • innodb_flush_method parameter can also help but you must test yourself to see the right combination for your hardware and your database (possible values: fdatasync, O_DSYNC, O_DIRECT).
  • Make your log files big, even as big as the buffer pool and make the log buffer quite large as well
  • Disable autocommit during import operation (surround it with SET autocommit and COMMIT statements)
    SET autocommit=0;
     SQL queries
    COMMIT;
  • Temporarily turning off the uniqueness checks during the import session will help.
    SET unique_checks=0;
     SQL queries
    SET unique_checks=1;
  • Turn off foreign key checks during imports.
    SET foreign_key_checks=0;
     SQL queries
    SET foreign_key_checks=1;
  • If you often have repeating queries for tables that are not updated frequently, enable the query cache with
    query_cache_type = 1
    query_cache_size = 10M
  • Use the multiple-row INSERT syntax to reduce communication overhead between the client and the server if you need to insert many rows:
    INSERT INTO tbl VALUES (1,2), (5,5), ...;

The list above is not the final one. Please check the next link for more details about those parameters. Link
In my case, I won’t change a lot of parameters. The only parameter which I will change is the innodb_flush_log_at_trx_commit = 1 (default value is 1).

Before and after performance will be tested with Sysbench (Link). Since reading is not problem right now, I’ll stick with the write operations.

R/W test

sysbench --num-threads=16 --max-requests=10000 --test=oltp --oltp-table-size=500000 --mysql-socket=/var/lib/mysql/mysql.sock --oltp-test-mode=complex --mysql-user=TEST_USER --mysql-password=TEST_PASSWORD run

The result

OLTP test statistics:
    queries performed:
        read:                            146216
        write:                           52220
        other:                           20446
        total:                           218882
    transactions:                        10002  (181.90 per sec.)
    deadlocks:                           442    (8.04 per sec.)
    read/write requests:                 198436 (3608.90 per sec.)
    other operations:                    20446  (371.85 per sec.)
 
Test execution summary:
    total time:                          54.9852s
    total number of events:              10002
    total time taken by event execution: 879.1034
    per-request statistics:
         min:                                 33.38ms
         avg:                                 87.89ms
         max:                                480.77ms
         approx.  95 percentile:             135.31ms
 
Threads fairness:
    events (avg/stddev):           625.1250/2.29
    execution time (avg/stddev):   54.9440/0.03

Total time: 54.98s

Now, when I change innodb_flush_log_at_trx_commit to 0 (default value was 1), I get:

OLTP test statistics:
    queries performed:
        read:                            140000
        write:                           50000
        other:                           20000
        total:                           210000
    transactions:                        10000  (780.35 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 190000 (14826.69 per sec.)
    other operations:                    20000  (1560.70 per sec.)
 
Test execution summary:
    total time:                          12.8147s
    total number of events:              10000
    total time taken by event execution: 204.8297
    per-request statistics:
         min:                                  1.19ms
         avg:                                 20.48ms
         max:                               1669.69ms
         approx.  95 percentile:              44.50ms
 
Threads fairness:
    events (avg/stddev):           625.0000/19.56
    execution time (avg/stddev):   12.8019/0.00

Total time: 12.81s

As you can see, changing innodb_flush_log_at_trx_commit from 1 to 0 increases the write speed but we can lose data in some cases (hardware or power failures, etc). To avoid this problem, use battery backups, UPS, RAID, …

Posted in CentOS, MySQL.