How to setup compression for high volume zabbix server

Jephe Wu - http://linuxtechres.blogspot.com 

Environment: CentOS 5 64bit, zabbix server with very large items.
Objective: make 'latest data' with group and host set to 'all' page loading faster.


Steps:
1. make sure deflate_module is enabled in apache configuration

LoadModule deflate_module modules/mod_deflate.so

2. filtered by type
for example:

<Location />
    AddOutputFilterByType DEFLATE text/html text/plain text/xml text/x-js text/css
</Location>


Note: you can use Lynx or similar to do a headers dump on the file and it will tell you want the mime type is,

e.g.:

$ lynx -head -dump http://www.example.com/js/jquery-1.2.6.min.js
HTTP/1.1 200 OK
Date: Wed, 25 Jun 2008 03:25:53 GMT
Server: Apache/2.2.3 (CentOS)
Last-Modified: Wed, 25 Jun 2008 03:24:47 GMT
ETag: "38016-d9de-37f45dc0"
Accept-Ranges: bytes
Content-Length: 55774
Connection: close
Content-Type: application/x-javascript

<Location />
    AddOutputFilterByType DEFLATE application/x-javascript
</Location>

--------------------
3. example settings
root@zabbixserver01:/var/log/httpd/ # more /etc/httpd/conf.d/deflate.conf
<IfModule mod_deflate.c>
        AddOutputFilterByType DEFLATE text/plain
        AddOutputFilterByType DEFLATE text/html
        AddOutputFilterByType DEFLATE text/xml
        AddOutputFilterByType DEFLATE text/css
        AddOutputFilterByType DEFLATE application/xml
        AddOutputFilterByType DEFLATE application/xhtml+xml
        AddOutputFilterByType DEFLATE application/rss+xml
        AddOutputFilterByType DEFLATE application/javascript
        AddOutputFilterByType DEFLATE application/x-javascript
        DeflateCompressionLevel 9
        BrowserMatch ^Mozilla/4 gzip-only-text/html
        BrowserMatch ^Mozilla/4\.0[678] no-gzip
        BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

    # Don't compress images
    SetEnvIfNoCase Request_URI \
    \.(?:gif|jpe?g|png)$ no-gzip dont-vary

    # Make sure proxies don't deliver the wrong content
    Header append Vary User-Agent env=!dont-vary
       
    DeflateFilterNote Input instream
        DeflateFilterNote Output outstream
        DeflateFilterNote Ratio ratio
        LogFormat '"%r" %{outstream}n/%{instream}n (%{ratio}n%%)' deflate
    CustomLog logs/deflate_log deflate
</IfModule>


4. References
a. Apache mod_deflate example:
http://httpd.apache.org/docs/2.0/mod/mod_deflate.html

b. http://www.linuxjournal.com/article/6802?page=0,1

c. use php zlib compress by modifying php.ini

PHP with the --with-zlib configure option and then reconfiguring the php.ini file.

Below is what the output buffer method looks like:

output_buffering = On
output_handler = ob_gzhandler
zlib.output_compression = Off


The ZLIB method uses:

output_buffering = Off
output_handler =
zlib.output_compression = On

How dbbix works with Zabbix

Jephe Wu - http://linuxtechres.blogspot.com

Environment
: dbforbix running on separated CentOS 5 box, Zabbix server or zabbix proxy is on another server
Objective: use separated server running dbbix to serve database performance check and show graph on zabbix web gui
Concept:
dbforbix collects data from separated server, then send it to zabbix server(destination server was configured without proxy) or zabbix proxy(server was configured with proxy)


Steps:
1. on dbforbix, configure it as follows:

ZabbixServerList=zabbixserver01,zabbixproxy01
zabbixserver01.Address=172.16.4.1
zabbixserver01.Port=10051

zabbixproxy01.Address=172.16.4.2
zabbixproxy01.Port=10051

DatabaseList=db01,db02


db01.Url=jdbc:mysql://db01:3306/mysqldb01.User=zabbix
db01.Password=zabbix
db01.DatabaseType=mysql
db01.QueryListFile=./conf/mysql/mysqlquery.props

oradb-01.Url=jdbc:oracle:thin:@oradb-01-vip:1521:LIVEDB1
oradb-01.User=zabbix
oradb-01.Password=zabbix
oradb-01.QueryListFile=./conf/oracle/query.props

dbent.Url=jdbc:jtds:sqlserver://172.16.7.2:1987/master
dbent.User=zabbix
dbent.Password=zabbix
dbent.DatabaseType=mssql
dbent.QueryListFile=./conf/mssql/mssqlquery.props

host1.Url=jdbc:postgresql://host1:5432/databasename
host1.User=zabbix
host1.Password=zabbix
host1.DatabaseType=pgsql
host1.QueryListFile=./conf/pgsql/pgsqlquery.props

.....

2. debug if the gui is not updating data

make sure dbforbix server can telnet zabbix server or proxy at port 10051, if not, make sure proxy or server is not running iptables.
otherwise, you will get !X (man traceroute, it's administratively prohibitted indication))
use traceroute or tcptraceroute to troubleshoot.

3. installation guide:
refer to http://www.smartmarmot.com/wiki/index.php/DBforBIX

How to use kickstart to install CentOS 6.1 on non-DHCP network

Jephe Wu - http://linuxtechres.blogspot.com

Environment: HP BladeSystem Proliant BL460c G6(ip: 192.168.0.0/24, DHCP/tftp/kickstart/PXE/HTTP server is on different network: 192.168.1.0/24)
Objective: install CentOS 6.1 into bare-metal blades from

Difficulties: We are not able to boot from PXE as it's in different network.
Concepts:
We can use CentOS netinstall ISO CD to boot up blades and specify dhcp/tftp/http server as installation source and use kickstart to install it automatically.

We can further modify CentOS netinstall iso CD to add kickstart location to make it automatically installed through kickstart


Steps:
1. make customized netinstall CentOS 6.1 CD

download iso from http://mirror.optus.net/centos/6.1/isos/x86_64/CentOS-6.1-x86_64-netinstall.iso and put it /root

cd /root; mkdir c
mount CentOS-6.1-x86_64-netinstall.iso c -o loop
cp -va c c.1
cd c.1
rm -fr images
cd isolinux
vi isolinux.cfg
-------------
default linux
prompt 0
timeout 1

display boot.msg

menu background splash.jpg
menu title Welcome to CentOS 6.2!
menu color border 0 #ffffffff #00000000
menu color sel 7 #ffffffff #ff000000
menu color title 0 #ffffffff #00000000
menu color tabmsg 0 #ffffffff #00000000
menu color unsel 0 #ffffffff #00000000
menu color hotsel 0 #ff000000 #ffffffff
menu color hotkey 7 #ffffffff #ff000000
menu color scrollbar 0 #ffffffff #00000000

label linux
  menu label ^Install or upgrade an existing system
  menu default
  kernel vmlinuz
  append initrd=initrd.img ks=http://jephe.server/ks_centos_6_x86_64 text ksdevice=eth0 gateway=192.168.0.254 ip=192.168.0.1 dns=192.168.1.1 netmask=255.255.255.0 --onboot=on
label vesa
  menu label Install system with ^basic video driver
  kernel vmlinuz
  append initrd=initrd.img xdriver=vesa nomodeset
label rescue
  menu label ^Rescue installed system
  kernel vmlinuz
  append initrd=initrd.img rescue
label local
  menu label Boot from ^local drive
  localboot 0xffff
label memtest86
  menu label ^Memory test
  kernel memtest
  append -
-------------------
cat > /root/c.1/mkiso.sh << END
mkisofs -o ../CentOS-6.1_x86_64-Kickstart-192.168.0.1.iso -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -R -J -v -T .
END
-------------------
then, cd /root/c.1, run
./mkiso.sh
it will create the iso file under /root

2. prepare kickstart file http://jephe.server/ks_centos_6_x86_64

Here is the KS file I used:
-------------
install
url --url=http://jephe.server/packages/centos_6_x86_64
text
lang en_US
keyboard us
rootpw password
firewall --disabled
services --disabled=NetworkManager,abrtd,acpid,certmonger,cgconfig,cgred,corosync,cpuspeed,cups,dovecot,ip6tables,iptables,kdump,modclusterd,multipathd,mysql-mmm-agent,mysql-proxy,netconsole,nfs,nscd,nslcd,ntpdate,oddjobd,psacct,quota_nld,rdisc,restorecond,rpcbind,rpcgssd,rpcidmapd,rpcsvcgssd,saslauthd,smartd,snmptrapd,spamassassin,sssd,vncserver,ypbind
firstboot --disable
selinux --disabled
authconfig --enableshadow --passalgo=sha512 --enablefingerprint
timezone Australia/Sydney
bootloader --location=mbr --driveorder=sda --append="rhgb crashkernel=auto rhgb quiet"
zerombr yes
clearpart --all --initlabel
part /boot --fstype=ext3 --size=256
part swap --asprimary --size=8192
part / --fstype=ext4 --grow --asprimary --size=200
%packages --ignoremissing
@base
@client-mgmt-tools
@compat-libraries
@console-internet
@core
@debugging
@directory-client
@mail-server
@hardware-monitoring
@java-platform
@large-systems
@mysql-client
@mysql
@network-file-system-client
@performance
@perl-runtime
@server-platform
@server-policy
@storage-client-multipath
@system-admin-tools
pax
oddjob
sgpio
certmonger
pam_krb5
krb5-workstation
nscd
pam_ldap
nss-pam-ldapd
perl-DBD-MySQL
perl-DBD-SQLite
screen
vlock
lsscsi
sendmail
sendmail-cf
lrzsz
dos2unix
unix2dos
tigervnc

%pre --log=/tmp/pre.log
ETH=eth0
IP=`ifconfig $ETH | grep "inet " | cut -d : -f 2 | cut -d " " -f 1`
MAC=`ifconfig $ETH | grep HWaddr | awk -F "HWaddr " '{print $2}'`
if [ $MAC == "00:00:B3:B0:44:00" ];then
    NAME=db01.domain
    SHORTNAME=db01
fi
if [ $MAC == "00:00:B3:B0:43:18" ];then
    NAME=db02.domain
    SHORTNAME=db02
fi
echo "NETWORKING=yes" > /tmp/network.txt
echo "HOSTNAME=${NAME}" >> /tmp/network.txt

cat > /tmp/hosts.txt <<'EOF'
127.0.0.1       localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
EOF
echo "$IP $NAME $SHORTNAME" >> /tmp/hosts.txt

%post --nochroot --log=/tmp/post.log
cp /tmp/network.txt /mnt/sysimage/etc/sysconfig/network
. /mnt/sysimage/etc/sysconfig/network
/mnt/sysimage/bin/hostname $HOSTNAME
cp /tmp/hosts.txt  /mnt/sysimage/etc/hosts
/usr/bin/cp /tmp/pre.log /mnt/sysimage/root/pre.log
/usr/bin/cp /tmp/post.log /mnt/sysimage/root/post.log

# if you use rollout for server configuration deployment. ( search 'rollout google' in google)
echo "Kickstarted `/usr/bin/date`" > /mnt/sysimage/root/kickstart_report
wget -O /mnt/sysimage/usr/local/sbin/rollout http://server.domain/rollout
sed -i -e 's#http://server.domain#http://rollout.domain#' /mnt/sysimage/usr/local/sbin/rollout
chmod +x /mnt/sysimage/usr/local/sbin/rollout

echo "Kickstart complete" >> /mnt/sysimage/root/kickstart_report
-------------


3. use HP ILO to boot from above-made CentOS netinstall ISO CD.

it will use kernel and initrd files from customized netinstall iso CD to boot up machine, and install packages from approved specified kickstart file
http://jephe.server/packages/centos_6_x86_64

For configuration files under pxelinux.cfg directory, you can use something like this:
append initrd=centos_6_x86_64/initrd.img ramdisk_size=8192 ks=http://kickstart.jephe/ks_centos_6_x86_64 ip=dhcp ksdevice=eth0

4. configuring bonding device

[root@db01 network-scripts]# more /etc/modprobe.d/bonding.conf
alias eth0 e1000e
alias eth1 e1000e
alias eth2 e1000e
alias eth3 e1000e
#alias eth4 bnx2x
#alias eth5 bnx2x
alias scsi_hostadapter lpfc
alias bond0 bonding

ifcfg-bond0
::::::::::::::
DEVICE=bond0
GATEWAY=192.168.0.254
IPADDR=192.168.0.1
IPV6INIT=no
NETMASK=255.255.255.0
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
BONDING_OPTS="mode=1 miimon=100"
::::::::::::::
ifcfg-eth0
::::::::::::::
DEVICE=eth0
IPV6INIT=no
MASTER=bond0
MTU=1500
ONBOOT=no
PEERDNS=yes
SLAVE=yes
TYPE=Ethernet
USERCTL=no
::::::::::::::
ifcfg-eth1
::::::::::::::
# Intel Corporation 82571EB Quad Port Gigabit Mezzanine Adapter
DEVICE=eth1
HWADDR=00:00:00:B0:44:01
ONBOOT=no
HOTPLUG=no
::::::::::::::
ifcfg-eth2
::::::::::::::
DEVICE=eth2
IPV6INIT=no
MASTER=bond0
MTU=1500
ONBOOT=no
PEERDNS=yes
SLAVE=yes
TYPE=Ethernet
USERCTL=no
::::::::::::::
ifcfg-eth3
::::::::::::::
# Intel Corporation 82571EB Quad Port Gigabit Mezzanine Adapter
DEVICE=eth3
HWADDR=00:00:00:B0:44:03
ONBOOT=no
HOTPLUG=no
::::::::::::::
::::::::::::::
ifcfg-eth4
::::::::::::::
# Broadcom Corporation NetXtreme II BCM57711E 10-Gigabit PCIe
DEVICE=eth4
HWADDR=F4:CE:46:7E:00:00
ONBOOT=no
HOTPLUG=no
::::::::::::::
ifcfg-eth5
::::::::::::::
# Broadcom Corporation NetXtreme II BCM57711E 10-Gigabit PCIe
DEVICE=eth5
HWADDR=F4:CE:46:7E:00:01
ONBOOT=no
HOTPLUG=no
::::::::::::::
ifcfg-lo
::::::::::::::
DEVICE=lo
IPADDR=127.0.0.1
NETMASK=255.0.0.0
NETWORK=127.0.0.0
# If you're having problems with gated making 127.0.0.0/8 a martian,
# you can change this to something else (255.255.255.255, for example)
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback

Note: prior to CentOS 6, you may add bonding options as follows in /etc/modprobe.d/bonding.conf:
options bonding mode=1 miimon=100

How to setup MMM control under CentOS 6.1

Jephe Wu - http://linuxtechres.blogspot.com

Environment: CentOs 6.1, MMM 2.2.1, MySQL 5.1.52
Objective: setup MMM among the databases to gain HA and load balance for read access


Steps:

1. install required packages for mmm agent on database servers
yum -y install perl-Config-IniFiles
yum -y install mysql-mmm mysql-mmm-agent mysql-mmm-tools mysql-proxy

2. install required packages for mmm monitor on monitoring host

yum -y install mysql-mmm-monitor

chkconfig mysql-mmm-agent on
chkconfig mysql-mmm-monitor on

3. preparing ips for databases
db1: 172.17.7.1 master
db2: 172.17.7.2 master
db3: 172.17.7.3 slave

master exclusive ip: 172.17.7.4
slave balanced ip addresses: 172.17.7.5, 172.17.7.6, 172.17.7.7

4. setup /etc/mysql-mmm/mmm_agent.conf to change 'this' line to db1/db2 or db3

5. setup /etc/mysql-mmm/mmm_common.conf for all monitor and agents

# more /etc/mysql-mmm/mmm_common.conf
active_master_role          writer

<host default>
    cluster_interface       bond0

    pid_path                /var/run/mysql-mmm/mmm_agentd.pid
    bin_path                /usr/libexec/mysql-mmm/

    replication_user        replication
    replication_password    replication_password

    agent_user              mmm_agent
    agent_password          agent_password
</host>

<host db1>
    ip                      172.17.7.1
    mode                    master
    peer                    db2
</host>

<host db2>
    ip                      172.17.7.2
    mode                    master
    peer                    db1
</host>

<host db3>
    ip                      172.17.7.3
    mode                    slave
</host>

<role writer>
    hosts                   db1, db2
    ips                     172.17.7.4
    mode                    exclusive
</role>

<role reader>
    hosts                   db1, db2, db3
    ips                     172.17.7.5, 172.17.7.6, 172.17.7.7
    mode                    balanced
</role>

6. start up agents on database servers first

on db1,db2 then db3, to start up mmm agent: /etc/init.d/mysql-mmm-agent start

7. start up monitor process on monitoring host

on mon01, to start up: /etc/init.d/mysql-mmm-monitor start

8. bring up them online from mon01

mon01# mmm_control set_online db1
mon01# mmm_control set_online db2
mon01# mmm_control set_online db3

root@mon01:/etc/mysql-proxy/ # mmm_control show
  db1(172.17.7.1) slave/REPLICATION_DELAY. Roles:
  db2(172.17.7.2) master/ONLINE. Roles: reader(172.17.7.5), reader(172.17.7.6)
  db3(172.17.7.3) master/ONLINE. Roles: reader(172.17.7.7), writer(172.17.7.4)

9. FAQ
a.   read_only (on/off) from show variables comand in mysql
if db2 and db3 are writer, when writer role goes to db3, mmm will set read_only flag for db2 to prevent any writing process, but only for normal user, not root.