Some interesting task related to recursion and arrays in PHP

~ tadmin

Sometimes it is difficult to find best solution for some problem quickly.
For example, if you need to walk some nested array. You don’t know how deep it could be nested and extract values from it to make it flattern you can use amazing function array_walk_recursive together with anonymous function to solve such task like it is shown in example below:

$inputArray = [9, [7, 5, [11, [3]]], 14];
array_walk_recursive($inputArray, function($value, $key) use (&$resultArray) {
    $resultArray[] = $value;
});

This amazing example of magic is very simple: it calls our anonymous function for each $inputArray element, storing nested element values in $resultArray. Note, usage of & in use (&$resultArray) it tells we shoud pass variable by reference.

MySQL GROUP_CONCAT default result length & possible problems with big results

~ tadmin

If you use such function like GROUP_CONCAT you should be aware of default result size is 1024 bytes. It could lead possible problems.

For example, check the SQL query below:

set @user_ids = (SELECT GROUP_CONCAT(user_id ORDER BY user_id SEPARATOR ',') FROM user WHERE enabled = 1);

If you expect result to be list of IDs separated by comma and you have data result bigger than 1024 characters it will be truncated to default length. If you use such result in another SQL queries such as:

SELECT some_stuff FROM user_stuff WHERE user_id IN (@user_ids);

It could lead to possible logical problems when some IDs will be cropped and wrong data will be used.
To solve such problem you can change session variable using SQL query:

SET SESSION group_concat_max_len = 1000000;

Where 1000000 is maximum possible GROUP_CONCAT result size.
Also, you can set this variable globally (it is preffered way if you work with big data in most cases. Then you should add such configuration to your /etc/my.cnf

group_concat_max_len = 1000000;

and restart MySQL server to apply new default configuration.

OpenSSL generate CSR using secure SHA256 instead of SHA1

OpenSSL
~ tadmin

To generate RSA private key using 2048 long modulus use command:

# openssl genrsa -out www.shkodenko.com.key 2048

If you want to password protect your key add -des3 parameter. You can also increase key modulus to 4096 to make it more secure.

To generate CSR using secure SHA256 algorithm the following command can be used:

# openssl req -sha256 -new -key www.shkodenko.com.key -out www.shkodenko.com-sha256.csr

There is also genkey utility. By default it will generate CSR with SHA1 but you can change it in configuration file /etc/pki/tls/openssl.cnf:

# diff -Nau /etc/pki/tls/openssl.cnf-2015-06-26.bak /etc/pki/tls/openssl.cnf

— /etc/pki/tls/openssl.cnf-2015-06-26.bak 2015-01-13 13:34:32.000000000 +0000
+++ /etc/pki/tls/openssl.cnf 2015-06-26 16:13:28.000000000 +0000
@@ -104,7 +104,7 @@
####################################################################
[ req ]
default_bits = 2048
-default_md = sha1
+default_md = sha256
default_keyfile = privkey.pem
distinguished_name = req_distinguished_name
attributes = req_attributes

Linux malware detect aka maldet installation procedure

~ tadmin

I’m using the following procedure to install and configure Linux Malware Detect (LMD):

1. Download it:

# wget http://www.rfxn.com/downloads/maldetect-current.tar.gz

–2013-12-06 16:10:52– http://www.rfxn.com/downloads/maldetect-current.tar.gz
Resolving www.rfxn.com… 129.121.132.46
Connecting to www.rfxn.com|129.121.132.46|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 811434 (792K) [application/x-gzip]
Saving to: `maldetect-current.tar.gz’

100%[====================================================================================================>] 811,434 –.-K/s in 0.1s

2013-12-06 16:10:53 (7.51 MB/s) – `maldetect-current.tar.gz’ saved [811434/811434]

2. Verify downloaded package:

# ls -alh maldetect-current.tar.gz

-rw-r–r– 1 root root 793K Apr 13 2013 maldetect-current.tar.gz

# sha1sum -b maldetect-current.tar.gz
fe2b0629ca473997afc717d56dc8e2bc2ec868a8 *maldetect-current.tar.gz

#

3. Unpack archive:

# tar xzvf maldetect-current.tar.gz

maldetect-1.4.2/
maldetect-1.4.2/CHANGELOG
maldetect-1.4.2/cron.daily
maldetect-1.4.2/README
maldetect-1.4.2/files/
maldetect-1.4.2/files/ignore_file_ext
maldetect-1.4.2/files/sigs/
maldetect-1.4.2/files/sigs/rfxn.ndb
maldetect-1.4.2/files/sigs/md5.dat
maldetect-1.4.2/files/sigs/hex.dat
maldetect-1.4.2/files/sigs/maldet.sigs.ver
maldetect-1.4.2/files/sigs/rfxn.hdb
maldetect-1.4.2/files/tmp/
maldetect-1.4.2/files/VERSION.hash
maldetect-1.4.2/files/ignore_inotify
maldetect-1.4.2/files/hexstring.pl
maldetect-1.4.2/files/maldet
maldetect-1.4.2/files/conf.maldet
maldetect-1.4.2/files/clean/
maldetect-1.4.2/files/clean/gzbase64.inject.unclassed
maldetect-1.4.2/files/clean/base64.inject.unclassed
maldetect-1.4.2/files/sess/
maldetect-1.4.2/files/pub/
maldetect-1.4.2/files/internals.conf
maldetect-1.4.2/files/quarantine/
maldetect-1.4.2/files/inotify/
maldetect-1.4.2/files/inotify/tlog
maldetect-1.4.2/files/inotify/inotifywait
maldetect-1.4.2/files/inotify/libinotifytools.so.0
maldetect-1.4.2/files/hexfifo.pl
maldetect-1.4.2/files/ignore_paths
maldetect-1.4.2/files/ignore_sigs
maldetect-1.4.2/files/modsec.sh
maldetect-1.4.2/cron.d.pub
maldetect-1.4.2/COPYING.GPL
maldetect-1.4.2/.ca.def
maldetect-1.4.2/install.sh
#

4. Install it:

# cd maldetect-1.4.2/
# ./install.sh

Linux Malware Detect v1.4.1
(C) 2002-2013, R-fx Networks (C) 2013, Ryan MacDonald
inotifywait (C) 2007, Rohan McGovern
This program may be freely redistributed under the terms of the GNU GPL

installation completed to /usr/local/maldetect
config file: /usr/local/maldetect/conf.maldet
exec file: /usr/local/maldetect/maldet
exec link: /usr/local/sbin/maldet
exec link: /usr/local/sbin/lmd
cron.daily: /etc/cron.daily/maldet

maldet(12981): {sigup} performing signature update check…
maldet(12981): {sigup} local signature set is version 25
maldet(12981): {sigup} new signature set (260) available
maldet(12981): {sigup} downloaded http://www.rfxn.com/downloads/md5.dat
maldet(12981): {sigup} downloaded http://www.rfxn.com/downloads/hex.dat
maldet(12981): {sigup} downloaded http://www.rfxn.com/downloads/rfxn.ndb
maldet(12981): {sigup} downloaded http://www.rfxn.com/downloads/rfxn.hdb
maldet(12981): {sigup} downloaded http://www.rfxn.com/downloads/maldet-clean.tgz
maldet(12981): {sigup} signature set update completed
maldet(12981): {sigup} 11362 signatures (9490 MD5 / 1872 HEX)
#

5. Edit configuration file to send e-mail notifications to administrator:

# conf /usr/local/maldetect/conf.maldet

# The destination addresses for email alerts
# [ values are comma (,) spaced ]
email_addr=”taras@shkodenko.com”

# ls -alh /usr/local/maldetect/conf.malde*

-rw-r–r– 1 root root 4.3K Dec 6 16:12 /usr/local/maldetect/conf.maldet
-rw-r–r– 1 root root 4.3K Apr 9 2013 /usr/local/maldetect/conf.maldet-2013-12-06.bak

# sha1sum -b /usr/local/maldetect/conf.malde*

8f28ec69d1007120d4b96f99cb14eb219091134d */usr/local/maldetect/conf.maldet
586016e43d31454e683320c18fe79c7e9e3273ce */usr/local/maldetect/conf.maldet-2013-12-06.bak
#

6. View cron job shell script which was added to run malware check on daily basis:
# more /etc/cron.daily/maldet

#!/bin/bash

# clear quarantine/session/tmp data every 14 days
/usr/sbin/tmpwatch 336 /usr/local/maldetect/tmp >> /dev/null 2>&1
/usr/sbin/tmpwatch 336 /usr/local/maldetect/sess >> /dev/null 2>&1
/usr/sbin/tmpwatch 336 /usr/local/maldetect/quarantine >> /dev/null 2>&1
/usr/sbin/tmpwatch 336 /usr/local/maldetect/pub/*/ >> /dev/null 2>&1

# check for new release version
/usr/local/maldetect/maldet -d >> /dev/null 2>&1

# check for new definition set
/usr/local/maldetect/maldet -u >> /dev/null 2>&1

# if were running inotify monitoring, send daily hit summary
if [ "$(ps -A --user root -o "comm" | grep inotifywait)" ]; then
        /usr/local/maldetect/maldet --alert-daily >> /dev/null 2>&1
else
        # scan the last 2 days of file changes
        if [ -d "/home/virtual" ] && [ -d "/usr/lib/opcenter" ]; then
                # ensim
                /usr/local/maldetect/maldet -b -r /home/virtual/?/fst/var/www/html 2 >> /dev/null 2>&1
                /usr/local/maldetect/maldet -b -r /home/virtual/?/fst/home/?/public_html 2 >> /dev/null 2>&1
        elif [ -d "/etc/psa" ] && [ -d "/var/lib/psa" ]; then
                # psa
                /usr/local/maldetect/maldet -b -r /var/www/vhosts/?/httpdocs 2 >> /dev/null 2>&1
                /usr/local/maldetect/maldet -b -r /var/www/vhosts/?/subdomains/?/httpdocs 2 >> /dev/null 2>&1
        elif [ -d "/usr/local/directadmin" ]; then
                # DirectAdmin
                /usr/local/maldetect/maldet -b -r /var/www/html/?/ 2 >> /dev/null 2>&1
                /usr/local/maldetect/maldet -b -r /home?/?/domains/?/public_html 2 >> /dev/null 2>&1
        else
                # cpanel, interworx and other standard home/user/public_html setups
                /usr/local/maldetect/maldet -b -r /home?/?/public_html 2 >> /dev/null 2>&1
        fi

        # scan default apache docroot paths
        if [ -d "/var/www/html" ]; then
                /usr/local/maldetect/maldet -b -r /var/www/html 2 >> /dev/null 2>&1
        fi
        if [ -d "/usr/local/apache/htdocs" ]; then
                /usr/local/maldetect/maldet -b -r /usr/local/apache/htdocs 2 >> /dev/null 2>&1
        fi
fi

#

You can edit it to add your web server specific layout if it does not looks like: cPanel, DirectAdmin, Plesk or any of other shown above.

nginx: [warn] 4096 worker_connections exceed open file resource limit: 1024

~ tadmin

If you see error messages like:
nginx: [warn] 4096 worker_connections exceed open file resource limit: 1024

Then checking nginx web server configuration using command:

# service nginx configtest

It can be fixed permanently (after system reboot) by editing /etc/security/limits.conf configuration file:

#@student        -       maxlogins       4

* soft nofile 65536
* hard nofile 65536

# End of file

Default Linux system limits can be viewed using command:

# ulimit -a

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 14586
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 14586
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

To fix open files limit value temporary (it works until server reboot) use command:

# ulimit -n 65536

To view new values of system limits run once again:

# ulimit -a

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 14586
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 14586
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Then checking nginx web server configuration use command:

# service nginx configtest

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

And, finally, restart nginx web server using command:

# service nginx restart

Stopping nginx: [ OK ]
Starting nginx: [ OK ]
#

Errors running mysqldump related to LOCK TABLES

~ tadmin

If you see such error patterns when running mysqldump backup:

# mysqldump -u wordpress_u -p wordpress_db > ./tmp/db_backups/wordpress_db-`date +'%Y%m%d%H%M'`.sql

Enter password:
mysqldump: Got error: 1016: Can’t open file: ‘./wordpress_db/wp_1050_postmeta.frm’ (errno: 24) when using LOCK TABLES

# mysqldump -u wordpress_u -p wordpress_db > ./tmp/db_backups/wordpress_db-`date +'%Y%m%d%H%M'`.sql

Enter password:
mysqldump: Got error: 23: Out of resources when opening file ‘./wordpress_db/wp_1050_options.MYD’ (Errcode: 24) when using LOCK TABLES

try to add –lock-tables=false to dump command.

Work with davfs2 on Fedora CentOs RHEL

~ tadmin

This post can be useful if you need to work with remote file system using davfs2 protocol.

To start work with it you should install davfs2 support using command:

# sudo yum install davfs2 -y

To start work you have to create mount point using command:

# sudo mkdir /mnt/drive_davfs2

Mount drive using command:

# sudo mount -t davfs https://file-store.shkodenko.com/ /mnt/drive_davfs2

You will need to provide username and password to connect.

Now, you can work with remote DAVFS folder /mnt/drive_davfs2 as with local folder.

To view list of currently mounted filesystems use command:

# sudo mount -l

Then you don’t need remote folder you should unmount it using command:

# sudo umount /mnt/drive_davfs2

RHEL Downloadonly plugin for yum

~ tadmin

If you need to get package source code only, and you don’t want to install it, you can use Downloadonly plugin for yum.

Install the package including “downloadonly” plugin:

(RHEL5)

# yum install yum-downloadonly

(RHEL6)

# yum install yum-plugin-downloadonly

Run yum command with “–downloadonly” option as follows:

# yum install --downloadonly --downloaddir=<directory> <package>

Confirm the RPM files are available in the specified download directory.

Source: https://access.redhat.com/solutions/10154

Import SSL certificate from web server Apache OpenSSL to Tomcat format

~ tadmin

To import SSL certificate into pkcs12 format you can use command:

# openssl pkcs12 -export -in /home/taras/ssl/shkodenko.com.crt -inkey /home/taras/ssl/shkodenko.com.key -out /home/taras/ssl/shkodenko.com.p12 -name tomcat -CAfile /home/taras/ssl/shkodenko.com.ca.crt -caname root -chain
Enter Export Password:
Verifying - Enter Export Password:

You will be prompted to input password.

# keytool -importkeystore -srckeystore /home/taras/ssl/shkodenko.com.p12 -srcstoretype PKCS12 -deststoretype JKS -destkeystore /home/taras/ssl/shkodenko.com.jks
Enter destination keystore password:  
Re-enter new password: 
Enter source keystore password:  
Entry for alias tomcat successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

Now, you can use /home/taras/ssl/shkodenko.com.jks certificates storage in your Apache Tomcat SSL virtual host configuration.

Set up NFS on CentOS and RedHat enterprise remote server folder sharing

~ tadmin

To set up NFS on CentOS and/or RedHat enterprise Linux the following package should be installed:

# yum install nfs-utils
Loaded plugins: product-id, rhnplugin, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
Excluding Packages in global exclude list
Finished
Excluding Packages from Red Hat Enterprise Linux (v. 5 for 64-bit x86_64)
Finished
Setting up Install Process
Package 1:nfs-utils-1.0.9-71.el5.x86_64 already installed and latest version
Nothing to do

If you see error while trying to start nfs service:

# /etc/init.d/nfs start
Starting NFS services:                                     [  OK  ]
Starting NFS quotas: Cannot register service: RPC: Unable to receive; errno = Connection refused
rpc.rquotad: unable to register (RQUOTAPROG, RQUOTAVERS, udp).
                                                           [FAILED]
Starting NFS daemon:   
                                                           [FAILED]

check if portmapper service is working:

# rpcinfo -p
rpcinfo: can't contact portmapper: RPC: Remote system error - Connection refused
#

If not, add portmap service to auto start:

# chkconfig portmap on

Start the portmap serice:

# service portmap start
Starting portmap:                                          [  OK  ]

When portmap service is started it should show some output like:

# rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
#

You should start nfs service

# service nfs start
Starting NFS services:                                     [  OK  ]
Starting NFS quotas:                                       [  OK  ]
Starting NFS daemon:                                       [  OK  ]
Starting NFS mountd:                                       [  OK  ]
#

Add an entry what folder do you want to share to configuration file /etc/exports

/home/taras/backups/shared 11.22.3.4/255.255.255.0(rw,sync)

Where 11.22.3.4 IP address of remote side where you will mount shared folder.

You can also add IP to configuration file:

# more /etc/hosts.allow
#
# hosts.allow   This file describes the names of the hosts which are
#               allowed to use the local INET services, as decided
#               by the '/usr/sbin/tcpd' server.
#
ALL: 11.22.3.4

Create folder /mnt/shared and mount remote folder in it

# mkdir -pv /mnt/shared
# mount shkodenko.com:/home/taras/backups/shared /mnt/shared -o nolock