Latest Posts

WordPress Troubleshooting Tips and Tricks

I have been hosting my blog 51sec.org in multiple site, AWS, GCP, IBM Openshift , etc. During migration between those sites, there were multiple issues happened and I spent some time to resolve them.  This post is to record those typical issue bothered me and hope it can save you some time if you have same problem.

Also Wordpress plug-in "All-in-One WP Migration" helped a lot during this migration or upgrade process. Whole blog was exported to a 50M to120M package depending on plug-in numbers.

Here are list of those troubles:
1. PHP is out of memory.
2. WordPress Site and Home URL Change
3. Change Upload file limitation 2M
4. Wordpress PermaLinks Issue
5. Prompted for FTP details when update plugins
6. Plugin JetPack Error Updating Settings – FetchNetworkError
7. Disable WordPress Plugin from PhpMyAdmin
8. Switch Between Prod site and Dev Site
9. Featured Image From URL Issue - Left White Section on Page

1 - PHP Out of Memory


At second day after migration, I got my monitis alert email to say my site is down.

After rebooted server website came up right away. But it happened again after a couple of hours. I know it must be something wrong. I am starting to dig in the logs. Here is what I found from logs:

[[email protected] httpd]# tail /var/log/httpd/error_log -n 50

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory

[Thu Sep 28 09:29:37.890816 2017] [mpm_prefork:notice] [pid 2853] AH00169: caught SIGTERM, shutting down
[Thu Sep 28 09:29:38.196735 2017] [suexec:notice] [pid 3692] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Sep 28 09:29:38.217716 2017] [lbmethod_heartbeat:notice] [pid 3693] AH02282: No slotmem from mod_heartmonitor
[Thu Sep 28 09:29:38.312526 2017] [mpm_prefork:notice] [pid 3693] AH00163: Apache/2.4.27 (Amazon) PHP/7.0.21 configured -- resuming normal operations
[Thu Sep 28 09:29:38.312545 2017] [core:notice] [pid 3693] AH00094: Command line: '/usr/sbin/httpd'

mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
[Thu Sep 28 11:12:03.262604 2017] [:error] [pid 4278] [client 190.94.81.146:4505] PHP Fatal error:  Out of memory (allocated 25165824) (tried to allocate 65536 bytes) in /var/www/html/wp-content/themes/startup-blog/functions.php on line 132, referer: //www.51sec.org/2016/01/cisco-switch-2960-3560-password-recovery-procedures/
[Thu Sep 28 11:12:03.262669 2017] [:error] [pid 4280] [client 190.94.81.146:4504] PHP Fatal error:  Out of memory (allocated 25165824) (tried to allocate 65536 bytes) in /var/www/html/wp-content/themes/startup-blog/functions.php on line 132, referer: //www.51sec.org/2016/01/cisco-switch-2960-3560-password-recovery-procedures/
[Thu Sep 28 11:12:04.397148 2017] [:error] [pid 4277] [client 66.249.89.16:36770] PHP Fatal error:  Out of memory (allocated 20971520) (tried to allocate 32768 bytes) in /var/www/html/wp-content/plugins/jetpack/modules/sharedaddy/sharing.php on line 224

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory


PHP is out of memory. I got many suggestions from Internet. Here are some methods I took.


Solution one

My temporary solution one was to set up a schedule to restart httpd service every two hours using CRON.

[[email protected] /]# cat /etc/crontab 
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed
2 */2 * * * root sudo service httpd restart && curl -sm 30 k.wdt.io/[email protected]/reboot_httpd?c=0_*/2_*_*_*
[[email protected] /]# 


I used https://crontab.guru service to get proper command  and wdt.io to get alert email for cron jobs. It did works and helps.


[[email protected]10-10-0-50 /]# tail /var/log/cron 
Sep 28 13:00:01 ip-10-10-0-50 CROND[4990]: (root) CMD (sudo service httpd restart && curl -sm 30 k.wdt.io/[email protected]/reboot_httpd?c=0_*_*_*_*)
Sep 28 13:01:01 ip-10-10-0-50 CROND[5071]: (root) CMD (run-parts /etc/cron.hourly)
Sep 28 13:01:01 ip-10-10-0-50 run-parts(/etc/cron.hourly)[5071]: starting 0anacron
Sep 28 13:01:01 ip-10-10-0-50 run-parts(/etc/cron.hourly)[5080]: finished 0anacron
Sep 28 13:50:01 ip-10-10-0-50 crond[2864]: (*system*) RELOAD (/etc/crontab)
Sep 28 14:00:01 ip-10-10-0-50 CROND[5279]: (root) CMD (sudo service httpd restart && curl -sm 30 k.wdt.io/[email protected]/reboot_httpd?c=0_*/2_*_*_*)
Sep 28 14:01:01 ip-10-10-0-50 CROND[5344]: (root) CMD (run-parts /etc/cron.hourly)
Sep 28 14:01:01 ip-10-10-0-50 run-parts(/etc/cron.hourly)[5344]: starting 0anacron
Sep 28 14:01:01 ip-10-10-0-50 run-parts(/etc/cron.hourly)[5353]: finished 0anacron
Sep 28 14:19:01 ip-10-10-0-50 crond[2864]: (*system*) RELOAD (/etc/crontab)
[[email protected] /]# 


From the httpd/error_log, I can see the service was shut down and resumed at scheduled time.

[[email protected] /]# tail /var/log/httpd/error_log -n 20
[[email protected] /]# tail /var/log/httpd/error_log -n 20
[Thu Sep 28 12:00:01.375912 2017] [suexec:notice] [pid 4673] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Sep 28 12:00:01.402479 2017] [lbmethod_heartbeat:notice] [pid 4674] AH02282: No slotmem from mod_heartmonitor
[Thu Sep 28 12:00:01.420712 2017] [mpm_prefork:notice] [pid 4674] AH00163: Apache/2.4.27 (Amazon) PHP/7.0.21 configured -- resuming normal operations
[Thu Sep 28 12:00:01.420736 2017] [core:notice] [pid 4674] AH00094: Command line: '/usr/sbin/httpd'
[Thu Sep 28 12:39:01.020101 2017] [:error] [pid 4889] [client 45.43.101.69:59405] PHP Warning:  mysqli_query(): MySQL server has gone away in /var/www/html/wp-includes/wp-db.php on line 1887, referer: //www.51sec.org/2015/10/advanced-checkpoint-gaia-cli-commands-tips-and-tricks/
[Thu Sep 28 12:39:01.028056 2017] [:error] [pid 4889] [client 45.43.101.69:59405] PHP Warning:  mysqli_query(): Error reading result set's header in /var/www/html/wp-includes/wp-db.php on line 1887, referer: //www.51sec.org/2015/10/advanced-checkpoint-gaia-cli-commands-tips-and-tricks/
[Thu Sep 28 12:39:01.044760 2017] [:error] [pid 4756] [client 45.43.101.69:58628] PHP Warning:  Error while sending QUERY packet. PID=4756 in /var/www/html/wp-includes/wp-db.php on line 1887, referer: //www.51sec.org/2015/10/advanced-checkpoint-gaia-cli-commands-tips-and-tricks/
[Thu Sep 28 13:00:02.531359 2017] [mpm_prefork:notice] [pid 4674] AH00169: caught SIGTERM, shutting down
[Thu Sep 28 13:00:03.123712 2017] [suexec:notice] [pid 5019] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Sep 28 13:00:03.145836 2017] [lbmethod_heartbeat:notice] [pid 5020] AH02282: No slotmem from mod_heartmonitor
[Thu Sep 28 13:00:03.198017 2017] [mpm_prefork:notice] [pid 5020] AH00163: Apache/2.4.27 (Amazon) PHP/7.0.21 configured -- resuming normal operations
[Thu Sep 28 13:00:03.198034 2017] [core:notice] [pid 5020] AH00094: Command line: '/usr/sbin/httpd'
[Thu Sep 28 14:00:01.734292 2017] [mpm_prefork:notice] [pid 5020] AH00169: caught SIGTERM, shutting down
[Thu Sep 28 14:00:02.141090 2017] [suexec:notice] [pid 5307] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Sep 28 14:00:02.169154 2017] [lbmethod_heartbeat:notice] [pid 5308] AH02282: No slotmem from mod_heartmonitor
[Thu Sep 28 14:00:02.236204 2017] [mpm_prefork:notice] [pid 5308] AH00163: Apache/2.4.27 (Amazon) PHP/7.0.21 configured -- resuming normal operations
[Thu Sep 28 14:00:02.236230 2017] [core:notice] [pid 5308] AH00094: Command line: '/usr/sbin/httpd'


Solution Two


There are some posts mentioning some kind of attack related to XML-RPC, which can be exploited to send thousands of requests to WordPress in a short amount of time.

To recognize this attack, I did some search from the httpd access log:

[[email protected] ec2-user]# grep xmlrpc /var/log/httpd/access_log 
180.191.61.7 - - [27/Sep/2017:02:24:09 +0000] "POST /xmlrpc.php HTTP/1.1" 200 403 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
97.73.96.19 - - [27/Sep/2017:02:43:52 +0000] "POST /xmlrpc.php HTTP/1.1" 200 403 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
112.200.195.163 - - [27/Sep/2017:02:46:41 +0000] "POST /xmlrpc.php HTTP/1.1" 200 403 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
115.79.220.211 - - [27/Sep/2017:02:54:56 +0000] "POST /xmlrpc.php HTTP/1.1" 200 403 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
160.238.72.10 - - [27/Sep/2017:03:00:06 +0000] "POST /xmlrpc.php HTTP/1.1" 200 403 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
39.46.183.191 - - [27/Sep/2017:03:06:05 +0000] "POST /xmlrpc.php HTTP/1.1" 200 403 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
2.24.37.37 - - [27/Sep/2017:03:08:31 +0000] "POST /xmlrpc.php HTTP/1.1" 200 403 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
115.178.26.232 - - [27/Sep/2017:03:15:42 +0000] "POST /xmlrpc.php HTTP/1.1" 200 403 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
112.210.202.81 - - [27/Sep/2017:03:16:09 +0000] "POST /xmlrpc.php HTTP/1.1" 200 403 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
98.124.36.154 - - [27/Sep/2017:03:19:52 +0000] "POST /xmlrpc.php HTTP/1.1" 200 403 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
..........


There are thousands of similar logs like that.

Based on the post suggestion, install jetpack plug in helps to reduce this kind of attack.

The Protect function is automatically enabled. You can now see a Jetpack dashboard which also displays the Protect function as being Active. After running a couple of hours, we can see there were 47 malicious attacks blocked by Jetpack plugin.



Solution Three

After did a google online and found there are many posts to suggest change some settings on php.ini file.

Here is what I did :


[[email protected] html]# echo "<?php phpinfo(); ?>" > /var/www/html/phpinfo.php

by access //www.51sec.org/phpinfo.php I got the php.ini's location /etc/php.ini



[[email protected] etc]# vi php.ini
; Maximum amount of memory a script may consume (128MB is Default)
; http://php.net/memory-limit
memory_limit = 256M


I also tried to increase my PHP memory limit with wp-config.php file. This file is located in /var/www/html/ directory.


[[email protected] html]# vi wp-config.php
define('WP_MEMORY_LIMIT', '256M');

Add it to the last line of the file wp-config.php.


2 - WordPress URL Change

During testing, you might use public ip such as http://54.32.104.11 as your WordPress Address and Site Address. Sometimes, your website might be wrong after you changed to your domain name and you will need to change it back but your Web Admin is not reachable.

Here is a way to hard coded WP Home and WP SiteURL in wp-config.php file.

[[email protected] html]# sudo nano /var/www/html/wordpress/wp-config.php
define('WP_HOME','hxxp://www.51sec.org');
define('WP_SITEURL','hxxp://www.51sec.org');







3 - Change Upload file limitation 2M

You will need to find out right folder to put in this .user.ini file, usually it is /var/www/html and your wordpress folder is /var/www/html/wordpress. You will need to put it under /var/www/html folder, not wordpress folder.

Go to your site’s root directory and open or create a .user.ini file using vi or nano. You can then just paste in the following code and save the change:
upload_max_filesize = 64M
post_max_size = 13M
memory_limit = 64M
Note:By default, new installation only allows 2MB uploading. This change would not require reboot any services ,such as php, nginix.


4: Wordpress PermaLinks Issue

I met this issue when installed a new Ubuntu and wordpress. Here is the solution I found from this post:

There can be multiple things preventing the rewrite rule from working. My ubuntu server needed 3 things changed to get permalinks working.
In newer versions of apache2, you need to enable the module:
sudo a2enmod rewrite
sudo service apache2 restart
You may also need to modify the apache2.conf file.
sudo nano /etc/apache2/apache2.conf
Change your web directory override rule to AllowOverride All.
<Directory /var/www/>
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
</Directory>
After that, restart the service again.
The .htaccess file in the WordPress install directory needs to be owned or readable/modifiable by the webserver.
This can be accomplished by changing owner to www-data (usually the webserver user), or changing permissions.
sudo chown www-data /var/www/wordpress-install/.htaccess 
OR
sudo chmod 664 /var/www/wordpress-install/.htaccess
Login to your Wordpress admin backend and save the permalink settings, and they should hopefully be working.

5. Prompted for FTP details when update plugins



Usually it is relating to file permission issue. But some times, you used right user www-data or nginx with right permission chmod 755 , you still got that ftp screen prompt up to ask you to enter ftp user name and password.

1. Using apache
sudo chown -R www-data:www-data /var/www/html/
sudo chmod -R 755 /var/www/html/

2. Using Nginx
sudo chown -R nginx:nginx /var/www/html/
sudo chmod -R 755 /var/www/html/


You will need to set define('FS_METHOD', 'direct'); in wp-config.php.
Actualluy it will looks like this in wp-config.php file:'
define( 'DB_COLLATE', '' );
define('FS_METHOD', 'direct');

6. Plugin JetPack Error Updating Settings – FetchNetworkError

One of my sites met this issue for a couple of months already. I have tried to uninstall / re-install this JetPack plug-in many times, but could not fix it until saw this article.

For me, the solution 2 - Install Really Simple SSL works. After installed and activate plug-in - Really Simple SSL, you will have to activate SSL for your site in the plug-in. That is all you will need to do to get JetPack working again. It seems issue is relating to SSL communication.


7. Disable Plugins from PhpMyAdmin

There are some scenarios you might lost access your wp-admin page from Web GUI because of some plug-ins you enabled. In that case, if you still have access phpMyAdmin, you can use following steps to disable all plug-ins then bring your website back.
Step 1 - Access your database in phpMyAdmin
Step 2 - Click wp_options table
Step 3 - Locate  active_plugins from option_value by clicking it to sort with alphabet order. 
Step 4 - Edit active_plugins. Remove all content and press enter to save.

8. Steps to Switch between Prod / Dev WordPress Sites

I am having one Prod site with www.51sec.org domain and also using some other sub-domains as my developing sites to test all kinds of technologies such as plug-ins, cloud VPS, or redundancy / load balance in Nginx, etc.

Here are some steps I will need to write down.
1. Make sure www.51sec.org and sub-domain sites (dev1.51sec.org,dev2.51sec.org), points to the specifc nginx site. It is DNS configuration at ezoic.com site. Or, make sure they are pointing to right site's public ip.
2. Make sure my nginx site configuration has right configuration for each site. Point to right server ip:port.
3. Make sure /etc/nginx/conf.d or /etc/nginx/sites-enabled configuration file has right ip and domain name (gcp1.51sec.org, www.51sec.org, 51sec.org) in it.
4. Make sure prod and dev site has right wp-config.php configuration for each site, define home url and site url.

9. Featured Image From URL Issue - Left White Section on Page


Make sure showing external image from this plug in.


10. Manualy Install Latest Wordpress to Resolve a Wordpress Update Error




It is relating to file permission issue. Rather than to troubleshoot this issue, I just manual unzip whole downloaded package to overwrite existing installation.

cd /var/www/html
wget https://wordpress.org/latest.zip
unzip wordpress-x-version.zip

This will complete overwrite any files at your wordpress installation at cd /var/www/html/wordpress
unzip wordpress-x-version.zip will unzip everthing under wordpress folder.



References:



No comments