Thursday, September 5, 2019

Monit - Linux Open Source System or Process Monitor Program

1. What is Monit?
Monit is a utility for managing and monitoring processes, programs, files, directories and file systems on a Unix like system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. E.g. Monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses too much resources. You can use Monit to monitor files, directories and file systems for changes, such as timestamps changes, checksum changes or size changes.

Monit is controlled via an easy to configure control file (/etc/monit/monitrc) based on a free-format, token-oriented syntax. Monit logs  to syslog or to its own log file (/var/log/monit.log) and notifies you about error conditions via customisable alert messages.



2. Install


[email protected]:~$ sudo apt install monit
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following package was automatically installed and is no longer required:
  grub-pc-bin
Use 'sudo apt autoremove' to remove it.
Suggested packages:
  exim4 | postfix | mail-transport-agent sysvinit-core
The following NEW packages will be installed:
  monit
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 318 kB of archives.
After this operation, 946 kB of additional disk space will be used.
Get:1 http://us-central1.gce.archive.ubuntu.com/ubuntu bionic/universe amd64 monit amd64 1:5.25.1-1build1 [318 kB]
Fetched 318 kB in 0s (9343 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package monit.
(Reading database ... 45688 files and directories currently installed.)
Preparing to unpack .../monit_1%3a5.25.1-1build1_amd64.deb ...
Unpacking monit (1:5.25.1-1build1) ...
Setting up monit (1:5.25.1-1build1) ...
Processing triggers for systemd (237-3ubuntu10.28) ...
[email protected]:~$ 



3. Service Control and Show Service Status


[email protected]:~$ sudo monit -t
Control file syntax OK
[email protected]:~$ sudo service monit restart
[email protected]:~$ sudo service monit status
● monit.service - LSB: service and resource monitoring daemon
   Loaded: loaded (/etc/init.d/monit; generated)
   Active: active (running) since Thu 2019-09-05 18:20:09 UTC; 26s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 27892 ExecStop=/etc/init.d/monit stop (code=exited, status=0/SUCCESS)
  Process: 27896 ExecStart=/etc/init.d/monit start (code=exited, status=0/SUCCESS)
    Tasks: 1 (limit: 690)
   CGroup: /system.slice/monit.service
           └─27911 /usr/bin/monit -c /etc/monit/monitrc

Sep 05 18:20:09 ubuntu-min-nginx systemd[1]: Stopped LSB: service and resource monitoring daemon.
Sep 05 18:20:09 ubuntu-min-nginx systemd[1]: Starting LSB: service and resource monitoring daemon...
Sep 05 18:20:09 ubuntu-min-nginx monit[27896]:  * Starting daemon monitor monit
Sep 05 18:20:09 ubuntu-min-nginx monit[27896]:    ...done.
Sep 05 18:20:09 ubuntu-min-nginx systemd[1]: Started LSB: service and resource monitoring daemon.
[email protected]:~$ 



4. Monitor System CPU, Memory, Load

Configure monit to monitor system load, cpu usage and memory usage. Once gauge exceeded, monit will trigger a reboot using command '/sbin/shutdown -r now'.


[email protected]:~$ sudo vi /etc/monit/monitrc
check system $HOST
        if loadavg (1min) > 1 then alert
        if loadavg (5min) > 1.05 then exec "/sbin/shutdown -r now"
        if cpu usage > 50% for 5 cycles then alert
        if cpu usage > 75% for 3 cycles then exec "/sbin/shutdown -r now"
        if memory usage > 85% then alert
#       if memory usage > 95% then reboot
        if swap usage > 25% then alert



Here are some logs when system resource threshold reached , and monit service was triggered:


[email protected]:~$ cat /var/log/monit.log
[EDT Sep  4 15:46:17] info     : Starting Monit 5.25.1 daemon
[EDT Sep  4 15:46:17] info     : 'ubuntu18niginxmariadb2' Monit 5.25.1 started
[EDT Sep  4 20:02:19] warning  : 'ubuntu18niginxmariadb2' cpu usage of 98.7% matches resource limit [cpu usage > 75.0%]
[EDT Sep  4 20:02:19] warning  : 'ubuntu18niginxmariadb2' cpu usage of 98.7% matches resource limit [cpu usage > 50.0%]
[EDT Sep  4 20:02:19] error    : 'ubuntu18niginxmariadb2' loadavg(5min) of 1.6 matches resource limit [loadavg(5min) > 1.0]
[EDT Sep  4 20:02:19] info     : 'ubuntu18niginxmariadb2' exec: '/sbin/shutdown -r now'
[EDT Sep  4 20:02:19] error    : 'ubuntu18niginxmariadb2' loadavg(1min) of 4.1 matches resource limit [loadavg(1min) > 1.0]
[EDT Sep  4 20:02:24] info     : Monit daemon with pid [1121] stopped
[EDT Sep  4 20:02:24] info     : 'ubuntu18niginxmariadb2' Monit 5.25.1 stopped
[EDT Sep  4 20:02:48] info     : Starting Monit 5.25.1 daemon
[EDT Sep  4 20:02:48] info     : 'ubuntu18niginxmariadb2' Monit 5.25.1 started
[EDT Sep  5 01:32:50] error    : 'ubuntu18niginxmariadb2' loadavg(1min) of 1.0 matches resource limit [loadavg(1min) > 1.0]
[EDT Sep  5 01:34:51] warning  : 'ubuntu18niginxmariadb2' cpu usage of 87.7% matches resource limit [cpu usage > 75.0%]
[EDT Sep  5 01:34:51] warning  : 'ubuntu18niginxmariadb2' cpu usage of 87.7% matches resource limit [cpu usage > 50.0%]
[EDT Sep  5 01:34:51] error    : 'ubuntu18niginxmariadb2' loadavg(5min) of 1.8 matches resource limit [loadavg(5min) > 1.0]
[EDT Sep  5 01:34:51] info     : 'ubuntu18niginxmariadb2' exec: '/sbin/shutdown -r now'
[EDT Sep  5 01:34:51] error    : 'ubuntu18niginxmariadb2' loadavg(1min) of 4.0 matches resource limit [loadavg(1min) > 1.0]
[EDT Sep  5 01:34:55] info     : Monit daemon with pid [977] stopped
[EDT Sep  5 01:34:55] info     : 'ubuntu18niginxmariadb2' Monit 5.25.1 stopped
[EDT Sep  5 01:35:18] info     : Starting Monit 5.25.1 daemon
[EDT Sep  5 01:35:18] info     : 'ubuntu18niginxmariadb2' Monit 5.25.1 started
[EDT Sep  5 02:23:19] error    : 'ubuntu18niginxmariadb2' loadavg(1min) of 1.3 matches resource limit [loadavg(1min) > 1.0]
[EDT Sep  5 02:25:19] warning  : 'ubuntu18niginxmariadb2' cpu usage of 50.7% matches resource limit [cpu usage > 50.0%]
[EDT Sep  5 02:25:19] error    : 'ubuntu18niginxmariadb2' loadavg(1min) of 2.0 matches resource limit [loadavg(1min) > 1.0]
[EDT Sep  5 02:27:19] warning  : 'ubuntu18niginxmariadb2' cpu usage of 99.5% matches resource limit [cpu usage > 75.0%]
[EDT Sep  5 02:27:19] warning  : 'ubuntu18niginxmariadb2' cpu usage of 99.5% matches resource limit [cpu usage > 50.0%]
[EDT Sep  5 02:27:19] error    : 'ubuntu18niginxmariadb2' loadavg(5min) of 2.1 matches resource limit [loadavg(5min) > 1.0]
[EDT Sep  5 02:27:19] info     : 'ubuntu18niginxmariadb2' exec: '/sbin/shutdown -r now'
[EDT Sep  5 02:27:19] error    : 'ubuntu18niginxmariadb2' loadavg(1min) of 4.3 matches resource limit [loadavg(1min) > 1.0]
[EDT Sep  5 02:27:22] info     : Monit daemon with pid [1003] stopped
[EDT Sep  5 02:27:22] info     : 'ubuntu18niginxmariadb2' Monit 5.25.1 stopped
[EDT Sep  5 02:27:47] info     : Starting Monit 5.25.1 daemon
[EDT Sep  5 02:27:47] info     : 'ubuntu18niginxmariadb2' Monit 5.25.1 started
[EDT Sep  5 02:41:47] error    : 'ubuntu18niginxmariadb2' loadavg(1min) of 2.1 matches resource limit [loadavg(1min) > 1.0]
[EDT Sep  5 02:43:48] warning  : 'ubuntu18niginxmariadb2' cpu usage of 99.5% matches resource limit [cpu usage > 75.0%]
[EDT Sep  5 02:43:48] warning  : 'ubuntu18niginxmariadb2' cpu usage of 99.5% matches resource limit [cpu usage > 50.0%]
[EDT Sep  5 02:43:48] error    : 'ubuntu18niginxmariadb2' loadavg(5min) of 1.9 matches resource limit [loadavg(5min) > 1.0]
[EDT Sep  5 02:43:48] info     : 'ubuntu18niginxmariadb2' exec: '/sbin/shutdown -r now'
[EDT Sep  5 02:43:48] error    : 'ubuntu18niginxmariadb2' loadavg(1min) of 4.1 matches resource limit [loadavg(1min) > 1.0]
[EDT Sep  5 02:43:52] info     : Monit daemon with pid [1041] stopped
[EDT Sep  5 02:43:52] info     : 'ubuntu18niginxmariadb2' Monit 5.25.1 stopped
[EDT Sep  5 02:44:16] info     : Starting Monit 5.25.1 daemon
[EDT Sep  5 02:44:16] info     : 'ubuntu18niginxmariadb2' Monit 5.25.1 started



5. Enable Monit Web Service



[email protected]:~$ sudo nano /etc/monit/monitrc



[email protected]:~$ sudo service monit restart
[email protected]:~$ netstat -na
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:2812            0.0.0.0:*               LISTEN     
tcp        0      0 10.128.0.3:59048        169.254.169.254:80      ESTABLISHED
tcp        0      0 10.128.0.3:80           52.53.175.20:61496      TIME_WAIT  
tcp        0      0 10.128.0.3:59058        169.254.169.254:80      ESTABLISHED
tcp        0      0 10.128.0.3:80           3.83.192.114:46676      TIME_WAIT  
tcp        0      0 10.128.0.3:80           54.193.33.178:52530     TIME_WAIT  
tcp        0      0 10.128.0.3:80           3.90.252.110:24496      TIME_WAIT  
tcp        0      0 10.128.0.3:59044        169.254.169.254:80      CLOSE_WAIT 
tcp        0    384 10.128.0.3:22           173.194.90.33:40188     ESTABLISHED
tcp        0      0 10.128.0.3:59052        169.254.169.254:80      ESTABLISHED
tcp6       0      0 :::22                   :::*                    LISTEN     
tcp6       0      0 :::2812                 :::*                    LISTEN     
udp        0      0 127.0.0.1:323           0.0.0.0:*                          
udp        0      0 127.0.0.53:53           0.0.0.0:*                          
udp        0      0 10.128.0.3:68           0.0.0.0:*                          
udp6       0      0 ::1:323                 :::*                               
raw6       0      0 :::58                   :::*                    

Browse to your Monit server's http://yourserverip:2821 page














No comments:

Post a Comment