Linux Virtual Machine PHP Invoked OOM-killer (Out of Memory) Issue

One of my GCP VM is running at f1-Micro type which only has limited memory, 614MB. From syslog, I found it has a PHP invoked OOM killer messages. This usually means this server essentially ran out of memory and extra memory should add into this system, Based on some posts online, especially this 2010's  post :

"The Linux kernel likes to always allocate memory if applications asking for it. Per default it doesn’t really check if there is enough memory available. Given that behavior applications can allocate more memory as really is available. At some point it can definitely cause an out of memory situation. As result the OOM killer will be invoked and will kill that process:"

Determining Free Memory and Swap Activity

Before start optimizing your server, let’s review memory using on it. You can use the following command to display memory:

$ free -m

To see a list of your running processes sorted by memory use:

$ ps -eo pmem,pcpu,rss,vsize,args | sort -k 1 -r | less

Checking Logs Error

more /var/log/syslog

Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315747] php-fpm7.2 invoked oom-killer: gfp_mask=0x15080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null), order=0, oom_score_adj=0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315750] php-fpm7.2 cpuset=/ mems_allowed=0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315758] CPU: 0 PID: 961 Comm: php-fpm7.2 Not tainted 4.15.0-1040-gcp #42-Ubuntu
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315759] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315760] Call Trace:
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315774]  dump_stack+0x8e/0xcb
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315780]  dump_header+0x71/0x285
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315782]  oom_kill_process+0x220/0x440
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315784]  out_of_memory+0x2d1/0x4f0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315786]  __alloc_pages_slowpath+0xa53/0xe00
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315793]  ? memcg_kmem_charge_memcg+0x7d/0xb0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315795]  __alloc_pages_nodemask+0x29a/0x2c0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315800]  alloc_pages_current+0x6a/0xe0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315805]  pte_alloc_one+0x17/0x40
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315810]  __pte_alloc+0x1e/0x120
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315811]  copy_pte_range+0x535/0x7a0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315814]  copy_page_range+0x3ed/0x6a0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315819]  copy_process.part.35+0xe1e/0x1ba0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315822]  _do_fork+0xdf/0x400
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315826]  ? __do_page_fault+0x2aa/0x4c0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315828]  SyS_clone+0x19/0x20
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315834]  do_syscall_64+0x7b/0x150
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315840]  entry_SYSCALL_64_after_hwframe+0x42/0xb7

Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315844] RIP: 0033:0x7efcde571b1c
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315845] RSP: 002b:00007ffc4e3848c0 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315847] RAX: ffffffffffffffda RBX: 00007ffc4e3848c0 RCX: 00007efcde571b1c
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315847] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315848] RBP: 00007ffc4e384970 R08: 00007efce0ceac00 R09: 0000000000000000
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315849] R10: 00007efce0ceaed0 R11: 0000000000000246 R12: 0000000000000000
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315850] R13: 0000000000000020 R14: 0000000000000000 R15: ffffffffffffffff
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315852] Mem-Info:
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315856] active_anon:105955 inactive_anon:8694 isolated_anon:0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315856]  active_file:69 inactive_file:144 isolated_file:0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315856]  unevictable:0 dirty:0 writeback:0 unstable:0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315856]  slab_reclaimable:4470 slab_unreclaimable:7737
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315856]  mapped:8482 shmem:8871 pagetables:2067 bounce:0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315856]  free:7589 free_pcp:94 free_cma:0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315860] Node 0 active_anon:423820kB inactive_anon:34776kB active_file:276kB inactive_file:576kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:33928kB dirty:0kB writeback:0kB shmem:35484kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315860] Node 0 DMA free:2948kB min:788kB low:984kB high:1180kB active_anon:8952kB inactive_anon:1892kB active_file:4kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:32kB pagetables:156kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315865] lowmem_reserve[]: 0 540 540 540 540
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315867] Node 0 DMA32 free:27408kB min:27460kB low:34324kB high:41188kB active_anon:414868kB inactive_anon:32884kB active_file:276kB inactive_file:576kB unevictable:0kB writepending:0kB present:612340kB managed:579656kB mlocked:0kB kernel_stack:2492kB pagetables:8112kB bounce:0kB free_pcp:376kB local_pcp:376kB free_cma:0kB
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315870] lowmem_reserve[]: 0 0 0 0 0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315873] Node 0 DMA: 27*4kB (UME) 21*8kB (UME) 39*16kB (UME) 20*32kB (UME) 12*64kB (UME) 5*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2948kB
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315882] Node 0 DMA32: 278*4kB (UMEH) 421*8kB (UMEH) 307*16kB (UMEH) 147*32kB (UMEH) 40*64kB (UME) 16*128kB (UME) 2*256kB (U) 2*512kB (UE) 1*1024kB (M) 1*2048kB (M) 1*4096kB (U) = 27408kB
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315894] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315896] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315896] 9092 total pagecache pages
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315898] 0 pages in swap cache
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315899] Swap cache stats: add 0, delete 0, find 0/0
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315900] Free swap  = 0kB
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315900] Total swap = 0kB
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315901] 157083 pages RAM
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315901] 0 pages HighMem/MovableOnly
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315902] 8192 pages reserved
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315902] 0 pages cma reserved
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315903] 0 pages hwpoisoned
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315903] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315908] [  396]     0   396    27480      230   200704        0             0 systemd-journal
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315909] [  400]     0   400    24427       44    90112        0             0 lvmetad
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315911] [  412]     0   412    10926      397   118784        0         -1000 systemd-udevd
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315913] [  726]   100   726    20011      182   176128        0             0 systemd-network
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315914] [  746]   101   746    17688      175   167936        0             0 systemd-resolve
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315916] [  870]   103   870    12525      182   135168        0          -900 dbus-daemon
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315918] [  885]     0   885   157192      186   147456        0             0 lxcfs
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315919] [  918]     0   918    17663      200   176128        0             0 systemd-logind
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315921] [  919]     0   919    72021      234   192512        0             0 accounts-daemon
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315922] [  920]     0   920   163116     3151   233472        0          -900 snapd
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315924] [  924]     0   924     7083       51   106496        0             0 atd
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315926] [  925]   102   925    66817      322   167936        0             0 rsyslogd
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315927] [  949]   111   949    25333       85    98304        0             0 chronyd
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315929] [  961]     0   961   115363     1861   602112        0             0 php-fpm7.2
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315930] [  972]     0   972    42721     1955   237568        0             0 networkd-dispat
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315932] [  977]     0   977     7937       75    98304        0             0 cron
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315933] [  987]     0   987     1157       16    57344        0             0 sshguard-journa
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315935] [  990]     0   990    28251      140   233472        0             0 journalctl
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315936] [  991]     0   991     4091       99    65536        0             0 sshguard
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315937] [ 1003]     0  1003     1157       16    57344        0             0 sshg-fw
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315939] [ 1006]     0  1006    35435      386   233472        0             0 nginx
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315940] [ 1007]    33  1007    36181      680   241664        0             0 nginx
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315942] [ 1137]    33  1137   153381    27842   835584        0             0 php-fpm7.2
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315943] [ 1169]     0  1169    72220      219   212992        0             0 polkitd
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315945] [ 1299]     0  1299    46931     1985   270336        0             0 unattended-upgr
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315946] [ 1362]     0  1362     4103       37    73728        0             0 agetty
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315947] [ 1405]   112  1405   372876    45749   753664        0             0 mysqld
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315949] [ 1455]     0  1455     3722       34    73728        0             0 agetty
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315950] [ 1456]     0  1456    18852     3644   196608        0             0 google_network_
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315952] [ 1469]     0  1469    18916     3687   192512        0          -999 google_accounts
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315953] [ 1472]     0  1472    18866     3606   192512        0             0 google_clock_sk
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315955] [ 1482]     0  1482    18074      189   180224        0         -1000 sshd
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315956] [ 2375]    33  2375   146970    21314   782336        0             0 php-fpm7.2

Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315957] [ 9304]     0  9304    10048      157   110592        0             0 monit
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315959] [20018]    33 20018   138997    13458   720896        0             0 php-fpm7.2
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315960] [22124]     0 22124   115363     1432   557056        0             0 php-fpm7.2
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315961] Out of memory: Kill process 1405 (mysqld) score 308 or sacrifice child
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.324223] Killed process 1405 (mysqld) total-vm:1491504kB, anon-rss:182996kB, file-rss:0kB, shmem-rss:0kB
Sep  1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.389736] oom_reaper: reaped process 1405 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep  1 10:46:25 ubuntu18niginxmariadb2 systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
Sep  1 10:46:25 ubuntu18niginxmariadb2 systemd[1]: mariadb.service: Failed with result 'signal'.
Sep  1 10:46:28 ubuntu18niginxmariadb2 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
Sep  1 10:46:28 ubuntu18niginxmariadb2 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
Sep  1 10:46:28 ubuntu18niginxmariadb2 systemd[1]: Stopped MariaDB 10.4.7 database server.
Sep  1 10:46:28 ubuntu18niginxmariadb2 systemd[1]: Starting MariaDB 10.4.7 database server...
Sep  1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] /usr/sbin/mysqld (mysqld 10.4.7-MariaDB-1:10.4.7+maria~bionic-log) starting as process 22301 ...
Sep  1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Using Linux native AIO
Sep  1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Sep  1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Uses event mutexes
Sep  1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Sep  1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Number of pools: 1
Sep  1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Using SSE2 crc32 instructions
Sep  1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
Sep  1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
Sep  1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Completed initialization of buffer pool
Sep  1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
Sep  1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=589758381
Sep  1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000014', position 22151790
Sep  1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
Sep  1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
Sep  1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Sep  1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Sep  1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Sep  1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: Waiting for purge to start
Sep  1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: 10.4.7 started; log sequence number 589758390; transaction id 501113
Sep  1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] Plugin 'FEEDBACK' is disabled.
Sep  1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Sep  1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
Sep  1 10:46:31 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:31 0 [Note] Starting crash recovery...
Sep  1 10:46:31 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:31 0 [Note] Crash recovery finished.
Sep  1 10:46:31 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:31 0 [Note] Server socket created on IP: '127.0.0.1'.
Sep  1 10:46:31 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:31 0 [Note] Reading of all Master_info entries succeeded
Sep  1 10:46:31 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:31 0 [Note] Added new Master_info '' to hash table
Sep  1 10:46:31 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:31 0 [Note] /usr/sbin/mysqld: ready for connections.
Sep  1 10:46:31 ubuntu18niginxmariadb2 mysqld[22301]: Version: '10.4.7-MariaDB-1:10.4.7+maria~bionic-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
Sep  1 10:46:31 ubuntu18niginxmariadb2 /etc/mysql/debian-start[22342]: Upgrading MySQL tables if necessary.
Sep  1 10:46:31 ubuntu18niginxmariadb2 systemd[1]: Started MariaDB 10.4.7 database server.
Sep  1 10:46:32 ubuntu18niginxmariadb2 /etc/mysql/debian-start[22345]: Looking for 'mysql' as: /usr/bin/mysql
Sep  1 10:46:32 ubuntu18niginxmariadb2 /etc/mysql/debian-start[22345]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Sep  1 10:46:32 ubuntu18niginxmariadb2 /etc/mysql/debian-start[22345]: This installation of MariaDB is already upgraded to 10.4.7-MariaDB, use --force if you still need to run mysql_upgrade
Sep  1 10:46:32 ubuntu18niginxmariadb2 /etc/mysql/debian-start[22354]: Checking for insecure root accounts.
Sep  1 10:46:32 ubuntu18niginxmariadb2 /etc/mysql/debian-start[22358]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
Sep  1 10:46:32 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:32 0 [Note] InnoDB: Buffer pool(s) load completed at 190901 10:46:32

Bascically, php-fpm7.2 invoked OOM killer to kill mysql process on this VM.  MariaDB was able to be re-launched successfully this time. But many times, whole system become unaccessed even ssh session.

GCP Monitoring shows CPU, Network Traffic and Disk IO increased during OOM killer killing mysql process.

There are a couple of ways to do it. One dirty way I tried is to use monit monitoring system resources or those critical processes and reboot machine. Another good way is to follow the post's recommendation, to change the kernel behavior which stop overcommitting the memory for application requests such as php-fpm7.2. There are two values need to be included into the /etc/sysctl.conf , which will be automatically applied on start-up:

vm.overcommit_memory = 2
vm.overcommit_ratio = 90

This overcommit_memory switch knows 3 different settings:

• 0: The Linux kernel is free to overcommit memory (this is the default), a heuristic algorithm is applied to figure out if enough memory is available.
• 1: The Linux kernel will always overcommit memory, and never check if enough memory is available. This increases the risk of out-of-memory situations, but also improves memory-intensive workloads.
• 2: The Linux kernel will not overcommit memory, and only allocate as much memory as defined in overcommit_ratio.

Overcommit_ratio, this setting is only used when overcommit_memory = 2, and defines how many percent of the physical RAM are used. Swap space goes on top of that. The default is “50”, or 50%. I have tried different values from 75, 80, 90 for my vm. Eventually I found 90 is best value for my VM to get MariaDB started.

sysctl vm.overcommit_memory=2 : This will make the value active in this session without requiring a restart.

You can also configure your server to reboot when an OOM-Killer event occurs with following values in /etc/sysctl.conf based on this post:

vm.panic_on_oom=1
kernel.panic=5

or from command line, type:
sysctl vm.panic_on_oom=1
sysctl kernel.panic=5

References:

• Virtual memory settings in Linux - The Problem with Overcommit
• Adjusting PHP-FPM for Performance + Low Memory
• Configuring Apache/PHP/MySQL for Low Memory (RAM) VPS