I’ve noticed some OOM issues on one of my Route10 (1.3s), probably linked to IDS/IDP updates (?) or configuration updates.
Even without IDS/IDP enabled I’m seeing quite high usage. Also a bit worried on the SWAP since it most likely will cause degradation on the eMMC:
total used free shared buff/cache available
Mem: 1002316 890212 88156 11752 23948 0
Swap: 2097148 1194976 902172
It seems /usr/sbin/rcstats is the culprit here
Here’s the OOM log:
[98788.901702] suricata-update invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[98788.901709] CPU: 1 PID: 11923 Comm: suricata-update Tainted: P 5.4.213 #0
[98788.901711] Hardware name: Qualcomm Technologies, Inc. IPQ9574/Alta-Route10 (DT), Version: 1.3s
[98788.901713] Call trace:
[98788.901719] dump_backtrace+0x0/0x16c
[98788.901721] show_stack+0x14/0x1c
[98788.901724] dump_stack+0xb4/0xf8
[98788.901727] dump_header+0x40/0x1e0
[98788.901730] oom_kill_process+0xcc/0x1d4
[98788.901731] out_of_memory+0x2a4/0x2cc
[98788.901734] __alloc_pages_nodemask+0x8fc/0xaec
[98788.901737] __handle_mm_fault+0x210/0xa48
[98788.901738] handle_mm_fault+0x124/0x174
[98788.901740] do_page_fault+0x284/0x41c
[98788.901742] do_translation_fault+0x3c/0x64
[98788.901743] do_mem_abort+0x3c/0x94
[98788.901745] el0_da+0x1c/0x20
[98788.901746] Mem-Info:
[98788.901752] active_anon:101121 inactive_anon:101139 isolated_anon:0
[98788.901752] active_file:34 inactive_file:0 isolated_file:0
[98788.901752] unevictable:0 dirty:18 writeback:0 unstable:0
[98788.901752] slab_reclaimable:2966 slab_unreclaimable:17649
[98788.901752] mapped:6829 shmem:4122 pagetables:1649 bounce:0
[98788.901752] free:16120 free_pcp:759 free_cma:0
[98788.901757] Node 0 active_anon:404484kB inactive_anon:404556kB active_file:136kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:27316kB dirty:72kB writeback:0kB shmem:16488kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[98788.901762] DMA32 free:64480kB min:66560kB low:82176kB high:97792kB active_anon:404740kB inactive_anon:404260kB active_file:136kB inactive_file:0kB unevictable:0kB writepending:472kB present:1032192kB managed:1002316kB mlocked:0kB kernel_stack:2416kB pagetables:6596kB bounce:0kB free_pcp:3036kB local_pcp:308kB free_cma:0kB
[98788.901763] lowmem_reserve[]: 0 0 0
[98788.901766] DMA32: 645*4kB (UEH) 658*8kB (UMEH) 518*16kB (UMEH) 255*32kB (UMEH) 126*64kB (UMEH) 87*128kB (UMEH) 39*256kB (UMEH) 19*512kB (UM) 1*1024kB (M) 0*2048kB 0*4096kB = 64228kB
[98788.901777] 5370 total pagecache pages
[98788.901779] 1213 pages in swap cache
[98788.901781] Swap cache stats: add 1182080, delete 1180902, find 214392/342558
[98788.901782] Free swap = 0kB
[98788.901783] Total swap = 2097148kB
[98788.901783] 258048 pages RAM
[98788.901784] 0 pages HighMem/MovableOnly
[98788.901785] 7469 pages reserved
[98788.901786] Tasks state (memory values in pages):
[98788.901788] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[98788.901797] [ 145] 81 145 303 124 32768 5 0 ubusd
[98788.901799] [ 146] 0 146 204 6 24576 11 0 askfirst
[98788.901802] [ 180] 0 180 236 0 28672 13 0 urngd
[98788.901805] [ 224] 0 224 432 68 32768 159 0 haveged
[98788.901808] [ 888] 0 888 323 0 24576 12 0 setupd.sh
[98788.901811] [ 904] 0 904 334 183 28672 5 0 oopsd
[98788.901813] [ 911] 0 911 323 153 24576 12 0 lldpd.sh
[98788.901817] [ 1054] 0 1054 446 14 32768 33 0 rpcd
[98788.901819] [ 1102] 0 1102 323 40 24576 18 0 klogd
[98788.901821] [ 1103] 0 1103 324 145 28672 3 0 syslogd
[98788.901825] [ 3615] 0 3615 486 193 32768 28 0 netifd
[98788.901827] [ 4543] 0 4543 317 89 32768 16 0 pppd
[98788.901830] [ 6241] 453 6241 768 289 36864 229 0 dnsmasq
[98788.901832] [ 6386] 0 6386 2434 24 49152 524 0 radiusd
[98788.901834] [ 6528] 65534 6528 537 254 36864 30 0 avahi-daemon
[98788.901837] [ 6582] 0 6582 2006 175 45056 112 0 ips
[98788.901839] [ 6611] 0 6611 343 8 32768 16 0 sh
[98788.901842] [ 6666] 0 6666 256 54 24576 17 0 dropbear
[98788.901844] [ 6818] 0 6818 240 58 28672 1 0 fiber_led
[98788.901846] [ 6937] 0 6937 324 160 24576 1 0 crond
[98788.901848] [ 6990] 0 6990 1673 136 36864 86 0 route-swd
[98788.901850] [ 7101] 0 7101 1650 430 40960 385 0 uhttpd
[98788.901853] [ 7432] 0 7432 326 101 28672 4 0 lldpd
[98788.901855] [ 7436] 121 7436 338 140 28672 5 0 lldpd
[98788.901857] [ 7696] 0 7696 324 159 24576 3 0 ntpd
[98788.901860] [ 7721] 0 7721 289 0 28672 19 0 thermald
[98788.901862] [ 7755] 0 7755 2102 571 45056 367 0 rc
[98788.901864] [ 7803] 0 7803 1673 75 36864 96 0 rcmon
[98788.901866] [ 7847] 0 7847 297897 116530 2412544 179742 0 rcstats
[98788.901869] [ 8499] 0 8499 243 133 24576 7 0 odhcp6c
[98788.901871] [ 8914] 65534 8914 3723 2483 61440 417 0 https-dns-proxy
[98788.901873] [ 8915] 65534 8915 3887 2481 61440 236 0 https-dns-proxy
[98788.901876] [ 8916] 65534 8916 3760 2439 57344 65 0 https-dns-proxy
[98788.901879] [ 9853] 149 9853 769856 31401 3227648 321174 0 Suricata-Main
[98788.901882] [ 11921] 0 11921 323 151 24576 0 0 sh
[98788.901884] [ 11923] 0 11923 49864 48660 430080 0 0 suricata-update
[98788.901887] [ 11982] 0 11982 323 1 28672 0 0 sleep
[98788.901889] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=Suricata-Main,pid=9853,uid=149
[98788.901926] Out of memory: Killed process 9853 (Suricata-Main) total-vm:3079424kB, anon-rss:106296kB, file-rss:16648kB, shmem-rss:2660kB, UID:149 pgtables:3152kB oom_score_adj:0
[98789.086652] device br-lan_500 left promiscuous mode
[98789.234656] device br-lan_300 left promiscuous mode
[98789.378643] device br-lan_200 left promiscuous mode
[98789.534647] device br-lan_100 left promiscuous mode
[98789.670644] device br-lan left promiscuous mode
[100058.801571] Suricata-Main invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[100058.801579] CPU: 2 PID: 17017 Comm: Suricata-Main Tainted: P 5.4.213 #0
[100058.801580] Hardware name: Qualcomm Technologies, Inc. IPQ9574/Alta-Route10 (DT), Version: 1.3s
[100058.801582] Call trace:
[100058.801589] dump_backtrace+0x0/0x16c
[100058.801591] show_stack+0x14/0x1c
[100058.801595] dump_stack+0xb4/0xf8
[100058.801597] dump_header+0x40/0x1e0
[100058.801601] oom_kill_process+0xcc/0x1d4
[100058.801602] out_of_memory+0x2a4/0x2cc
[100058.801605] __alloc_pages_nodemask+0x8fc/0xaec
[100058.801608] __handle_mm_fault+0x210/0xa48
[100058.801610] handle_mm_fault+0x124/0x174
[100058.801612] do_page_fault+0x284/0x41c
[100058.801613] do_translation_fault+0x3c/0x64
[100058.801615] do_mem_abort+0x3c/0x94
[100058.801616] el0_da+0x1c/0x20
[100058.801617] Mem-Info:
[100058.801624] active_anon:103883 inactive_anon:103917 isolated_anon:25
[100058.801624] active_file:72 inactive_file:29 isolated_file:0
[100058.801624] unevictable:0 dirty:0 writeback:67 unstable:0
[100058.801624] slab_reclaimable:4019 slab_unreclaimable:18164
[100058.801624] mapped:974 shmem:1100 pagetables:1606 bounce:0
[100058.801624] free:13411 free_pcp:355 free_cma:0
[100058.801629] Node 0 active_anon:415532kB inactive_anon:415668kB active_file:288kB inactive_file:116kB unevictable:0kB isolated(anon):100kB isolated(file):0kB mapped:3896kB dirty:0kB writeback:268kB shmem:4400kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[100058.801635] DMA32 free:53644kB min:55296kB low:76288kB high:89088kB active_anon:415720kB inactive_anon:414824kB active_file:164kB inactive_file:508kB unevictable:0kB writepending:548kB present:1032192kB managed:1002316kB mlocked:0kB kernel_stack:2048kB pagetables:6424kB bounce:0kB free_pcp:1420kB local_pcp:0kB free_cma:0kB
[100058.801636] lowmem_reserve[]: 0 0 0
[100058.801638] DMA32: 816*4kB (UMEH) 665*8kB (UMEH) 454*16kB (UMEH) 335*32kB (UMEH) 168*64kB (UMH) 72*128kB (UMH) 27*256kB (UMH) 2*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 54472kB
[100058.801649] 1406 total pagecache pages
[100058.801652] 192 pages in swap cache
[100058.801653] Swap cache stats: add 7029555, delete 7029473, find 2075984/3945703
[100058.801654] Free swap = 0kB
[100058.801655] Total swap = 2097148kB
[100058.801656] 258048 pages RAM
[100058.801657] 0 pages HighMem/MovableOnly
[100058.801658] 7469 pages reserved
[100058.801659] Tasks state (memory values in pages):
[100058.801660] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[100058.801670] [ 145] 81 145 303 11 32768 35 0 ubusd
[100058.801673] [ 146] 0 146 204 6 24576 12 0 askfirst
[100058.801676] [ 180] 0 180 236 0 28672 14 0 urngd
[100058.801679] [ 224] 0 224 432 9 32768 158 0 haveged
[100058.801682] [ 888] 0 888 323 0 24576 13 0 setupd.sh
[100058.801685] [ 904] 0 904 334 97 28672 8 0 oopsd
[100058.801688] [ 911] 0 911 323 24 24576 13 0 lldpd.sh
[100058.801690] [ 1054] 0 1054 446 12 32768 34 0 rpcd
[100058.801693] [ 1102] 0 1102 323 25 24576 14 0 klogd
[100058.801696] [ 1103] 0 1103 324 91 28672 3 0 syslogd
[100058.801699] [ 3615] 0 3615 486 59 32768 54 0 netifd
[100058.801702] [ 4543] 0 4543 317 64 32768 16 0 pppd
[100058.801705] [ 6241] 453 6241 768 207 36864 230 0 dnsmasq
[100058.801707] [ 6386] 0 6386 2434 23 49152 525 0 radiusd
[100058.801710] [ 6528] 65534 6528 537 154 36864 39 0 avahi-daemon
[100058.801712] [ 6582] 0 6582 2006 28 45056 170 0 ips
[100058.801715] [ 6611] 0 6611 343 24 32768 17 0 sh
[100058.801718] [ 6666] 0 6666 256 2 24576 18 0 dropbear
[100058.801720] [ 6818] 0 6818 240 18 28672 2 0 fiber_led
[100058.801723] [ 6937] 0 6937 324 72 24576 4 0 crond
[100058.801726] [ 6990] 0 6990 1673 64 36864 91 0 route-swd
[100058.801728] [ 7101] 0 7101 1650 25 40960 390 0 uhttpd
[100058.801730] [ 7432] 0 7432 326 23 28672 34 0 lldpd
[100058.801732] [ 7436] 121 7436 338 81 28672 9 0 lldpd
[100058.801735] [ 7696] 0 7696 324 25 24576 20 0 ntpd
[100058.801738] [ 7721] 0 7721 289 0 28672 20 0 thermald
[100058.801740] [ 7755] 0 7755 2102 261 45056 410 0 rc
[100058.801743] [ 7803] 0 7803 1673 45 36864 98 0 rcmon
[100058.801745] [ 7847] 0 7847 298675 19167 2420736 277867 0 rcstats
[100058.801748] [ 8499] 0 8499 243 7 24576 23 0 odhcp6c
[100058.801751] [ 8914] 65534 8914 3723 482 61440 2165 0 https-dns-proxy
[100058.801753] [ 8915] 65534 8915 3887 942 61440 1518 0 https-dns-proxy
[100058.801755] [ 8916] 65534 8916 3760 1126 57344 1344 0 https-dns-proxy
[100058.801759] [ 11921] 0 11921 323 0 24576 13 0 sh
[100058.801761] [ 11923] 0 11923 172320 25705 1417216 141928 0 suricata-update
[100058.801765] [ 17017] 149 17017 251285 159687 2043904 65124 0 Suricata-Main
[100058.801767] [ 17238] 0 17238 323 135 24576 0 0 sleep
[100058.801770] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=rcstats,pid=7847,uid=0
[100058.801786] Out of memory: Killed process 7847 (rcstats) total-vm:1194700kB, anon-rss:75532kB, file-rss:8kB, shmem-rss:1128kB, UID:0 pgtables:2364kB oom_score_adj:0