[vpsFree.cz: community-list] OOM kill s dostatkem paměti
Petr Gregor
gregy at gregy.cz
Mon Oct 19 15:18:58 CEST 2020
Zdar,
rád bych poprosil o pomoc s debugováním OOM killů v mém VM. Měl jsem za to,
že mám spoustu volného místa a proto mě kill překvapil (díky za nové
reportování z vpsadminu!). VM monitoruji zabbixem a ten tvrdí, že dostupné
paměti je dost (špička na konci je kill)
[image: image.png]
výstup z free -m vypadá takto:
root at hostingf:~# free -m
total used free shared buff/cache
available
Mem: 1000 238 100 621 661
761
Swap: 0 0 0
/proc/meminfo MemoryAvailable sedí se zabbixem a free: "MemAvailable:
777548 kB"
Trošku zarážející je, že "volné" paměti je pouze 100M. Rozhodující pro OOM
je ale "dostupná" paměť, že?
V příloze posílám výstup z dmesg náležící k OOM killu.
Není mi jasné, proč OOM kill vůbec nastal. Součet všech rss v dmesg reportu
ani zdaleka nedosahuje 1G. Předpokládám, že mi nějak uniká něco
podstatného, ale nevím moc co hledat. Budu rád za každé nakopnutí.
Gregy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.vpsfree.cz/pipermail/community-list/attachments/20201019/84ce1f62/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 155714 bytes
Desc: not available
URL: <http://lists.vpsfree.cz/pipermail/community-list/attachments/20201019/84ce1f62/attachment-0001.png>
-------------- next part --------------
[2264666.442643] apache2 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[2264666.442650] CPU: 14 PID: 1521442 Comm: apache2 Tainted: G OE 5.7.14 #1-NixOS
[2264666.442652] Hardware name: Dell Inc. PowerEdge R620/01W23F, BIOS 2.8.0 06/26/2019
[2264666.442655] Call Trace:
[2264666.442679] dump_stack+0x66/0x90
[2264666.442693] dump_header+0x51/0x2f3
[2264666.442701] oom_kill_process.cold+0xb/0x10
[2264666.442707] out_of_memory+0x198/0x460
[2264666.442718] mem_cgroup_out_of_memory+0xba/0xd0
[2264666.442727] try_charge+0x6c0/0x810
[2264666.442733] mem_cgroup_try_charge+0x6c/0x190
[2264666.442739] __add_to_page_cache_locked+0x2a2/0x2f0
[2264666.442749] ? scan_shadow_nodes+0x30/0x30
[2264666.442753] add_to_page_cache_lru+0x48/0xc0
[2264666.442758] pagecache_get_page+0xfa/0x260
[2264666.442763] filemap_fault+0x77d/0x890
[2264666.442768] ? filemap_map_pages+0x230/0x340
[2264666.442775] __do_fault+0x37/0x90
[2264666.442783] __handle_mm_fault+0xe55/0x1660
[2264666.442792] ? apic_timer_interrupt+0xa/0x20
[2264666.442800] handle_mm_fault+0xc4/0x1f0
[2264666.442807] do_user_addr_fault+0x1c4/0x470
[2264666.442813] page_fault+0x34/0x40
[2264666.442819] RIP: 0033:0x7fadc98ad63e
[2264666.442829] Code: Bad RIP value.
[2264666.442831] RSP: 002b:00007ffd1b8608c0 EFLAGS: 00010202
[2264666.442835] RAX: 00007fadc97a4964 RBX: 00007fadc97a24f0 RCX: 0000000000000000
[2264666.442838] RDX: 00000000000000da RSI: 00007fadc97c859c RDI: 00007fadc97a3ce4
[2264666.442840] RBP: 00007fadc989cf30 R08: 0000000000000000 R09: 0000000000000001
[2264666.442842] R10: 00007fadc98bf500 R11: 0000000000000246 R12: 00007fadc95d5ac8
[2264666.442844] R13: 0000000000000001 R14: 00000000eb80fcc3 R15: 00007ffd1b860984
[2264666.442849] memory: usage 886316kB, limit 1024000kB, failcnt 7823
[2264666.442852] memory+swap: usage 886316kB, limit 9007199254740988kB, failcnt 0
[2264666.442854] kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
[2264666.442855] Memory cgroup stats for /osctl/pool.tank/group.default/user.2397/ct.17320:
[2264668.142210] anon 224686080
file 668160000
kernel_stack 0
slab 0
sock 0
shmem 652210176
file_mapped 50417664
file_dirty 0
file_writeback 405504
anon_thp 0
inactive_anon 205455360
active_anon 682057728
inactive_file 0
active_file 2269184
unevictable 0
slab_reclaimable 0
slab_unreclaimable 0
pgfault 1958213328
pgmajfault 3377523
workingset_refault 860922
workingset_activate 312745
workingset_nodereclaim 0
pgrefill 1011953
pgscan 209885217
pgsteal 2786704
pgactivate 109858
pgdeactivate 882671
pglazyfree 0
pglazyfreed 0
thp_fault_alloc 0
thp_collapse_alloc 0
[2264668.142213] Tasks state (memory values in pages):
[2264668.142214] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[2264668.142229] [ 167133] 100050 167133 15320 5135 151552 0 -1000 osctld-ct-wrapp
[2264668.142235] [ 167938] 100050 167938 1133 756 49152 0 0 lxc-start
[2264668.142244] [ 183239] 812777472 183239 3964 215 69632 0 -1000 sshd
[2264668.142253] [3122794] 812777472 3122794 283 1 45056 0 0 sshd
[2264668.142259] [3123806] 812777472 3123806 3809 64 65536 0 0 sshd
[2264668.142266] [3136688] 812777472 3136688 3964 215 69632 0 0 sshd
[2264668.142272] [ 977500] 812777580 977500 10921 209 77824 0 0 tlsmgr
[2264668.142277] [ 187996] 812777472 187996 10868 209 77824 0 0 master
[2264668.142283] [2400019] 812777580 2400019 10875 201 77824 0 0 pickup
[2264668.142288] [1194455] 812777472 1194455 68331 3448 212992 0 0 apache2
[2264668.142295] [ 414795] 812777505 414795 76565 2619 212992 0 0 apache2
[2264668.142303] [1508403] 812777505 1508403 76565 2619 212992 0 0 apache2
[2264668.142309] [1767780] 812777505 1767780 76565 2619 212992 0 0 apache2
[2264668.142316] [1768765] 812777505 1768765 76565 2619 212992 0 0 apache2
[2264668.142323] [4141147] 812777505 4141147 76565 2619 212992 0 0 apache2
[2264668.142329] [4141182] 812777505 4141182 76565 2619 212992 0 0 apache2
[2264668.142465] [ 221695] 812777505 221695 76565 2619 212992 0 0 apache2
[2264668.142472] [2873019] 812777505 2873019 76565 2619 212992 0 0 apache2
[2264668.142478] [2873514] 812777505 2873514 76565 2619 212992 0 0 apache2
[2264668.142486] [2873515] 812777505 2873515 76565 2619 212992 0 0 apache2
[2264668.142491] [3114096] 812777505 3114096 76575 2629 212992 0 0 apache2
[2264668.142496] [3114134] 812778490 3114134 78971 10753 294912 0 0 apache2
[2264668.142501] [3118746] 812777505 3118746 76575 2616 212992 0 0 apache2
[2264668.142506] [3118786] 812777505 3118786 76575 2608 212992 0 0 apache2
[2264668.142511] [3118821] 812777505 3118821 76575 2608 212992 0 0 apache2
[2264668.142516] [3118830] 812778490 3118830 79419 11549 311296 0 0 apache2
[2264668.142522] [3118842] 812777505 3118842 76573 2606 212992 0 0 apache2
[2264668.142528] [3118845] 812778490 3118845 78907 10865 303104 0 0 apache2
[2264668.142532] [3118854] 812777505 3118854 76573 2623 217088 0 0 apache2
[2264668.142537] [3118861] 812778490 3118861 78395 10291 290816 0 0 apache2
[2264668.142544] [3119141] 812777505 3119141 76565 2598 212992 0 0 apache2
[2264668.142550] [3119162] 812778490 3119162 77883 9885 282624 0 0 apache2
[2264668.142555] [3119364] 812777505 3119364 76565 2598 212992 0 0 apache2
[2264668.142560] [3119365] 812777505 3119365 76565 2617 212992 0 0 apache2
[2264668.142565] [3120168] 812778490 3120168 78889 10594 290816 0 0 apache2
[2264668.142571] [3120303] 812777505 3120303 76562 2569 212992 0 0 apache2
[2264668.142576] [3120325] 812777505 3120325 76562 2572 212992 0 0 apache2
[2264668.142581] [3120335] 812777505 3120335 76562 2580 212992 0 0 apache2
[2264668.142587] [3120346] 812777505 3120346 76562 2577 212992 0 0 apache2
[2264668.142597] [3121014] 812778477 3121014 77683 6889 266240 0 0 apache2
[2264668.142602] [3121067] 812777505 3121067 76572 2605 212992 0 0 apache2
[2264668.142606] [3121070] 812777505 3121070 76572 2605 212992 0 0 apache2
[2264668.142611] [3121071] 812777505 3121071 76572 2607 212992 0 0 apache2
[2264668.142616] [3121072] 812777505 3121072 76572 2605 212992 0 0 apache2
[2264668.142623] [3121160] 812777505 3121160 76565 2598 212992 0 0 apache2
[2264668.142628] [3121177] 812777505 3121177 76567 2604 212992 0 0 apache2
[2264668.142633] [3121324] 812777505 3121324 68356 2568 196608 0 0 apache2
[2264668.142638] [3121325] 812777505 3121325 76558 2569 208896 0 0 apache2
[2264668.142643] [3121434] 812777505 3121434 68337 2550 188416 0 0 apache2
[2264668.142648] [3121441] 812777505 3121441 68335 2549 188416 0 0 apache2
[2264668.142652] [3121452] 812777505 3121452 68337 2550 188416 0 0 apache2
[2264668.142658] [3121572] 812777505 3121572 76565 2598 212992 0 0 apache2
[2264668.142666] [3122430] 812778490 3122430 76575 3535 237568 0 0 apache2
[2264668.142671] [ 229977] 812777472 229977 3803 268 65536 0 0 haproxy
[2264668.142682] [ 183112] 812777472 183112 1347 28 49152 0 0 agetty
[2264668.142689] [ 183128] 812777472 183128 1347 27 49152 0 0 agetty
[2264668.142695] [ 183143] 812777472 183143 1347 29 45056 0 0 agetty
[2264668.142701] [ 183180] 812777472 183180 1347 28 45056 0 0 agetty
[2264668.142709] [ 188051] 812777580 188051 10906 219 77824 0 0 qmgr
[2264668.142716] [ 180946] 812777472 180946 15344 5022 167936 0 0 systemd-journal
[2264668.142724] [ 181086] 812777472 181086 4888 190 69632 0 -1000 systemd-udevd
[2264668.142731] [ 181210] 812777472 181210 37677 70 57344 0 0 bindfs
[2264668.142738] [ 181181] 812777472 181181 37677 70 61440 0 0 bindfs
[2264668.142745] [ 181179] 812777472 181179 37677 70 53248 0 0 bindfs
[2264668.142751] [ 181171] 812777472 181171 37677 70 65536 0 0 bindfs
[2264668.142757] [ 181170] 812777472 181170 37677 69 61440 0 0 bindfs
[2264668.142764] [ 181184] 812777472 181184 37677 70 57344 0 0 bindfs
[2264668.142770] [ 181183] 812777472 181183 37677 69 57344 0 0 bindfs
[2264668.142776] [ 181136] 812777472 181136 37677 70 57344 0 0 bindfs
[2264668.142783] [ 181265] 812777472 181265 37677 69 57344 0 0 bindfs
[2264668.142789] [ 181275] 812777472 181275 37677 69 57344 0 0 bindfs
[2264668.142795] [ 181231] 812777472 181231 37677 70 53248 0 0 bindfs
[2264668.142802] [ 181229] 812777472 181229 37677 69 53248 0 0 bindfs
[2264668.142810] [ 181366] 812777472 181366 37677 70 57344 0 0 bindfs
[2264668.142817] [ 181520] 812777472 181520 37677 69 53248 0 0 bindfs
[2264668.142822] [ 181430] 812777472 181430 37677 70 61440 0 0 bindfs
[2264668.142828] [ 181521] 812777472 181521 37677 69 57344 0 0 bindfs
[2264668.142834] [ 181484] 812777472 181484 37677 69 61440 0 0 bindfs
[2264668.142840] [ 181464] 812777472 181464 37677 70 57344 0 0 bindfs
[2264668.142847] [ 181454] 812777472 181454 37677 70 53248 0 0 bindfs
[2264668.142854] [ 181514] 812777472 181514 37677 69 57344 0 0 bindfs
[2264668.142860] [ 181476] 812777472 181476 37677 70 61440 0 0 bindfs
[2264668.142867] [ 181562] 812777472 181562 37677 70 53248 0 0 bindfs
[2264668.142874] [ 181533] 812777472 181533 37677 69 49152 0 0 bindfs
[2264668.142880] [ 181530] 812777472 181530 37677 69 53248 0 0 bindfs
[2264668.142887] [ 181571] 812777472 181571 37677 70 61440 0 0 bindfs
[2264668.142893] [ 181554] 812777472 181554 37677 70 53248 0 0 bindfs
[2264668.142899] [ 181567] 812777472 181567 37677 69 57344 0 0 bindfs
[2264668.142906] [ 181613] 812777472 181613 37677 70 53248 0 0 bindfs
[2264668.142914] [ 181785] 812777472 181785 1373 45 53248 0 0 atd
[2264668.142921] [ 181773] 812777472 181773 4870 250 73728 0 0 systemd-logind
[2264668.142927] [ 181776] 812777576 181776 2206 144 57344 0 -900 dbus-daemon
[2264668.142933] [ 181782] 812777472 181782 2127 68 57344 0 0 cron
[2264668.142939] [3156225] 812777472 3156225 2127 65 53248 0 0 cron
[2264668.142946] [ 183105] 812777472 183105 62913 3490 122880 0 0 fail2ban-server
[2264668.142961] [ 183647] 812777582 183647 697418 0 675840 0 0 mysqld
[2264668.142967] [ 183278] 812777578 183278 19118 154 65536 0 0 ntpd
[2264668.142974] [ 183063] 812777472 183063 12918 5070 139264 0 0 salt-minion
[2264668.142980] [ 188572] 812777472 188572 248186 10597 303104 0 0 salt-minion
[2264668.142985] [ 188774] 812777472 188774 32986 5472 159744 0 0 salt-minion
[2264668.142993] [ 183073] 812777472 183073 27584 1939 110592 0 0 unattended-upgr
[2264668.142999] [ 183153] 812777472 183153 1347 29 49152 0 0 agetty
[2264668.143043] [ 969537] 812777472 969537 56457 368 81920 0 0 rsyslogd
[2264668.143052] [ 230060] 812777583 230060 4007 418 65536 0 0 haproxy
[2264668.143070] [3689752] 812777581 3689752 6145 223 86016 0 0 zabbix_agentd
[2264668.143078] [3689766] 812777581 3689766 6145 340 90112 0 0 zabbix_agentd
[2264668.143086] [3689767] 812777581 3689767 6184 279 90112 0 0 zabbix_agentd
[2264668.143100] [ 170899] 812777472 170899 42720 704 94208 0 0 systemd
[2264668.143105] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=lxc.payload.17320,mems_allowed=0-1,oom_memcg=/osctl/pool.tank/group.default/user.2397/ct.17320,task_memcg=/osctl/pool.tank/group.default/user.2397/ct.17320/user-owned/lxc.payload.17320,task=apache2,pid=3118830,uid=812778490
[2264668.143160] Memory cgroup out of memory: Killed process 3118830 (apache2) total-vm:317676kB, anon-rss:22752kB, file-rss:788kB, shmem-rss:22656kB, UID:812778490 pgtables:304kB oom_score_adj:0
More information about the Community-list
mailing list