[2264666.442643] apache2 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [2264666.442650] CPU: 14 PID: 1521442 Comm: apache2 Tainted: G OE 5.7.14 #1-NixOS [2264666.442652] Hardware name: Dell Inc. PowerEdge R620/01W23F, BIOS 2.8.0 06/26/2019 [2264666.442655] Call Trace: [2264666.442679] dump_stack+0x66/0x90 [2264666.442693] dump_header+0x51/0x2f3 [2264666.442701] oom_kill_process.cold+0xb/0x10 [2264666.442707] out_of_memory+0x198/0x460 [2264666.442718] mem_cgroup_out_of_memory+0xba/0xd0 [2264666.442727] try_charge+0x6c0/0x810 [2264666.442733] mem_cgroup_try_charge+0x6c/0x190 [2264666.442739] __add_to_page_cache_locked+0x2a2/0x2f0 [2264666.442749] ? scan_shadow_nodes+0x30/0x30 [2264666.442753] add_to_page_cache_lru+0x48/0xc0 [2264666.442758] pagecache_get_page+0xfa/0x260 [2264666.442763] filemap_fault+0x77d/0x890 [2264666.442768] ? filemap_map_pages+0x230/0x340 [2264666.442775] __do_fault+0x37/0x90 [2264666.442783] __handle_mm_fault+0xe55/0x1660 [2264666.442792] ? apic_timer_interrupt+0xa/0x20 [2264666.442800] handle_mm_fault+0xc4/0x1f0 [2264666.442807] do_user_addr_fault+0x1c4/0x470 [2264666.442813] page_fault+0x34/0x40 [2264666.442819] RIP: 0033:0x7fadc98ad63e [2264666.442829] Code: Bad RIP value. [2264666.442831] RSP: 002b:00007ffd1b8608c0 EFLAGS: 00010202 [2264666.442835] RAX: 00007fadc97a4964 RBX: 00007fadc97a24f0 RCX: 0000000000000000 [2264666.442838] RDX: 00000000000000da RSI: 00007fadc97c859c RDI: 00007fadc97a3ce4 [2264666.442840] RBP: 00007fadc989cf30 R08: 0000000000000000 R09: 0000000000000001 [2264666.442842] R10: 00007fadc98bf500 R11: 0000000000000246 R12: 00007fadc95d5ac8 [2264666.442844] R13: 0000000000000001 R14: 00000000eb80fcc3 R15: 00007ffd1b860984 [2264666.442849] memory: usage 886316kB, limit 1024000kB, failcnt 7823 [2264666.442852] memory+swap: usage 886316kB, limit 9007199254740988kB, failcnt 0 [2264666.442854] kmem: usage 0kB, limit 9007199254740988kB, failcnt 0 [2264666.442855] Memory cgroup stats for /osctl/pool.tank/group.default/user.2397/ct.17320: [2264668.142210] anon 224686080 file 668160000 kernel_stack 0 slab 0 sock 0 shmem 652210176 file_mapped 50417664 file_dirty 0 file_writeback 405504 anon_thp 0 inactive_anon 205455360 active_anon 682057728 inactive_file 0 active_file 2269184 unevictable 0 slab_reclaimable 0 slab_unreclaimable 0 pgfault 1958213328 pgmajfault 3377523 workingset_refault 860922 workingset_activate 312745 workingset_nodereclaim 0 pgrefill 1011953 pgscan 209885217 pgsteal 2786704 pgactivate 109858 pgdeactivate 882671 pglazyfree 0 pglazyfreed 0 thp_fault_alloc 0 thp_collapse_alloc 0 [2264668.142213] Tasks state (memory values in pages): [2264668.142214] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [2264668.142229] [ 167133] 100050 167133 15320 5135 151552 0 -1000 osctld-ct-wrapp [2264668.142235] [ 167938] 100050 167938 1133 756 49152 0 0 lxc-start [2264668.142244] [ 183239] 812777472 183239 3964 215 69632 0 -1000 sshd [2264668.142253] [3122794] 812777472 3122794 283 1 45056 0 0 sshd [2264668.142259] [3123806] 812777472 3123806 3809 64 65536 0 0 sshd [2264668.142266] [3136688] 812777472 3136688 3964 215 69632 0 0 sshd [2264668.142272] [ 977500] 812777580 977500 10921 209 77824 0 0 tlsmgr [2264668.142277] [ 187996] 812777472 187996 10868 209 77824 0 0 master [2264668.142283] [2400019] 812777580 2400019 10875 201 77824 0 0 pickup [2264668.142288] [1194455] 812777472 1194455 68331 3448 212992 0 0 apache2 [2264668.142295] [ 414795] 812777505 414795 76565 2619 212992 0 0 apache2 [2264668.142303] [1508403] 812777505 1508403 76565 2619 212992 0 0 apache2 [2264668.142309] [1767780] 812777505 1767780 76565 2619 212992 0 0 apache2 [2264668.142316] [1768765] 812777505 1768765 76565 2619 212992 0 0 apache2 [2264668.142323] [4141147] 812777505 4141147 76565 2619 212992 0 0 apache2 [2264668.142329] [4141182] 812777505 4141182 76565 2619 212992 0 0 apache2 [2264668.142465] [ 221695] 812777505 221695 76565 2619 212992 0 0 apache2 [2264668.142472] [2873019] 812777505 2873019 76565 2619 212992 0 0 apache2 [2264668.142478] [2873514] 812777505 2873514 76565 2619 212992 0 0 apache2 [2264668.142486] [2873515] 812777505 2873515 76565 2619 212992 0 0 apache2 [2264668.142491] [3114096] 812777505 3114096 76575 2629 212992 0 0 apache2 [2264668.142496] [3114134] 812778490 3114134 78971 10753 294912 0 0 apache2 [2264668.142501] [3118746] 812777505 3118746 76575 2616 212992 0 0 apache2 [2264668.142506] [3118786] 812777505 3118786 76575 2608 212992 0 0 apache2 [2264668.142511] [3118821] 812777505 3118821 76575 2608 212992 0 0 apache2 [2264668.142516] [3118830] 812778490 3118830 79419 11549 311296 0 0 apache2 [2264668.142522] [3118842] 812777505 3118842 76573 2606 212992 0 0 apache2 [2264668.142528] [3118845] 812778490 3118845 78907 10865 303104 0 0 apache2 [2264668.142532] [3118854] 812777505 3118854 76573 2623 217088 0 0 apache2 [2264668.142537] [3118861] 812778490 3118861 78395 10291 290816 0 0 apache2 [2264668.142544] [3119141] 812777505 3119141 76565 2598 212992 0 0 apache2 [2264668.142550] [3119162] 812778490 3119162 77883 9885 282624 0 0 apache2 [2264668.142555] [3119364] 812777505 3119364 76565 2598 212992 0 0 apache2 [2264668.142560] [3119365] 812777505 3119365 76565 2617 212992 0 0 apache2 [2264668.142565] [3120168] 812778490 3120168 78889 10594 290816 0 0 apache2 [2264668.142571] [3120303] 812777505 3120303 76562 2569 212992 0 0 apache2 [2264668.142576] [3120325] 812777505 3120325 76562 2572 212992 0 0 apache2 [2264668.142581] [3120335] 812777505 3120335 76562 2580 212992 0 0 apache2 [2264668.142587] [3120346] 812777505 3120346 76562 2577 212992 0 0 apache2 [2264668.142597] [3121014] 812778477 3121014 77683 6889 266240 0 0 apache2 [2264668.142602] [3121067] 812777505 3121067 76572 2605 212992 0 0 apache2 [2264668.142606] [3121070] 812777505 3121070 76572 2605 212992 0 0 apache2 [2264668.142611] [3121071] 812777505 3121071 76572 2607 212992 0 0 apache2 [2264668.142616] [3121072] 812777505 3121072 76572 2605 212992 0 0 apache2 [2264668.142623] [3121160] 812777505 3121160 76565 2598 212992 0 0 apache2 [2264668.142628] [3121177] 812777505 3121177 76567 2604 212992 0 0 apache2 [2264668.142633] [3121324] 812777505 3121324 68356 2568 196608 0 0 apache2 [2264668.142638] [3121325] 812777505 3121325 76558 2569 208896 0 0 apache2 [2264668.142643] [3121434] 812777505 3121434 68337 2550 188416 0 0 apache2 [2264668.142648] [3121441] 812777505 3121441 68335 2549 188416 0 0 apache2 [2264668.142652] [3121452] 812777505 3121452 68337 2550 188416 0 0 apache2 [2264668.142658] [3121572] 812777505 3121572 76565 2598 212992 0 0 apache2 [2264668.142666] [3122430] 812778490 3122430 76575 3535 237568 0 0 apache2 [2264668.142671] [ 229977] 812777472 229977 3803 268 65536 0 0 haproxy [2264668.142682] [ 183112] 812777472 183112 1347 28 49152 0 0 agetty [2264668.142689] [ 183128] 812777472 183128 1347 27 49152 0 0 agetty [2264668.142695] [ 183143] 812777472 183143 1347 29 45056 0 0 agetty [2264668.142701] [ 183180] 812777472 183180 1347 28 45056 0 0 agetty [2264668.142709] [ 188051] 812777580 188051 10906 219 77824 0 0 qmgr [2264668.142716] [ 180946] 812777472 180946 15344 5022 167936 0 0 systemd-journal [2264668.142724] [ 181086] 812777472 181086 4888 190 69632 0 -1000 systemd-udevd [2264668.142731] [ 181210] 812777472 181210 37677 70 57344 0 0 bindfs [2264668.142738] [ 181181] 812777472 181181 37677 70 61440 0 0 bindfs [2264668.142745] [ 181179] 812777472 181179 37677 70 53248 0 0 bindfs [2264668.142751] [ 181171] 812777472 181171 37677 70 65536 0 0 bindfs [2264668.142757] [ 181170] 812777472 181170 37677 69 61440 0 0 bindfs [2264668.142764] [ 181184] 812777472 181184 37677 70 57344 0 0 bindfs [2264668.142770] [ 181183] 812777472 181183 37677 69 57344 0 0 bindfs [2264668.142776] [ 181136] 812777472 181136 37677 70 57344 0 0 bindfs [2264668.142783] [ 181265] 812777472 181265 37677 69 57344 0 0 bindfs [2264668.142789] [ 181275] 812777472 181275 37677 69 57344 0 0 bindfs [2264668.142795] [ 181231] 812777472 181231 37677 70 53248 0 0 bindfs [2264668.142802] [ 181229] 812777472 181229 37677 69 53248 0 0 bindfs [2264668.142810] [ 181366] 812777472 181366 37677 70 57344 0 0 bindfs [2264668.142817] [ 181520] 812777472 181520 37677 69 53248 0 0 bindfs [2264668.142822] [ 181430] 812777472 181430 37677 70 61440 0 0 bindfs [2264668.142828] [ 181521] 812777472 181521 37677 69 57344 0 0 bindfs [2264668.142834] [ 181484] 812777472 181484 37677 69 61440 0 0 bindfs [2264668.142840] [ 181464] 812777472 181464 37677 70 57344 0 0 bindfs [2264668.142847] [ 181454] 812777472 181454 37677 70 53248 0 0 bindfs [2264668.142854] [ 181514] 812777472 181514 37677 69 57344 0 0 bindfs [2264668.142860] [ 181476] 812777472 181476 37677 70 61440 0 0 bindfs [2264668.142867] [ 181562] 812777472 181562 37677 70 53248 0 0 bindfs [2264668.142874] [ 181533] 812777472 181533 37677 69 49152 0 0 bindfs [2264668.142880] [ 181530] 812777472 181530 37677 69 53248 0 0 bindfs [2264668.142887] [ 181571] 812777472 181571 37677 70 61440 0 0 bindfs [2264668.142893] [ 181554] 812777472 181554 37677 70 53248 0 0 bindfs [2264668.142899] [ 181567] 812777472 181567 37677 69 57344 0 0 bindfs [2264668.142906] [ 181613] 812777472 181613 37677 70 53248 0 0 bindfs [2264668.142914] [ 181785] 812777472 181785 1373 45 53248 0 0 atd [2264668.142921] [ 181773] 812777472 181773 4870 250 73728 0 0 systemd-logind [2264668.142927] [ 181776] 812777576 181776 2206 144 57344 0 -900 dbus-daemon [2264668.142933] [ 181782] 812777472 181782 2127 68 57344 0 0 cron [2264668.142939] [3156225] 812777472 3156225 2127 65 53248 0 0 cron [2264668.142946] [ 183105] 812777472 183105 62913 3490 122880 0 0 fail2ban-server [2264668.142961] [ 183647] 812777582 183647 697418 0 675840 0 0 mysqld [2264668.142967] [ 183278] 812777578 183278 19118 154 65536 0 0 ntpd [2264668.142974] [ 183063] 812777472 183063 12918 5070 139264 0 0 salt-minion [2264668.142980] [ 188572] 812777472 188572 248186 10597 303104 0 0 salt-minion [2264668.142985] [ 188774] 812777472 188774 32986 5472 159744 0 0 salt-minion [2264668.142993] [ 183073] 812777472 183073 27584 1939 110592 0 0 unattended-upgr [2264668.142999] [ 183153] 812777472 183153 1347 29 49152 0 0 agetty [2264668.143043] [ 969537] 812777472 969537 56457 368 81920 0 0 rsyslogd [2264668.143052] [ 230060] 812777583 230060 4007 418 65536 0 0 haproxy [2264668.143070] [3689752] 812777581 3689752 6145 223 86016 0 0 zabbix_agentd [2264668.143078] [3689766] 812777581 3689766 6145 340 90112 0 0 zabbix_agentd [2264668.143086] [3689767] 812777581 3689767 6184 279 90112 0 0 zabbix_agentd [2264668.143100] [ 170899] 812777472 170899 42720 704 94208 0 0 systemd [2264668.143105] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=lxc.payload.17320,mems_allowed=0-1,oom_memcg=/osctl/pool.tank/group.default/user.2397/ct.17320,task_memcg=/osctl/pool.tank/group.default/user.2397/ct.17320/user-owned/lxc.payload.17320,task=apache2,pid=3118830,uid=812778490 [2264668.143160] Memory cgroup out of memory: Killed process 3118830 (apache2) total-vm:317676kB, anon-rss:22752kB, file-rss:788kB, shmem-rss:22656kB, UID:812778490 pgtables:304kB oom_score_adj:0