cve 病毒排查记录

2023-09-08

腾讯云服务出现报警, 说是个人购买的云主机里有异常文件. 上去排查看起来确实是木马病毒, 但是已经被腾讯云自动删除了. 试用了高级版进行全面扫描, 确实没有异常文件了. 剩下需要担心的就是从哪里被入侵的, 漏洞是否仍然可以被侵入. 以前出现过几次类似的, 都是云主机部署了开源组件暴露了端口. 比较可疑的是新部署测试的flink集群, 但是我上去排查的时候直接把所有docker容器都关了, 导致后面想要详细排查flink是否有异常jar包也没法查找了. 虽然flink部署在docker容器里, 但后面排查发现有些漏洞会尝试突破docker overlay层的限制, 估计还是被突破了. 最后只能把所有用户的crontab排查了下, 把/tmp目录的文件给删了.

后面搜索了一些资料, 看起来比较合理的路径应该是这样: flink集群的web端口没有认证, 支持执行flink jar包 --> 黑客脚本上次jar包执行 -> 首先执行提权root脚本 -> 然后执行overlay突破脚本. 开放端口风险还是太大了, 开发测试机器可以这么搞, 部署了生产服务的机器这么玩就危险了, 一般是被用来cpu挖矿, 更严重的后果就是资料都被删除, 或是被加密勒索.

有两个异常文件, 涉及两个cve漏洞.

lushput: Win32.Trojan.Cve-2021-4043.Mqil
- 执行脚本会自动提权为root用户身份: https://github.com/berdav/CVE-2021-4034
loadit: Win32.Trojan.Cve-2021-3493.Swhl
- ubuntu漏洞, 可能会突破docker的overlay文件限制: https://github.com/briskets/CVE-2021-3493

flink web 支持上传jar包执行

flink web支持上传任意jar包执行, 并且没有鉴权信息

在机器上搜索, 文件保存在/var/lib/docker/overlay2中

ubuntu@geedev:~$ sudo find / -name "*stripe-0.0.1*"
/var/lib/docker/overlay2/d2700f7ffbfc5bf3b7c61e306f7abaabe7aa158e124df15124082a059e62aaaf/merged/tmp/flink-web-228c44b7-5fa2-491b-93a6-2cf356ca68cf/flink-web-upload/f006ebe7-2a01-47d0-8e18-fb811b9ac19e_stripe-0.0.1-SNAPSHOT.jar
/var/lib/docker/overlay2/d2700f7ffbfc5bf3b7c61e306f7abaabe7aa158e124df15124082a059e62aaaf/diff/tmp/flink-web-228c44b7-5fa2-491b-93a6-2cf356ca68cf/flink-web-upload/f006ebe7-2a01-47d0-8e18-fb811b9ac19e_stripe-0.0.1-SNAPSHOT.jar

排查的日志记录

出问题时候的cpu略微高一点, 内存也略高, 但还是没有暴露出导致oom的高峰.

/var/log里有不少系统日志, 出现问题的时候可以进行排查, 包含了用户执行命令, journal目录包含了内核操作信息.

/var/log$ ls
alternatives.log       auth.log.1     cloud-init-output.log  dpkg.log.1      journal        lastlog            syslog.3.gz                      ubuntu-advantage.log
alternatives.log.1     auth.log.2.gz  cloud-init.log         dpkg.log.2.gz   kern.log       ntpstats           syslog.4.gz                      ubuntu-advantage.log.1
alternatives.log.2.gz  auth.log.3.gz  dist-upgrade           dpkg.log.3.gz   kern.log.1     private            ubuntu-advantage-timer.log       unattended-upgrades
apport.log             auth.log.4.gz  dmesg                  dpkg.log.4.gz   kern.log.2.gz  qcloud_action.log  ubuntu-advantage-timer.log.1     wtmp
apport.log.1           bootstrap.log  dmesg.0                faillog         kern.log.3.gz  syslog             ubuntu-advantage-timer.log.2.gz
apt                    btmp           dmesg.1.gz             fontconfig.log  kern.log.4.gz  syslog.1           ubuntu-advantage-timer.log.3.gz
auth.log               btmp.1         dpkg.log               installer       landscape      syslog.2.gz        ubuntu-advantage-timer.log.4.gz

排查相关文件信息

/var/log$ sudo grep -R "lushput" .


./syslog:Sep  8 11:10:17 geedev kernel: [6207451.551251] traps: lushput[19164] general protection fault ip:7fa78a2c3540 sp:7ffd1ede1a18 error:0 in libc.so.6[7fa78a247000+195000]
grep: ./journal/50404d34021d4b4980706fd359fde87b/system.journal: binary file matches
grep: ./journal/50404d34021d4b4980706fd359fde87b/user-1000.journal: binary file matches
./kern.log:Sep  8 11:10:17 geedev kernel: [6207451.551251] traps: lushput[19164] general protection fault ip:7fa78a2c3540 sp:7ffd1ede1a18 error:0 in libc.so.6[7fa78a247000+195000]
./auth.log:Sep  8 12:22:14 geedev sudo:   ubuntu : TTY=pts/19 ; PWD=/var/log ; USER=root ; COMMAND=/usr/bin/grep -R lushput .
./auth.log:Sep  8 12:36:14 geedev sudo:   ubuntu : TTY=pts/19 ; PWD=/var/log ; USER=root ; COMMAND=/usr/bin/grep -R lushput .
./auth.log:Sep  8 14:19:33 geedev sudo:   ubuntu : TTY=pts/19 ; PWD=/var/log ; USER=root ; COMMAND=/usr/bin/grep -R lushput .

出现了几条关键记录, lushput进程异常访问内存被系统拦截住, 估计这就是恶意程序的操作了. 紧接着出现了oom, 估计也是恶意程序的内存操作触发, 导致一个linux程序sshd被自动关闭.

traps: lushput[19164] general protection fault ip:7fa78a2c3540 sp:7ffd1ede1a18 error:0 in libc.so.6[7fa78a247000+195000]
Sep  8 11:10:54 geedev kernel: [6207487.535202] systemd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0

vim syslog

Sep  8 11:09:07 geedev systemd[1]: run-docker-runtime\x2drunc-moby-ca39d45d1796cb69a3f32c064426f8ab31f051e9273a35a427968e192ebfd83c-runc.ooRZpj.mount: Deactivated successfully.
Sep  8 11:10:01 geedev CRON[19005]: (root) CMD (flock -xn /tmp/stargate.lock -c '/usr/local/qcloud/stargate/admin/start.sh > /dev/null 2>&1 &')
Sep  8 11:10:01 geedev CRON[19006]: (root) CMD (flock -xn /tmp/stargate.lock -c '/usr/local/qcloud/stargate/admin/start.sh > /dev/null 2>&1 &')
Sep  8 11:10:01 geedev CRON[19008]: (root) CMD (/usr/local/qcloud/YunJing/clearRules.sh > /dev/null 2>&1)
Sep  8 11:10:17 geedev kernel: [6207451.551248] show_signal: 8 callbacks suppressed
Sep  8 11:10:17 geedev kernel: [6207451.551251] traps: lushput[19164] general protection fault ip:7fa78a2c3540 sp:7ffd1ede1a18 error:0 in libc.so.6[7fa78a247000+195000]
Sep  8 11:10:54 geedev kernel: [6207487.535202] systemd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Sep  8 11:10:55 geedev kernel: [6207487.536514] CPU: 1 PID: 1 Comm: systemd Kdump: loaded Not tainted 5.15.0-72-generic #79-Ubuntu
Sep  8 11:10:55 geedev kernel: [6207487.537591] Hardware name: Tencent Cloud CVM, BIOS seabios-1.9.1-qemu-project.org 04/01/2014
Sep  8 11:10:55 geedev kernel: [6207487.538642] Call Trace:
Sep  8 11:10:55 geedev kernel: [6207487.538979]  <TASK>
Sep  8 11:10:55 geedev kernel: [6207487.539275]  show_stack+0x52/0x5c
Sep  8 11:10:55 geedev kernel: [6207487.539726]  dump_stack_lvl+0x4a/0x63
Sep  8 11:10:55 geedev kernel: [6207487.540208]  dump_stack+0x10/0x16
Sep  8 11:10:55 geedev kernel: [6207487.540652]  dump_header+0x53/0x228
Sep  8 11:10:55 geedev kernel: [6207487.541113]  oom_kill_process.cold+0xb/0x10
Sep  8 11:10:55 geedev kernel: [6207487.541662]  out_of_memory+0x106/0x2e0
Sep  8 11:10:55 geedev kernel: [6207487.542706]  __alloc_pages_slowpath.constprop.0+0x9b7/0xa80
Sep  8 11:10:55 geedev kernel: [6207487.543937]  __alloc_pages+0x311/0x330
Sep  8 11:10:55 geedev kernel: [6207487.544946]  alloc_pages+0x9e/0x1e0
Sep  8 11:10:55 geedev kernel: [6207487.545906]  __page_cache_alloc+0x7e/0x90
Sep  8 11:10:55 geedev kernel: [6207487.546907]  pagecache_get_page+0x152/0x590
Sep  8 11:10:55 geedev kernel: [6207487.547912]  ? page_cache_ra_unbounded+0x166/0x210
Sep  8 11:10:55 geedev kernel: [6207487.548980]  filemap_fault+0x488/0xab0
Sep  8 11:10:55 geedev kernel: [6207487.549914]  ? filemap_map_pages+0x309/0x400
Sep  8 11:10:55 geedev kernel: [6207487.550898]  __do_fault+0x3c/0x120
Sep  8 11:10:55 geedev kernel: [6207487.551766]  do_read_fault+0xeb/0x160
Sep  8 11:10:55 geedev kernel: [6207487.552654]  do_fault+0xa0/0x2e0
Sep  8 11:10:55 geedev kernel: [6207487.553478]  handle_pte_fault+0x1cd/0x240
Sep  8 11:10:55 geedev kernel: [6207487.554380]  __handle_mm_fault+0x405/0x6f0
Sep  8 11:10:55 geedev kernel: [6207487.555293]  handle_mm_fault+0xd8/0x2c0
Sep  8 11:10:55 geedev kernel: [6207487.556162]  do_user_addr_fault+0x1c9/0x670
Sep  8 11:10:55 geedev kernel: [6207487.557073]  exc_page_fault+0x77/0x170
Sep  8 11:10:55 geedev kernel: [6207487.557924]  asm_exc_page_fault+0x27/0x30
Sep  8 11:10:55 geedev kernel: [6207487.558799] RIP: 0033:0x7f64a7d61630
Sep  8 11:10:55 geedev kernel: [6207487.559617] Code: Unable to access opcode bytes at RIP 0x7f64a7d61606.
Sep  8 11:10:55 geedev kernel: [6207487.560774] RSP: 002b:00007fff6b5d8658 EFLAGS: 00010246
Sep  8 11:10:55 geedev kernel: [6207487.561778] RAX: 000000000000002f RBX: 00005646e785b938 RCX: 6c7967656e657261
Sep  8 11:10:55 geedev kernel: [6207487.563005] RDX: 00007fff6b5d8678 RSI: 0000000000000001 RDI: 00007fff6b5d8668
Sep  8 11:10:55 geedev kernel: [6207487.564223] RBP: 0000000000000000 R08: 00005646e79db540 R09: 00000000ffffffff
Sep  8 11:10:55 geedev kernel: [6207487.565441] R10: 0000000000001000 R11: f0e7713d96dbaae0 R12: 00007fff6b5d86d0
Sep  8 11:10:55 geedev kernel: [6207487.566646] R13: 00007fff6b5d8668 R14: 00007fff6b5d8678 R15: 00007fff6b5d8687
Sep  8 11:10:55 geedev kernel: [6207487.567843]  </TASK>
Sep  8 11:10:55 geedev kernel: [6207487.568496] Mem-Info:
Sep  8 11:10:55 geedev kernel: [6207487.569102] active_anon:233042 inactive_anon:539204 isolated_anon:0

Sep  8 11:10:55 geedev kernel: [6207487.613106] [  19195]     0 19195   251381      795   131072        0             0 YDQuaraV2
Sep  8 11:10:55 geedev kernel: [6207487.613108] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-f1a19755b45b945c3c9a6f059f89fb7bafd96e4598fc580d0caa9a24eae036fb.scope,task=sshd,pid=19090,uid=9999
Sep  8 11:10:55 geedev kernel: [6207487.613137] Out of memory: Killed process 19090 (sshd) total-vm:2434348kB, anon-rss:1684344kB, file-rss:0kB, shmem-rss:0kB, UID:9999 pgtables:3836kB oom_score_adj:0
Sep  8 11:10:56 geedev systemd[1]: docker-f1a19755b45b945c3c9a6f059f89fb7bafd96e4598fc580d0caa9a24eae036fb.scope: A process of this unit has been killed by the OOM killer.
Sep  8 11:10:56 geedev containerd[90897]: time="2023-09-08T11:10:55.401878141+08:00" level=error msg="post event" error="context deadline exceeded"
Sep  8 11:10:56 geedev dockerd[439828]: time="2023-09-08T11:10:55.908201356+08:00" level=error msg="Could not send KILL signal to container process" container=ca39d45d1796cb69a3f32c064426f8ab31f051e9273a35a427968e192ebfd83c error="process already finished: not found" exec=11928effd07c183c8c05820fcf193e9b573a886c31f851b1308d8cc838a2be3c
Sep  8 11:11:01 geedev CRON[19320]: (root) CMD (flock -xn /tmp/stargate.lock -c '/usr/local/qcloud/stargate/admin/start.sh > /dev/null 2>&1 &')
Sep  8 11:12:01 geedev CRON[19567]: (root) CMD (flock -xn /tmp/stargate.lock -c '/usr/local/qcloud/stargate/admin/start.sh > /dev/null 2>&1 &')
Sep  8 11:13:01 geedev CRON[19853]: (root) CMD (flock -xn /tmp/stargate.lock -c '/usr/local/qcloud/stargate/admin/start.sh > /dev/null 2>&1 &')
Sep  8 11:13:56 geedev systemd[1]: run-docker-runtime\x2drunc-moby-ca39d45d1796cb69a3f32c064426f8ab31f051e9273a35a427968e192ebfd83c-runc.bvbtuU.mount: Deactivated successfully.

咨询了chatgpt, 还是很给力的

查看journal记录, 二进制文件需要用命令才能查看.

 journalctl --file journal/50404d34021d4b4980706fd359fde87b/system.journal


Sep 08 11:10:17 geedev kernel: traps: lushput[19164] general protection fault ip:7fa78a2c3540 sp:7ffd1ede1a18 error:0 in libc.so.6[7fa78a247000+195000]
Sep 08 11:10:54 geedev kernel: systemd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Sep 08 11:10:55 geedev kernel: CPU: 1 PID: 1 Comm: systemd Kdump: loaded Not tainted 5.15.0-72-generic #79-Ubuntu
Sep 08 11:10:55 geedev kernel: Hardware name: Tencent Cloud CVM, BIOS seabios-1.9.1-qemu-project.org 04/01/2014
Sep 08 11:10:55 geedev kernel: Call Trace:
Sep 08 11:10:55 geedev kernel:  <TASK>
Sep 08 11:10:55 geedev kernel:  show_stack+0x52/0x5c
Sep 08 11:10:55 geedev kernel:  dump_stack_lvl+0x4a/0x63
Sep 08 11:10:55 geedev kernel:  dump_stack+0x10/0x16
Sep 08 11:10:55 geedev kernel:  dump_header+0x53/0x228
Sep 08 11:10:55 geedev kernel:  oom_kill_process.cold+0xb/0x10
Sep 08 11:10:55 geedev kernel:  out_of_memory+0x106/0x2e0
Sep 08 11:10:55 geedev kernel:  __alloc_pages_slowpath.constprop.0+0x9b7/0xa80
Sep 08 11:10:55 geedev kernel:  __alloc_pages+0x311/0x330
Sep 08 11:10:55 geedev kernel:  alloc_pages+0x9e/0x1e0

ubuntu@geedev:/var/log$ journalctl --file journal/50404d34021d4b4980706fd359fde87b/user-1000.journal
Sep 08 11:23:27 geedev sudo[23483]:   ubuntu : TTY=pts/15 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/find -name *loadbit* /
Sep 08 11:23:27 geedev sudo[23483]: pam_unix(sudo:session): session opened for user root(uid=0) by ubuntu(uid=1000)
Sep 08 11:23:27 geedev sudo[23483]: pam_unix(sudo:session): session closed for user root
Sep 08 11:23:33 geedev sudo[23520]:   ubuntu : TTY=pts/15 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/find / -name *load

How to read *.journal files? [duplicate]

https://unix.stackexchange.com/questions/210150/how-to-read-journal-files

journalctl --file /path/to/some/file.jour

关联的漏洞介绍

关于漏洞 Cve-2021-4043

https://github.com/berdav/CVE-2021-4034

可以支持提权操作, 执行该漏洞文件后可以切换为root用户.

One day for the polkit privilege escalation exploit

If the exploit is working you'll get a root shell immediately:

vagrant@ubuntu-impish:~/CVE-2021-4034$ make
cc -Wall --shared -fPIC -o pwnkit.so pwnkit.c
cc -Wall    cve-2021-4034.c   -o cve-2021-4034
echo "module UTF-8// PWNKIT// pwnkit 1" > gconv-modules
mkdir -p GCONV_PATH=.
cp /usr/bin/true GCONV_PATH=./pwnkit.so:.
vagrant@ubuntu-impish:~/CVE-2021-4034$ ./cve-2021-4034
# whoami
root
# exit

https://www.cvedetails.com/cve/CVE-2021-4043/

Vulnerability Details : CVE-2021-4043

NULL Pointer Dereference in GitHub repository gpac/gpac prior to 1.1.0.

漏洞 CVE-2021-3493

CVE-2021-3493

https://github.com/briskets/CVE-2021-3493

ubuntu漏洞, 看起来可以突破文件的overlays限制, 能联想到的就是突破docker限制了.

Ubuntu OverlayFS Local Privesc, 看起来一直影响到ubuntu20版本

"Ubuntu specific issue in the overlayfs file system in the Linux kernel where it did not properly validate the application of file system capabilities with respect to user namespaces. A local attacker could use this to gain elevated privileges, due to a patch carried in Ubuntu to allow unprivileged overlayfs mounts." - Ubuntu Security

CVE-2021-3493：Overlay 文件系统 Ubuntu 本地提权漏洞分析

https://blog.csdn.net/song_lee/article/details/131452729

flink的web漏洞

其实不算是漏洞, flink web本来就支持用户执行flink jar包. 问题是flink web一般都是demo部署, 并没有密码信息, 可以上传任意jar包执行, 然后整个机器就被劫持了.

Apache-Flink 暴露外网远程代码执行预警

https://cert.360.cn/warning/detail?id=b700b168d1baf2b23d85a8835e957774

Flink作为业内流行的流式处理框架，允许用户上传代码进行执行是符合其设计逻辑的。但用户不应该把自身的 Flink 集群暴露于公网之中。由于 Flink 自身 Dashboard 默认无用户认证设计，360CERT判断该事件危害严重, 影响面广泛。

修复建议

保证 Flink 集群只能被可信任端点访问
禁止公网对内8081(Flink默认端口)的流量
配置相关的认证策略

检查所有用户的crontab文件

How do I list all cron jobs for all users?

https://stackoverflow.com/questions/134906/how-do-i-list-all-cron-jobs-for-all-users

for user in $(cut -f1 -d: /etc/passwd); do crontab -u $user -l; done

created at 2023-09-08

cve 病毒排查记录

flink web 支持上传jar包执行​

排查的日志记录​

关联的漏洞介绍​

关于漏洞 Cve-2021-4043​

漏洞 CVE-2021-3493​

flink的web漏洞​

检查所有用户的crontab文件​