Network scanning not working at all on fresh install

Troubleshooting
1

CMK version: Check_MK version 2.2.0p25 CCE
OS version: Debian Bookworm.

Screenshot 2024-04-21 at 9.51.49 PM
Screenshot 2024-04-21 at 9.51.49 PM926×1293 104 KB

Error message:
No cronjobs shown ; no logs seen indicating a scan executed… curl requests to cronjob.py claim ran too early.

All I want is for my new checkmk installation to go out there on 192.168.10.0/24 and find any host device that responds to pings. Put them on my folder list and let me decide what to do with it.

The “wato” file does not seem to be accepting any of my network scanning options I set in the GUI.

I did try creating a new site and the same bug happens…

https://checkmk.atlassian.net/wiki/spaces/KB/pages/9472703/How-to+force+a+network+scan

I do not see any 'network_scan_result': {}, as the documentation claims should be there?

root@checkmk:/opt/omd/sites/homemon# cat ./etc/check_mk/conf.d/wato/trusted/rules.mk
# Written by Checkmk store


globals().setdefault('cmc_smartping_settings', [])

cmc_smartping_settings = [
{'id': 'bf0ae88f-e307-4436-8933-4e4aa7ac3543', 'value': {'timeout': 2.5}, 'condition': {'host_folder': '/%s/' % FOLDER_PATH}, 'options': {'disabled': False, 'description': 'Ping checks TRUSTED folder'}},
] + cmc_smartping_settings


OMD[homemon]:~$ . $OMD_ROOT/etc/omd/site.conf ; curl http://localhost:$CONFIG_APACHE_TCP_PORT/homemon/check_mk/run_cron.py
Cron called too early. Skipping.
OMD[homemon]:~$

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

2

This is normal and expected output if you don’t stop the cron service inside the site.
I tested in one of my test container enviroments and there the scan was not starting automatically but if i executed the cron job manually it was running fine and found some hosts.
It’s a little bit strange.

3

Its’s not a bug. In my test environment Ubuntu 22.04,Rocky9 the scan works with a fresh 2.2.0p25. I have not tried this on a Debain 12 yet.
You can verify this by just doing a simple fping as site user (fping has to be installed and you it will tell you which IP is alive and unreachable):

fping -g 192.168.10.0/24

4

I can only say it is not working.
Site and folder created.

OMD[cmk]:~/etc/check_mk/conf.d/wato/ipscan$ ls -lisa
total 16
3275532 4 drwx------ 2 cmk cmk 4096 Apr 22 07:53 ./
3254731 4 drwxrwx--- 1 cmk cmk 4096 Apr 22 07:53 ../
3275547 4 -rw-rw---- 1 cmk cmk  500 Apr 22 07:53 .wato
3275548 4 -rw-rw---- 1 cmk cmk  495 Apr 22 07:53 .wato.pkl

Now we see here it is

OMD[cmk]:~/etc/check_mk/conf.d/wato/ipscan$ date
Mon Apr 22 09:05:59 UTC 2024

Scan interval is set to one hour.
Inside “.wato” you see no scan until now.

OMD[cmk]:~/etc/check_mk/conf.d/wato/ipscan$ cat .wato
{'title': 'IPScan', 'attributes': {'meta_data': {'created_at': 1713772395.9705136, 'updated_at': 1713772395.9710333, 'created_by': 'cmkadmin'}, 'network_scan': {'ip_ranges': [('ip_network', ('192.168.188.0', 24))], 'exclude_ranges': [], 'scan_interval': 3600, 'time_allowed': [((0, 0), (24, 0))], 'set_ipaddress': True, 'tag_criticality': 'offline', 'run_as': 'cmkadmin'}, 'tag_agent': 'no-agent'}, 'num_hosts': 0, 'lock': False, 'lock_subfolders': False, '__id': '5011ae51ec524680909bf44a65315188'}

If i now start the cronjob manually then it will show a scan result.

OMD[cmk]:~/etc/check_mk/conf.d/wato/ipscan$ ls -lisa
total 16
3275532 4 drwx------ 2 cmk cmk 4096 Apr 22 07:53 ./
3254731 4 drwxrwx--- 1 cmk cmk 4096 Apr 22 07:53 ../
3275547 4 -rw-rw---- 1 cmk cmk  500 Apr 22 07:53 .wato
3275548 4 -rw-rw---- 1 cmk cmk  495 Apr 22 07:53 .wato.pkl
OMD[cmk]:~/etc/check_mk/conf.d/wato/ipscan$ . ~/etc/omd/site.conf ; curl http://localhost:5000/cmk/check_mk/run_cron.py
OK
OMD[cmk]:~/etc/check_mk/conf.d/wato/ipscan$ ls -lisa
total 48
3275532  4 drwx------ 2 cmk cmk  4096 Apr 22 09:51 ./
3254731  4 drwxrwx--- 1 cmk cmk  4096 Apr 22 07:53 ../
3274413  4 -rw-rw---- 1 cmk cmk   646 Apr 22 09:51 .wato
3274454  4 -rw-rw---- 1 cmk cmk   610 Apr 22 09:51 .wato.pkl
3274515 24 -rw-rw---- 1 cmk cmk 22029 Apr 22 09:51 hosts.mk
3274533  8 -rw-rw---- 1 cmk cmk  5568 Apr 22 09:51 hosts.pkl

From my perspective it looks more like the cron is not correctly executed inside the container environment, as i see no entry inside the site Apache log.

As i use only the container environment for dev and testing it is not relevant for my systems if cron is running or not. But if you use the container for production systems i think it makes a difference.

5

I created a fresh site with a docker container:

sudo docker container run -dit -p 8081:5000 -p 8000:8010 --tmpfs /opt/omd/sites/cmk/tmp:uid=1000,gid=1000 --name monitoring -v /etc/localtime:/etc/localtime:ro --restart always registry.checkmk.com/enterprise/check-mk-enterprise:2.2.0p25

Configured a network scan on a folder and saw the hosts being added in the respective folder.

OMD[cmk]:~$ cat ~/etc/check_mk/conf.d/wato/scanfolder/.wato
{'title': 'scanfolder', 'attributes': {'meta_data': {'created_at': 1713788334.8909056, 'updated_at': 1713788347.847429, 'created_by': 'cmkadmin'}, 'network_scan': {'ip_ranges': [('ip_network', ('172.17.0.0', 24))], 'exclude_ranges': [], 'scan_interval': 3600, 'time_allowed': [((0, 0), (24, 0))], 'set_ipaddress': True, 'tag_criticality': 'offline', 'run_as': 'cmkadmin'}, 'network_scan_result': {'start': 1713788341.650651, 'end': 1713788347.8459663, 'state': True, 'output': 'The network scan found 3 new hosts.'}}, 'num_hosts': 3, 'lock': False, 'lock_subfolders': False, '__id': 'eed872c3679c4ee28355500c38757f16'}
OMD[cmk]:~$ ls -lisa ~/etc/check_mk/conf.d/wato/scanfolder/
total 24
6799473 4 drwx------ 2 cmk cmk 4096 Apr 22 14:19 ./
6628031 4 drwxrwx--- 3 cmk cmk 4096 Apr 22 14:18 ../
6754594 4 -rw-rw---- 1 cmk cmk  619 Apr 22 14:19 .wato
6754698 4 -rw-rw---- 1 cmk cmk  587 Apr 22 14:19 .wato.pkl
6754773 4 -rw-rw---- 1 cmk cmk 1709 Apr 22 14:19 hosts.mk
6754774 4 -rw-rw---- 1 cmk cmk  880 Apr 22 14:19 hosts.pkl
OMD[cmk]:~$

The cron’s are also being executed:

OMD[cmk]:~$ ls -lisa ~/etc/check_mk/conf.d/wato/scanfolder/
total 24
6799473 4 drwx------ 2 cmk cmk 4096 Apr 22 14:19 ./
6628031 4 drwxrwx--- 3 cmk cmk 4096 Apr 22 14:18 ../
6754594 4 -rw-rw---- 1 cmk cmk  619 Apr 22 14:19 .wato
6754698 4 -rw-rw---- 1 cmk cmk  587 Apr 22 14:19 .wato.pkl
6754773 4 -rw-rw---- 1 cmk cmk 1709 Apr 22 14:19 hosts.mk
6754774 4 -rw-rw---- 1 cmk cmk  880 Apr 22 14:19 hosts.pkl
OMD[cmk]:~$ clear^C
OMD[cmk]:~$ cat ~/var/log/apache/access_log |grep -i cron
- - - [22/Apr/2024:14:05:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:06:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:07:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:08:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:09:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:10:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:11:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:12:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:13:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:14:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:15:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:16:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:17:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:18:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:19:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:20:02 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [22/Apr/2024:14:21:01 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
OMD[cmk]:~$

Also, from the web.log (raising the log_level to Debug shows the following):

OMD[cmk]:~$ cat ~/var/log/web.log |grep execute_network_scan_job
2024-04-22 14:19:01,641 [10] [cmk.web 1466] Starting [execute_network_scan_job]
2024-04-22 14:19:07,856 [10] [cmk.web 1466] Finished [execute_network_scan_job]
2024-04-22 14:20:02,198 [10] [cmk.web 1201] Starting [execute_network_scan_job]
2024-04-22 14:20:02,251 [10] [cmk.web 1201] Finished [execute_network_scan_job]
2024-04-22 14:21:01,618 [10] [cmk.web 4979] Starting [execute_network_scan_job]
2024-04-22 14:21:01,624 [10] [cmk.web 4979] Finished [execute_network_scan_job]
2024-04-22 14:22:01,858 [10] [cmk.web 1466] Starting [execute_network_scan_job]
2024-04-22 14:22:01,862 [10] [cmk.web 1466] Finished [execute_network_scan_job]
2024-04-22 14:23:02,013 [10] [cmk.web 1204] Starting [execute_network_scan_job]
2024-04-22 14:23:02,018 [10] [cmk.web 1204] Finished [execute_network_scan_job]
2024-04-22 14:24:01,199 [10] [cmk.web 915] Starting [execute_network_scan_job]
2024-04-22 14:24:01,203 [10] [cmk.web 915] Finished [execute_network_scan_job]
2024-04-22 14:25:01,394 [10] [cmk.web 1201] Starting [execute_network_scan_job]
2024-04-22 14:25:01,399 [10] [cmk.web 1201] Finished [execute_network_scan_job]
2024-04-22 14:26:01,258 [10] [cmk.web 4979] Starting [execute_network_scan_job]
2024-04-22 14:26:01,264 [10] [cmk.web 4979] Finished [execute_network_scan_job]
2024-04-22 14:27:01,477 [10] [cmk.web 1466] Starting [execute_network_scan_job]
2024-04-22 14:27:01,483 [10] [cmk.web 1466] Finished [execute_network_scan_job]
6

It is not directly a problem with network scan more with crontab not executing the

# Run Multisite regular jobs, e.g. scheduled reports
* * * * * . $OMD_ROOT/etc/omd/site.conf ; curl http://localhost:$CONFIG_APACHE_TCP_PORT/cmk/check_mk/run_cron.py >/dev/null 2>&1

correctly.

7

I don’t use docker… LXC debian 12. Does fping need be inside omd yes? binary doesnt exist. That’s the bug?

root@checkmk:/opt/omd/sites/homemon/etc/check_mk/conf.d/wato# omd su homemon
OMD[homemon]:~$ fping
-bash: fping: command not found
OMD[homemon]:~$
8

I don’t think so - also the normal container image has no fping included.

9

fping is just a command line utility to check whether any of your hosts are alive in your subnet or not. I just mentioned it as an example.

You have to install it manually on your Linux server.

10

Ah okay. For me the network scan works without any problems. Regarding the crontab, do you mean the execution of the curl command in general?

11

I’m not sure what the action item for me is to resolve this issue? If it works for you and if you are not using Debian Bookworm then isn’t that a bug in the version I am running?

12

Screenshot 2024-04-24 at 2.54.53 AM
Screenshot 2024-04-24 at 2.54.53 AM1388×1179 221 KB

Running the docker container on my localhost mac mini - network scans on new install / site works just fine.

Same configuration in a LXC container (Debian 11 or Debian 12) both fail to discover any device… help? this is a bug and not sure what to do.

13

As a test, I created a fresh Debian 12 VM, installed Checkmk 2.2.0p25.cce and configured a network scan on the folder and it definitely added 11 hosts which were available in the subnet.
On the same VM, I installed “fping” via “apt install fping” and then did

fping -g mysubnet |grep alive|wc -l

and this number matched the number of hosts created in the folder.
Maybe you have some firewall or Proxy configured in your environment?

14

Looks like the root of the issue under Proxmox LXC containers related to running the OS as unprivileged.

Running the LXC container as root seems to fix the network scanning in Debian 11 and Debian 12 test installs I made.

1 Like
15

Glad to hear that you were able to find the problem with your setup.