Agent bakery error - cannot bake agents

rh76 · June 29, 2021, 12:06pm

Hi,
I get the following error when trying to bake agents:

2021-06-29 13:52:48,780 [30] [cmk.web.automations 3986689] ‘check_mk --automation bake-agents’ returned 'ERROR: Error on creating msi file: [Errno 2] No such file or directory: ‘/omd/sites/iuk/var/check_mk/agents/.files_cache/a03c5e4efdfd37a5/windows/checkmk.dat’, work_dir is /omd/sites/iuk/tmp/msi-update.sq06rrbd
’
2021-06-29 13:52:48,780 [40] [cmk.web.automations 3986689] Error running ‘check_mk --automation bake-agents’ (exit code 1)
2021-06-29 13:52:48,781 [40] [cmk.web.background-job 3986689] Exception in background function
Traceback (most recent call last):
File “/omd/sites/iuk/lib/python3/cmk/gui/background_job.py”, line 209, in _execute_function
func_ptr(*args, **kwargs)
File “/omd/sites/iuk/lib/python3/cmk/gui/cee/plugins/wato/agent_bakery/misc.py”, line 892, in bake_agents_background_job
watolib.check_mk_local_automation(‘bake-agents’,
File “/omd/sites/iuk/lib/python3/cmk/gui/watolib/automations.py”, line 143, in check_mk_local_automation
raise _local_automation_failure(command=command,
cmk.utils.exceptions.MKGeneralException: Error running automation call bake-agents (exit code 1), error:

ERROR: Error on creating msi file: [Errno 2] No such file or directory: ‘/omd/sites/iuk/var/check_mk/agents/.files_cache/a03c5e4efdfd37a5/windows/checkmk.dat’, work_dir is /omd/sites/iuk/tmp/msi-update.sq06rrbd

Did an update from 1.6 to 2.0.0p6 today, and now can not bake new agents

Rene

nvongeheur · June 29, 2021, 12:13pm

Have you tried to remove the agent bakery folder and recreated it?

If not, just delete /omd/sites/$CMK_SITE/var/check_mk/agents and bake all agents again via WebUI

rh76 · June 29, 2021, 12:36pm

OK, I deleted everything within agents-folder, then started bakery again. I saw that some agents were built, but then the error again:

Agent baking for all hosts started…
2021-06-29 14:34:24,143 [30] [cmk.web.automations 4061215] ‘check_mk --automation bake-agents’ returned 'ERROR: Error on creating msi file: [Errno 2] No such file or directory: ‘/omd/sites/iuk/var/check_mk/agents/.files_cache/f79c3166e5e3c014/windows/checkmk.dat’, work_dir is /omd/sites/iuk/tmp/msi-update.mm6wwqau
’
2021-06-29 14:34:24,144 [40] [cmk.web.automations 4061215] Error running ‘check_mk --automation bake-agents’ (exit code 1)
2021-06-29 14:34:24,145 [40] [cmk.web.background-job 4061215] Exception in background function
Traceback (most recent call last):
File “/omd/sites/iuk/lib/python3/cmk/gui/background_job.py”, line 209, in _execute_function
func_ptr(*args, **kwargs)
File “/omd/sites/iuk/lib/python3/cmk/gui/cee/plugins/wato/agent_bakery/misc.py”, line 892, in bake_agents_background_job
watolib.check_mk_local_automation(‘bake-agents’,
File “/omd/sites/iuk/lib/python3/cmk/gui/watolib/automations.py”, line 143, in check_mk_local_automation
raise _local_automation_failure(command=command,
cmk.utils.exceptions.MKGeneralException: Error running automation call bake-agents (exit code 1), error:
ERROR: Error on creating msi file: [Errno 2] No such file or directory: ‘/omd/sites/iuk/var/check_mk/agents/.files_cache/f79c3166e5e3c014/windows/checkmk.dat’, work_dir is /omd/sites/iuk/tmp/msi-update.mm6wwqau

Exception: Error running automation call bake-agents (exit code 1), error:
ERROR: Error on creating msi file: [Errno 2] No such file or directory: ‘/omd/sites/iuk/var/check_mk/agents/.files_cache/f79c3166e5e3c014/windows/checkmk.dat’, work_dir is /omd/sites/iuk/tmp/msi-update.mm6wwqau

nvongeheur · June 29, 2021, 12:44pm

Can you please verify that all directories are owned by your site user? Maybe the daemon just does not have the permissions to write to the agents folder or the tmp directory /omd/sites/iuk/tmp

rh76 · June 29, 2021, 1:08pm

whole directory tree is owned by iuk user (site name iuk) with at least rwx for owner, rwx or r-x for group, — for other.

rh76 · June 30, 2021, 6:55am

Tried it again, error appears in different sub-directories in .files_cache, and always when creating msi packet. Linux … agent packages seem to be OK.

thorian93 · June 30, 2021, 8:07am

What about disk space, are you good on that end?

rh76 · June 30, 2021, 8:57am

There is enough disk space.

andreas-doehler · June 30, 2021, 9:43am

What happens if you build the agents on the command line with “cmk --debug -vvA”?

rh76 · June 30, 2021, 10:58am

OMD[iuk]:~$ cmk --debug -vvA
VANILLA…linux_deb:uptodate…linux_rpm:uptodate…solaris_pkg:uptodate…windo ws_msi:uptodate…linux_tgz:uptodate…solaris_tgz:uptodate…aix_tgz:uptodate… .OK
GENERIC…linux_deb:uptodate…linux_rpm:uptodate…solaris_pkg:uptodate…windo ws_msi:uptodate…linux_tgz:uptodate…solaris_tgz:uptodate…aix_tgz:uptodate… .OK
DIGI4-IBE23…does not use agent, skipping.
VK-M09-U03-FL1-1…does not use agent, skipping.
AS5-ZD14…Found no Python 2 version for cmk_update_agent.py, deploying with Pyt hon 3 support only.linux_deb:baking…(fast repackage)…linux_rpm:baking…(fas t repackage)…solaris_pkg:baking…(fast repackage)…windows_msi:baking…(fas t repackage)…ERROR: Error on creating msi file: [Errno 2] No such file or dire ctory: ‘/omd/sites/iuk/var/check_mk/agents/.files_cache/af52b6edeffe57ef/windows /checkmk.dat’, work_dir is /omd/sites/iuk/tmp/msi-update.p5njrsen

andreas-doehler · June 30, 2021, 5:56pm

I don’t know what was changed for the “fast repackage” as this is a new output in 2.0.
If i want to be on the save side. I remove all build agents and the sym links. Then i make a clean build of all agents.
Has this host “AS5-ZD14” some special plugins configured?
Or what you can also do is also use the force switch to build all agents new.

rh76 · June 30, 2021, 8:34pm

OK, deleted everything under agents-directory.
Tried baking agents again, error still exists, on another host now. Very creepy.

OMD[iuk]:~/var/check_mk/agents$ cmk -vvAf
VANILLA…linux_deb:baking…linux_rpm:baking…(fast repackage)…solaris_pkg:baking…windows_msi:baking…linux_tgz:baking…(fast repackage)…solaris_tgz:baking…(fast repackage)…aix_tgz:baking…OK
GENERIC…linux_deb:baking…linux_rpm:baking…(fast repackage)…solaris_pkg:baking…windows_msi:baking…b’plugins/mk_inventory.vbs’ (14062)
linux_tgz:baking…(fast repackage)…solaris_tgz:baking…(fast repackage)…aix_tgz:baking…OK
CON2-VDI…Found no Python 2 version for cmk_update_agent.py, deploying with Python 3 support only.linux_deb:baking…linux_rpm:baking…(fast repackage)…solaris_pkg:baking…windows_msi:baking…linux_tgz:baking…(fast repackage)…solaris_tgz:baking…(fast repackage)…aix_tgz:baking…ERROR
TS1-DAKS…Found no Python 2 version for cmk_update_agent.py, deploying with Python 3 support only.linux_deb:linking…linux_rpm:linking…solaris_pkg:linking…windows_msi:baking…(fast repackage)…ERROR: Error on creating msi file: [Errno 2] No such file or directory: ‘/omd/sites/iuk/var/check_mk/agents/.files_cache/af52b6edeffe57ef/windows/checkmk.dat’, work_dir is /omd/sites/iuk/tmp/msi-update.z13x7fe0

andreas-doehler · July 1, 2021, 5:46am

The error is not Windows agent depended as it gives already an error for CON-VDI and AIX agent there.
It really looks like there is a space problem. Starting the baking is working and then at some point it fails.
No extra scripts installed for deployment or?

rh76 · July 1, 2021, 6:05am

Space should be OK:
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 1.9M 6.3G 1% /run
/dev/mapper/ubuntu–vg-ubuntu–lv 136G 76G 54G 59% /
tmpfs 32G 20K 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/sda2 976M 204M 706M 23% /boot
/dev/sda1 511M 7.9M 504M 2% /boot/efi
/dev/loop1 32M 32M 0 100% /snap/snapd/10707
/dev/loop2 70M 70M 0 100% /snap/lxd/19188
/dev/loop0 56M 56M 0 100% /snap/core18/1944
tmpfs 32G 32M 32G 1% /opt/omd/sites/iuk/tmp
tmpfs 6.3G 0 6.3G 0% /run/user/1000
tmpfs 32G 5.6M 32G 1% /opt/omd/sites/test/tmp
tmpfs 32G 26M 32G 1% /opt/omd/sites/iuk3/tmp
tmpfs 6.3G 0 6.3G 0% /run/user/1001

What is .files_cache used for? Temporary directory for agent building? It misses some files there.
Tried to create them with touch , like touch checkmk.dat as mentioned in error message, then I get error for another filename and so on.

andreas-doehler · July 1, 2021, 6:13am

I would first remove the “.files_cache” as it is only a cache it should be recreated.

rh76 · July 1, 2021, 6:21am

No success with that:
VANILLA…linux_deb:baking…linux_rpm:baking…(fast repackage)…solaris_pkg:baking…windows_msi:baking…linux_tgz:baking…(fast repackage)…solaris_tgz:baking…(fast repackage)…aix_tgz:baking…OK
GENERIC…linux_deb:baking…linux_rpm:baking…(fast repackage)…solaris_pkg:baking…windows_msi:baking…b’plugins/mk_inventory.vbs’ (14062)
linux_tgz:baking…(fast repackage)…solaris_tgz:baking…(fast repackage)…aix_tgz:baking…OK
DIGI3-IBE23…does not use agent, skipping.
AS6-PACS…Found no Python 2 version for cmk_update_agent.py, deploying with Python 3 support only.linux_deb:baking…linux_rpm:baking…(fast repackage)…solaris_pkg:baking…windows_msi:baking…linux_tgz:baking…(fast repackage)…solaris_tgz:baking…(fast repackage)…aix_tgz:baking…ERROR
TICKET1…Found no Python 2 version for cmk_update_agent.py, deploying with Python 3 support only.linux_deb:linking…linux_rpm:linking…solaris_pkg:linking…windows_msi:baking…(fast repackage)…ERROR: Error on creating msi file: [Errno 2] No such file or directory: ‘/omd/sites/iuk/var/check_mk/agents/.files_cache/af52b6edeffe57ef/windows/checkmk.dat’, work_dir is /omd/sites/iuk/tmp/msi-update.l6asw938

erik · July 6, 2021, 11:59am

@rh76
Have you got any custom plugins installed on your CheckMK site?
I recently experienced something similar after updating to 2.0 and found that I had to make adjustments to the custom plugins to get it fixed.

I had to update the bakery files to support the new CMK version… and I had to move the plugin files to a different location.

thorian93 · July 7, 2021, 11:11am

Ideally you can do that through an updated MKP file. But that is a good point nonetheless.

erik · July 7, 2021, 1:52pm

Yes correct, that’s what I did after I changed the files in my testing environment I created a new MKP and updated the production environment.
The baking errors similar to the ones described here we’re resolved after doing this.

system · July 7, 2022, 1:52pm

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.