Squidclient with "reverse proxy only squid"

cmkadminconc-kwe · May 16, 2024, 8:07am

CMK version: 2.2.0p23
OS version: Ubuntu 22.04.4 LTS

Error message: Item not found in monitoring data

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

Checkmk version 2.2.0p23
+ FETCHING DATA
  Source: SourceInfo(hostname='pseudonymised_hostname_squid', ipaddress='192.168.111.44', ident='agent', fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7fcd686e9690]
Read from cache: AgentFileCache(pseudonymised_hostname_squid, path_template=/omd/sites/conceto/tmp/check_mk/cache/{hostname}, max_age=MaxAge(checking=0, discovery=90.0, inventory=90.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 7 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 192.168.111.44:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.TLS (b'16')
Reading data from agent via TLS socket
Reading data from agent
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Closing TCP connection to 192.168.111.44:6556
Write data to cache file /omd/sites/conceto/tmp/check_mk/cache/pseudonymised_hostname_squid
Trying to acquire lock on /omd/sites/conceto/tmp/check_mk/cache/pseudonymised_hostname_squid
Got lock on /omd/sites/conceto/tmp/check_mk/cache/pseudonymised_hostname_squid
Releasing lock on /omd/sites/conceto/tmp/check_mk/cache/pseudonymised_hostname_squid
Released lock on /omd/sites/conceto/tmp/check_mk/cache/pseudonymised_hostname_squid
[cpu_tracking] Stop [7fcd686e9690 - Snapshot(process=posix.times_result(user=0.010000000000000231, system=0.0, children_user=0.0, children_system=0.0, elapsed=1.3100000023841858))]
  Source: SourceInfo(hostname='pseudonymised_hostname_squid', ipaddress='192.168.111.44', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7fcd690d1290]
Read from cache: NoCache(pseudonymised_hostname_squid, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
[PiggybackFetcher] Execute data source
Piggyback file '/omd/sites/conceto/tmp/check_mk/piggyback/pseudonymised_hostname_squid/pseudonymised_hostname_vsphere': Successfully processed from source 'pseudonymised_hostname_vsphere'
Piggyback file '/omd/sites/conceto/tmp/check_mk/piggyback/pseudonymised_hostname_squid/pseudonymised_hostname_veeam': Successfully processed from source 'pseudonymised_hostname_veeam'
No piggyback files for '192.168.111.44'. Skip processing.
[cpu_tracking] Stop [7fcd690d1290 - Snapshot(process=posix.times_result(user=0.0, system=0.009999999999999953, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<wmi_cpuload:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<df:sep(9)>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<winperf_phydisk>>> / Transition HostSectionParser -> HostSectionParser
<<<winperf_if>>> / Transition HostSectionParser -> HostSectionParser
<<<winperf_processor>>> / Transition HostSectionParser -> HostSectionParser
<<<fileinfo:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<logwatch>>> / Transition HostSectionParser -> HostSectionParser
<<<services>>> / Transition HostSectionParser -> HostSectionParser
<<<dotnet_clrmemory:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_win:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<ps:sep(9)>>> / Transition HostSectionParser -> HostSectionParser
Transition HostSectionParser -> NOOPParser
Transition NOOPParser -> NOOPParser
<<<systemtime>>> / Transition NOOPParser -> HostSectionParser
<<<esx_vsphere_vm:cached(1715846125,90)>>> / Transition NOOPParser -> HostSectionParser
<<<veeam_client:cached(1715846081,90):sep(9)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
  HostKey(hostname='pseudonymised_hostname_squid', source_type=<SourceType.HOST: 1>)  -> Add sections: ['check_mk', 'checkmk_agent_plugins_win', 'cmk_agent_ctl_status', 'df', 'dotnet_clrmemory', 'fileinfo', 'logwatch', 'mem', 'ps', 'services', 'systemtime', 'uptime', 'winperf_if', 'winperf_phydisk', 'winperf_processor', 'wmi_cpuload']
  HostKey(hostname='pseudonymised_hostname_squid', source_type=<SourceType.HOST: 1>)  -> Add sections: ['esx_vsphere_vm', 'labels', 'veeam_client']
Received no piggyback data
[cpu_tracking] Start [7fcd68739a50]
value store: synchronizing
Trying to acquire lock on /omd/sites/conceto/tmp/check_mk/counters/pseudonymised_hostname_squid
Got lock on /omd/sites/conceto/tmp/check_mk/counters/pseudonymised_hostname_squid
value store: loading from disk
Releasing lock on /omd/sites/conceto/tmp/check_mk/counters/pseudonymised_hostname_squid
Released lock on /omd/sites/conceto/tmp/check_mk/counters/pseudonymised_hostname_squid
CPU utilization      Total CPU: 7.73%
Check_MK Agent       Version: 2.2.0p23, OS: windows, Agent plugins: 0, Local checks: 0
Disk IO SUMMARY      Read: 0.00 B/s, Write: 69.8 kB/s, Latency: 668 microseconds
DotNet Memory Management _Global_ Time in GC: 0%
ESX CPU              demand is 0.062 Ghz, 2 virtual CPUs
ESX Datastores       Stored on DS-03-02 (3.64 TiB/49.1% free)
ESX Guest Tools      VMware Tools are installed and the version is current
ESX Heartbeat        Heartbeat status is green
ESX Hostsystem       Running on 192.168.111.61
ESX Memory           Host: 16.1 GiB, Guest: 327 MiB, Ballooned: 0 B, Private: 16.0 GiB, Shared: 10.0 MiB
ESX Mounted Devices  HA functionality guaranteed
ESX Snapshots        Count: 0
Filesystem C:/       Used: 13.66% - 21.8 GiB of 160 GiB, trend per 1 day 0 hours: +190 MiB, trend per 1 day 0 hours: +0.12%, Time left until disk full: 2 years 13 days
Filesystem E:/       Used: 14.07% - 5.63 GiB of 40.0 GiB (warn/crit at 82.59%/91.29% used), trend per 1 day 0 hours: -164 KiB, trend per 1 day 0 hours: -0.00%
Interface Intel[R] 82574L Gigabit Network Connection [1], (Connected), Speed: 1 GBit/s, In: 1.61 kB/s (<0.01%), Out: 1.43 kB/s (<0.01%)
Memory               RAM: 23.50% - 3.76 GiB of 16.0 GiB, Commit charge: 18.31% - 3.36 GiB of 18.4 GiB
Processor Queue      15 min load: 0.01, 15 min load per core: 0.01 (2 logical cores)
Service Summary      Autostart services: 64, Stopped services: 5
System Time          Offset: -180 milliseconds
Uptime               Up since Feb 06 2024 00:26:20, Uptime: 100 days 7 hours
VEEAM Client Backup_Job_Standard_Server Status: Success, Size (total/read/transferred): 200 GiB/ 3.75 GiB/ 1.86 GiB, Last backup: 10 hours 58 minutes ago, Duration: 4 minutes 18 seconds, Average Speed: 37.6 MB/s, Backup server: pseudonymised_hostname_veeam
Piggyback file '/omd/sites/conceto/tmp/check_mk/piggyback/pseudonymised_hostname_squid/pseudonymised_hostname_vsphere': Successfully processed from source 'pseudonymised_hostname_vsphere'
Piggyback file '/omd/sites/conceto/tmp/check_mk/piggyback/pseudonymised_hostname_squid/pseudonymised_hostname_veeam': Successfully processed from source 'pseudonymised_hostname_veeam'
No piggyback files for '192.168.111.44'. Skip processing.
[cpu_tracking] Stop [7fcd68739a50 - Snapshot(process=posix.times_result(user=0.029999999999999805, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.03999999910593033))]
[agent] Success, [piggyback] Successfully processed from source 'pseudonymised_hostname_vsphere', Successfully processed from source 'pseudonymised_hostname_veeam', execution time 1.4 sec | execution_time=1.350 user_time=0.040 system_time=0.010 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=1.290

i recplaced IPs and hostname to pseudo
** END**

I wanted to check our squid with this plugin: check_mk/plugins/squid at master · allangood/check_mk · GitHub
but the only result i got is the mentiond error message.

After first coping die plugin to /usr/lib/check_mk_agent/plugins, I noticed that i have to install squidclient. I installed it via apt, but after doing some research here i think, i can only use it for ‘normal’ proxy, but not reverse proxy, am i right? Or did i oversee something?
If there are any other hints to check reverse proxy with checkmk, i am equally grateful too.

aeckstein · May 16, 2024, 10:26am

Hi,

what happens when you run the command from the agent plugin manually ?

cmkadminconc-kwe · May 16, 2024, 11:13am

Hello,
thank you for your fast help.
i tried your command, but variable port was empty, so i reduced to ss -tlpn and got this:

State     Recv-Q    Send-Q    Local Address:Port       Peer Address:Port     Process
LISTEN    0         4096            0.0.0.0:111             0.0.0.0:*
LISTEN    0         4096      127.0.0.53%lo:53              0.0.0.0:*
LISTEN    0         128             0.0.0.0:22              0.0.0.0:*
LISTEN    0         256                   *:443                   *:*
LISTEN    0         4096                  *:6556                  *:*
LISTEN    0         256                   *:3299                  *:*
LISTEN    0         256                   *:44300                 *:*
LISTEN    0         4096               [::]:111                [::]:*
LISTEN    0         256                   *:50000                 *:*
LISTEN    0         128                [::]:22                 [::]:*

443, 3299, 44300, 50000 are the ports on which squid is listening

ss | grep squid returns empty

nevermind, i tried squidclient -p 50000 -T2 mgr:5min | grep = and the other ports, but alle returns nothing

btw. i didn’t found in man grep the meaning for the equals behind grep

one additional info: squid was not installed via apt, but self compiled and installed

aeckstein · May 16, 2024, 12:50pm

Is the squid server really listining to all of these ports ?

ss -tulpn | grep squid | tr -s " " | cut -d " " -f 4 | cut -d “:” -f 2

But you tried to enter the port manually and this should at least work.
The grep = is just outputting the lines from the sqiudclient output containing the “=” sign.

cmkadminconc-kwe · May 16, 2024, 3:49pm

Is the squid server really listining to all of these ports ?

yes

The grep = is just outputting the lines from the sqiudclient output containing the “=” sign.

Oh, sure, it’s just the pattern … that was to easy … i’m so stupid

ss -tulpn | grep apache | tr -s " " | cut -d " " -f 4 | cut -d “:” -f 2
sorry, I am grateful for the help, but i didn’t get, what we want to discover, because with option -n we have numbered ports, not named, so there should be no ‘squid’ (apache isn’t used here) for grep to match?!

aeckstein · May 16, 2024, 4:00pm

Copy and paste error, please exchange Apache with squid…

aeckstein · May 16, 2024, 4:28pm

At first you need to get the squidclient application to output the data of your squid proxy, we can manually enter the port in the check plugin if we can not find it that way.

But at least -tulpn should work if the squid is running as the user squid.

thl-cmk · May 16, 2024, 7:41pm

@cmkadminconc-kwe as @aeckstein said, you need to get squidclient produce the requiered output. I played a bit with it my self.

Here the output from ss -tlpn, keep in mind to run it as root, to get the Process name.

vsquid:~$ sudo ss -tlpn
[sudo] password for thl: 
State          Recv-Q         Send-Q                 Local Address:Port                   Peer Address:Port         Process                                     
LISTEN         0              128                          0.0.0.0:22                          0.0.0.0:*             users:(("sshd",pid=479,fd=3))              
LISTEN         0              128                             [::]:22                             [::]:*             users:(("sshd",pid=479,fd=4))              
LISTEN         0              256                                *:88                                *:*             users:(("squid",pid=273247,fd=11))         
LISTEN         0              4096                               *:6556                              *:*             users:(("cmk-agent-ctl",pid=1018,fd=9))    
vsquid:~$

here the complete command to get the squid port from the plugin

vsquid:~$ sudo ss -tlpn | grep squid | tr -s ' ' | cut -d' ' -f 4 | cut -d':' -f 2
88

As you can see squid is running on port 88. Port 88 is configured as a very simple reverse proxy

http_port 88 accel defaultsite=mysite.test.internal no-vhost
cache_peer backend.test.internal parent 80 0 no-query originserver name=test

all other listenig ports are commented out

# Squid normally listens to port 3128
# http_port 3128

now try to get the infos with squidclient

vsquid:~$ squidclient -p 88 -T2 mgr:5min
HTTP/1.1 200 OK
Server: squid/4.13
Mime-Version: 1.0
Date: Thu, 16 May 2024 19:28:25 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 16 May 2024 19:28:25 GMT
Last-Modified: Thu, 16 May 2024 19:28:25 GMT
X-Cache: MISS from vsquid
X-Cache-Lookup: MISS from vsquid:88
Via: 1.1 vsquid (squid/4.13)
Connection: close

sample_start_time = 1715887404.797459 (Thu, 16 May 2024 19:23:24 GMT)
sample_end_time = 1715887704.807290 (Thu, 16 May 2024 19:28:24 GMT)
client_http.requests = 0.000000/sec
client_http.hits = 0.000000/sec
client_http.errors = 0.000000/sec
client_http.kbytes_in = 0.000000/sec
client_http.kbytes_out = 0.000000/sec
client_http.all_median_svc_time = 0.000000 seconds
client_http.miss_median_svc_time = 0.000000 seconds
client_http.nm_median_svc_time = 0.000000 seconds
client_http.nh_median_svc_time = 0.000000 seconds
client_http.hit_median_svc_time = 0.000000 seconds
server.all.requests = 0.003333/sec
server.all.errors = 0.000000/sec
server.all.kbytes_in = 0.000000/sec
server.all.kbytes_out = 0.003333/sec
server.http.requests = 0.003333/sec
server.http.errors = 0.000000/sec
server.http.kbytes_in = 0.000000/sec
server.http.kbytes_out = 0.003333/sec
server.ftp.requests = 0.000000/sec
server.ftp.errors = 0.000000/sec
server.ftp.kbytes_in = 0.000000/sec
server.ftp.kbytes_out = 0.000000/sec
server.other.requests = 0.000000/sec
server.other.errors = 0.000000/sec
server.other.kbytes_in = 0.000000/sec
server.other.kbytes_out = 0.000000/sec
icp.pkts_sent = 0.000000/sec
icp.pkts_recv = 0.000000/sec
icp.queries_sent = 0.000000/sec
icp.replies_sent = 0.000000/sec
icp.queries_recv = 0.000000/sec
icp.replies_recv = 0.000000/sec
icp.replies_queued = 0.000000/sec
icp.query_timeouts = 0.000000/sec
icp.kbytes_sent = 0.000000/sec
icp.kbytes_recv = 0.000000/sec
icp.q_kbytes_sent = 0.000000/sec
icp.r_kbytes_sent = 0.000000/sec
icp.q_kbytes_recv = 0.000000/sec
icp.r_kbytes_recv = 0.000000/sec
icp.query_median_svc_time = 0.000000 seconds
icp.reply_median_svc_time = 0.000000 seconds
dns.median_svc_time = 0.000000 seconds
unlink.requests = 0.000000/sec
page_faults = 0.000000/sec
select_loops = 2.379922/sec
select_fds = 0.096663/sec
average_select_fd_period = 0.000000/fd
median_select_fds = -1.000000
swap.outs = 0.000000/sec
swap.ins = 0.000000/sec
swap.files_cleaned = 0.000000/sec
aborted_requests = 0.003333/sec
syscalls.disk.opens = 0.000000/sec
syscalls.disk.closes = 0.000000/sec
syscalls.disk.reads = 0.000000/sec
syscalls.disk.writes = 0.000000/sec
syscalls.disk.seeks = 0.000000/sec
syscalls.disk.unlinks = 0.000000/sec
syscalls.sock.accepts = 0.000000/sec
syscalls.sock.sockets = 0.003333/sec
syscalls.sock.connects = 0.003333/sec
syscalls.sock.binds = 0.000000/sec
syscalls.sock.closes = 0.003333/sec
syscalls.sock.reads = 0.003333/sec
syscalls.sock.writes = 0.003333/sec
syscalls.sock.recvfroms = 0.086664/sec
syscalls.sock.sendtos = 0.000000/sec
cpu_time = 0.221548 seconds
wall_time = 300.009831 seconds
cpu_usage = 0.073847%

btw. there is a newer version of the squid plugin. This is based on the one you are using, but rewritten for CMK 2.2.

Cheers
Thomas

cmkadminconc-kwe · May 17, 2024, 9:45am

many thanks for the hint to newer Version.

I am on step further. We don’t have http ports, these are all https. I red man again and created this command:
squidclient -h FQDN -v --https cert=/path/to/cert --trusted=/path/to/cert -p 443

and got this response:

GnuTLS: There was a non-CA certificate in the trusted list: CN=*.domain.de

Request:
GET 443 HTTP/1.0
User-Agent: squidclient/4.10
Accept: */*
Connection: close


.
VERIFY DATUM: The certificate is trusted.
SUCCESS: CA verified Encrypted Connection
WARNING: Resource temporarily unavailable, try again.

when i have more time, i will think about just open http port for suidclient to use, that should be much easier…