SYSTEM WARNING: 'date_default_timezone_get(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone.' in '/usr/share/mantis/www/core.php' line 264

0002681: Dashboard stats have wrong values - MantisBT
MantisBT - Endian Firewall
View Issue Details
0002681Endian FirewallGUIpublic2010-02-13 10:402010-11-22 12:08
baldy 
peter-endian 
highmajoralways
closedfixed 
2.3 
2.3.12.4.1 
0002681: Dashboard stats have wrong values
Statistics for the pop3 proxy in the dashboard are way too high.

Stats show for the last hour around 2000 received mails and for Today around 4000.

In reality there should only be about 5 per hour and max 50 per day.

purple
related to 0002785closed peter-endian collectd unixsocket sometimes refueses connections 
Issue History
2010-02-13 10:40baldyNew Issue
2010-02-15 09:03luca-endianStatusnew => confirmed
2010-02-15 14:20luca-endianNote Added: 0003792
2010-02-15 14:23luca-endianSummaryPOP3 proxy stats way too high => Dashboard stats have wrong values
2010-02-15 14:23luca-endianTag Attached: purple
2010-02-18 20:44baldyNote Added: 0003818
2010-02-18 20:47baldyNote Edited: 0003818
2010-02-19 17:53baldyNote Edited: 0003818
2010-03-03 16:01ra-endianAssigned To => ra-endian
2010-03-03 16:01ra-endianStatusconfirmed => assigned
2010-03-03 16:01ra-endianTarget Version => future
2010-03-04 07:56ra-endianPrioritynormal => high
2010-03-04 07:56ra-endianTarget Versionfuture => 2.3.1
2010-03-15 15:06ra-endianSeveritytweak => major
2010-03-16 08:56ra-endianNote Added: 0004039
2010-03-16 10:09luca-endianNote Added: 0004043
2010-03-18 08:07ra-endianStatusassigned => new
2010-03-18 08:07ra-endianAssigned Tora-endian => peter-endian
2010-03-18 08:07ra-endianStatusnew => confirmed
2010-03-18 17:14peter-endianNote Added: 0004067
2010-03-18 17:19peter-endianNote Added: 0004068
2010-03-18 19:29peter-endianNote Added: 0004069
2010-03-18 19:32peter-endianNote Added: 0004070
2010-03-19 17:11peter-endianNote Added: 0004073
2010-03-19 20:42peter-endianNote Added: 0004074
2010-03-22 15:37peter-endianNote Added: 0004077
2010-03-22 15:38peter-endianNote Added: 0004078
2010-03-22 15:38peter-endianRelationship addedrelated to 0002785
2010-03-22 16:09peter-endianNote Added: 0004079
2010-03-22 22:37peter-endianNote Added: 0004082
2010-03-22 23:10peter-endianNote Added: 0004083
2010-03-22 23:43peter-endianNote Added: 0004084
2010-03-23 20:29peter-endianNote Added: 0004089
2010-03-26 15:14peter-endianStatusconfirmed => resolved
2010-03-26 15:14peter-endianFixed in Version => 2.3.1
2010-03-26 15:14peter-endianResolutionopen => fixed
2010-03-26 15:15peter-endianNote Added: 0004101
2010-11-22 12:08peter-endianFixed in Version2.3.1 => 2.4.1
2010-11-22 12:08peter-endianStatusresolved => closed

Notes
(0003792)
luca-endian   
2010-02-15 14:20   
Same for smtp proxy
(0003818)
baldy   
2010-02-18 20:44   
(edited on: 2010-02-19 17:53)
Found similar issue in memory usage.

Dashboard shows 26% RAM in use, on the status page shows 38% RAM in use.

Looks like the dashboard shows the +/- buffers/cache value instead of actual ram usage.

Similar to http://bugs.endian.com/view.php?id=2195 [^]

Regards,

Klaas-Jan

(0004039)
ra-endian   
2010-03-16 08:56   
The pop counter on the dashboard are counting connections not e-mails.

logfile:
Mar 15 16:17:25 efw-v2 p3scan[27016]: POP3 Connection from 10.0.0.4:19120
Mar 15 16:17:25 efw-v2 p3scan[27016]: Real-server address is 85.25.144.249:110
Mar 15 16:17:25 efw-v2 p3scan[27016]: USER 'userxy'
Mar 15 16:17:27 efw-v2 p3scan[27016]: Session done (Clean Exit). Mails: 0 Bytes: 0

configuration:
/etc/collectd.d/p3scan.conf
<Plugin "tail">
  <File "/var/log/messages">
    Instance "pop"
    Missingok "on"
    <Match>
      Regex "POP3 Connection from "
      DSType "CounterInc"
      Type "connections"
      Instance "request"
    </Match>
(0004043)
luca-endian   
2010-03-16 10:09   
The values however seem too high even for counting the connections
(0004067)
peter-endian   
2010-03-18 17:14   
well, the memory usage percentage on dashboard ans status.cgi differs because dashboard uses memory usage - cache/buffers, which is what really counts.

status.cgi shows both percentages. we should remove that on status.cgi, but that page will disappear anyway in future, so unsure if we really should touch that.
(0004068)
peter-endian   
2010-03-18 17:19   
http proxy hit/miss will be double counted, since the connection passes twice
(squid -> havp -> dansguardian -> squid) which will double-log
(0004069)
peter-endian   
2010-03-18 19:29   
HTTP Proxy 'virus found' counter appears twice, while filter-counter does not show up.
-> Wrong label and will be read out from wrong rrd file
(0004070)
peter-endian   
2010-03-18 19:32   
POP Proxy:
p3scan does not log whether a virus or spam has been found or not, so we can eliminate those counters or need to patch p3scan
(0004073)
peter-endian   
2010-03-19 17:11   
there's no possibility to filter away squid access from 127.0.0.1 :(
i think we need to patch collectd in order to add a ExcludeRegEx
(0004074)
peter-endian   
2010-03-19 20:42   
- fixed pop proxy virus regexp (which was completely wrong)
- changed pop3 counter to count scanned mails instead of connections

spam counter is still missing, needs the p3scan patch

the fetched values are still strange.
rrdtool images show 400M scanned mails when there were only about 1000, and found 800m virus, when it was 1 (ok this could be ok, due to the fact that it is a rate per second not an absolute value)
(0004077)
peter-endian   
2010-03-22 15:37   
p3scan counts now also spam mails and displays them in dashboard
(0004078)
peter-endian   
2010-03-22 15:38   
found a new issue: 0002785
(0004079)
peter-endian   
2010-03-22 16:09   
and another one:
dashboard counter are counted *always* twice.

this is easily reproduceable:

logger -p local6.info -t squid <<EOF
Mar 19 15:24:55 UPLINKSTEST2 squid[3189]: 1269008695.976 134 127.0.0.1 \
TCP_MISS/200 1746 GET http://intranet.endian.it/logo.gif [^] - \
DIRECT/80.190.199.145 image/gif
EOF

appears as 1, and then after a short time increases by another 1
same on p3scan counters

i think this is on every other counter, too


probably this is because we read multiple times witin the collectd 5 seconds interval where the data does not change.
so if already read out the same value will be read out a second time.

probably we need to check if the timestamp of the last read is older than the current timestamp and ignore if it is (?)
(0004082)
peter-endian   
2010-03-22 22:37   
values counting twice is fixed
(0004083)
peter-endian   
2010-03-22 23:10   
dansguardian counter is correct in collectd, but is always 0 on gui
(0004084)
peter-endian   
2010-03-22 23:43   
dansguardian is ok now.

- http proxy values are all ok, last missing is "miss" which is counted twice (needs collectd patch)
- pop3 proxy values are also all ok now

smtp proxy:
- mails in queue is ok
- clean mail received is ok
- viruses found is ok
- mails received counts twice (from client and from amavis) (needs collectd
  patch)
- mails blocked does work but is in fact mails rejected.. it is not virus
  or spam mails blocked -> rename it
(0004089)
peter-endian   
2010-03-23 20:29   
- collectd patch is now ready
- smtp and proxy counters are ok now

last open things:
- collectd sometimes does not remove the socket file
- values sometimes are completely wrong (overflow?)
(0004101)
peter-endian   
2010-03-26 15:15   
- unixsock patch is sent upstream
- values have now an upper limit as suggested by collectd mailinglist