0003221: http proxy don't returns anything after some time. If flush cache of Squid, the firewall works again correctly

(0005204)
gmar_87 (reporter)
2010-11-25 10:14

I too am having this problem. I restored an Endian Firewall 2.3 backup config to an identical hardware spec server also running version 2.3 and see this issue everyday!

I reinstalled Endian, this time with 2.4.1 and restored the backup, but still occurring. The end-user sees "Read Error. Connection reset by peer".

My cache log shows:
2010/11/25 16:39:52| TCP connection to 127.0.0.2/9999 failed
2010/11/25 16:39:52| Detected DEAD Parent: content2

I have now disabled antivirus scans in my content filters to see if that is the cause...

(0005264)
lorenzo-endian (manager)
2010-12-02 09:26

Hi bortol and gmar_87,

does you efw work after disabling the antivirus scan?

This info would be useful to troubleshoot the problem!

Thanks in advance!

Lo

(0005265)
gmar_87 (reporter)
2010-12-02 09:34

Hi Lo,

Proxy seems to be stable after disabling anti-virus scanning under proxy content filter settings.
Uptime = 6d 22h 44m so far..

Cheers,
John

(0005266)
gmar_87 (reporter)
2010-12-02 09:36

also seemed to only occur under heavy load/traffic.

(0005303)
gmar_87 (reporter)
2010-12-05 23:23

Definitely related to having Anti virus scanning enabled under content filter.
EFW has been up for 10d 12h 20m after disabling this option.

(0005369)
bortol (reporter)
2010-12-14 09:27

I've reinstalled all with release 2.4.0 and I don't have any problem from 31d 9h 53m (with antivirus scan actived).

(0005383)
gmar_87 (reporter)
2010-12-15 22:24

I can confirm the this issue only occurs on release 2.4.1

(0005493)
lorenzo-endian (manager)
2011-01-18 11:54

Hi bordol,

can you provide please the version of you efw-clamav package?

You can get it with the command

rpm -q efw-clamav

Thanks in advance!

Lo

(0005501)
bortol (reporter)
2011-01-18 20:37

Now I have reinstalled version 2.4.0 and the efw-clamav is efw-clamav-2.3.17-0.endian5
I don't kwow the version when of efw-clamav in 2.4.1 ... sorry

Bye

Bortol

p.s. in italiano

non conosco bene l'inglese dunque fatico a scrivere in quella lingua... Ho visto che ti chiami Lorenzo: non è che sei italiano?

Ho deciso, non essendo riuscito a fare funzionare senza blocchi la 2.4.1, di riinstallare la 2.4.0 con cui non ho problemi se non nello scaricamento di alcuni file pdf di grosse dimensioni.

(0005502)
lorenzo-endian (manager)
2011-01-18 20:47

Hey,

yes, I am italian :-P we try to use english on the bugtrack so that the information of a ticket are useful for all the people around the world :)

Today I tried to replicate the problem on a 2.4.1 but without success, but I think I have discovered something interesting and the fact that you are using the package efw-clamav-2.3.17-0.endian5 is a great help for me!

Thanks a lot

Lo

--- TRANSLATED ---

Ciao!

sisi, sono italiano :-P cerchiamo di tenere l'inglese sul bagtracker perchè cosi le informazioni servono a tutti quelli che nel mondo hanno problemi. Io oggi ho provato a replicare il problema con una 2.4.1 ma non ci sono riuscito.

Ad ogni modo credo di aver scoperto qualche cosa ed il fatto che stai usando il pacchetto efw-clamav-2.3.17-0.endian5 mi aiuta un sacco!

Grazie mille davvero

Lo

(0005510)
gmar_87 (reporter)
2011-01-20 00:30

My EFW 2.4.1 shows efw-clamav-2.4.4-0.endian8

(0005512)
claurita (reporter)
2011-01-20 08:25

Hi everybody,
inserting myself in the thread because I have identical problem and made some tests I'd like to report.

Running 2.4.1, efw-clamav-2.4.4-0.endian8
The problem arises after a couple of hours since proxy reset, (under low traffic conditions). In my case, it's not related just to clam called from dansguardian. Using havp alone has about the same final effect, but squid reports a different error:
--------------
2011/01/18 23:57:00| helperOpenServers: Starting 20 'ncsa_auth' processes
2011/01/19 08:22:01| parseHttpRequest: Unsupported method '<D1><BC>Sp<D4><C1><D1><C6><AB><DD>^NY^R<89>^X<E3><E6><BA>^V^V=^Q^K<FC><D4><96>dx^S<93>bN^E<A8>KRi
<DF><99><8E>Wvh'
2011/01/19 08:22:01| clientReadRequest: FD 43 (192.168.18.54:1068) Invalid Request
--------------

I tried clamav updates from stellarcore.net (I've been using them since endian 2.0), actually clamav 0.95.5 and havp 0.91
Nothing seems changed, but I noticed that havp log claims an error in clamav:
-------------------
Jan 19 22:05:38 efw havp[32295]: Detected crashed ClamAV Library Scanner process
(getanswer, pid: 32296, lasturl: http://www.google.it/search? [^])
Jan 19 22:05:38 efw havp[32295]: Scanner errors: ClamAV: Scanner crashed (lastur
l: http://www.google.it/search? [^])
Jan 19 22:14:25 efw havp[537]: Detected crashed ClamAV Library Scanner process (
getanswer, pid: 539, lasturl: http://suggestqueries.google.com/complete/search? [^])
Jan 19 22:14:25 efw havp[537]: Scanner errors: ClamAV: Scanner crashed (lasturl:
http://suggestqueries.google.com/complete/search? [^])
---------------

If I could help with other tests, ask me.
Claudio

(0005651)
lorenzo-endian (manager)
2011-02-09 22:19

Hi everybody,

I have tested a lot havp and clamav and they don't freeze the system on my side.

Can I kindly ask to you which version of HAVP are your systems running?

You can get it using

rpm -q efw-havp

Thanks to all in advance!

Lo

(0005652)
lorenzo-endian (manager)
2011-02-09 22:28

ps: On my system:

root@efw-lo-ce-2:~ # rpm -q efw-clamav
efw-clamav-2.4.4-0.endian8
root@efw-lo-ce-2:~ # rpm -q efw-havp
efw-havp-2.3.19-0.endian3
root@efw-lo-ce-2:~ # rpm -q squid
squid-2.6.STABLE22-6.endian10
root@efw-lo-ce-2:~ #

(0005653)
gmar_87 (reporter)
2011-02-10 06:13

My system:
root@PROXY1:~ # rpm -q efw-clamav
efw-clamav-2.4.4-0.endian8
root@PROXY1:~ # rpm -q efw-havp
efw-havp-2.3.19-0.endian3
root@PROXY1:~ # rpm -q squid
squid-2.6.STABLE22-6.endian10
root@PROXY1:~ #

(0005655)
claurita (reporter)
2011-02-10 08:25

This is my "official" efw machine:
root@efw:~ # rpm -q efw-clamav
efw-clamav-2.4.4-0.endian8
root@efw:~ # rpm -q efw-havp
efw-havp-2.3.19-0.endian3
root@efw:~ # rpm -q squid
squid-2.6.STABLE22-6.endian10

Don't know if this could help:
I'm also actually testing efw on a pc without hd (I'm using a 2GB SD, tmpfs for /tmp and /var, 2 GB ram, NO swap). It's a fresh 2.4.1 install with the same patches applied as my "official" one, and works very well. Focusing, of course, on ram usage, I noticed that sometimes it starts rising and reaches 98% in few hours (normally is about 50%, low load, many days of working). At that point, havp crashes in a way much similar to the one focused in this thread. Haven't yet found the event which triggers this behaviour, but I suspect it could be exactly the same problem we're investigating here.
Claudio

(0005799)
ardit-endian (developer)
2011-03-03 10:01
edited on: 2011-03-03 10:54

Hi,

the problem is related to dansguardian, for some reason "the guy" goes down :)

http://pastie.org/1627966 [^]

The problem with dansguardian now is that it have no debug options [at least for this issue], if you want dansguardian in debug mode we need to compile the "debug version" of dansguardian:

http://contentfilter.futuragts.com/wiki/doku.php?id=using_a_debug_version [^]

As the wiki says, this version is used for
"Unexplained frequent crashes when not even a stack backtrace identifies a resolution. "

I think this is the case.

In past we have several reports of the same problem and unfortunately dansguardian doesn't "say" much om the logs.

Regards,
Ardit.

(0005879)
diwoda (reporter)
2011-03-08 12:15

Something new about this? I had a similar problem today, http proxy just stopped working, believing that it is the same issue. Flushing the Cache made it work again...Logs look quite the same as above...

greets
Johann

(0005889)
lorenzo-endian (manager)
2011-03-08 15:43

Hello diwoda,

we are working on it! I will keep you updated about the progress :)

Thanks for the patience

Lo

(0007087)
gvecchi (reporter)
2011-07-26 07:01
edited on: 2011-07-28 09:29

Hi all!
I think I have the problem me too.

root@PROXY:~ # rpm -q efw-clamav
efw-clamav-2.4.4-0.endian8
root@PROXY:~ # rpm -q efw-havp
efw-havp-2.3.19-0.endian3
root@PROXY:~ # rpm -q squid
squid-2.6.STABLE22-6.endian10
root@PROXY:~ # rpm -q efw-dansguardian
efw-dansguardian-2.4.1-2.endian15

after weekly/dayly automatic backup, dansguardian fails to start:

root@PROXY:~ # /etc/init.d/dansguardian restart
Stopping dansguardian: [FAILED]
Starting dansguardian: [FAILED]

Any workaround? Any news about solution?

Setting squid to allow traffic when dansguardian goes down may be a right workaround, isn't it?

Thanks!

(0008352)
victorhugops (reporter)
2012-12-10 15:49

Hello,

here, we have the same problem (with the last endian version) !!! :-(

(0008368)
rbianchi (reporter)
2013-02-08 15:15

We have the same problem with Endian Community ed. 2.5.1
Trying disable HAVP.

(0008415)
jejethx (reporter)
2013-04-06 20:27

Hello,

Do you find issue to this probleme?
I'am increase MAXSERVERS & SERVERNUMBER in /var/efw/havp/settings :
MAXSERVERS=500
SERVERNUMBER=200
Set 1Mb of Squid cache but it not resolv.

Regards

(0008418)
riaanjvr (reporter)
2013-04-16 19:29
edited on: 2013-04-16 19:38

Hallo
This happens in the commercial Endian as well. I have the latest version 2.5.1 Endian appliance. In the Web IF one can see HAVP is not running. It broke after a while from setting it up, and I changed the P.I.C.S score from 50 to 100 in the content filter.

Flushing the cache, rebooting, en/disabling the proxy doesnt help
Forcing an update of Dansguardian rules, doesnt help

Notes
(0005204) gmar_87 (reporter) 2010-11-25 10:14	I too am having this problem. I restored an Endian Firewall 2.3 backup config to an identical hardware spec server also running version 2.3 and see this issue everyday! I reinstalled Endian, this time with 2.4.1 and restored the backup, but still occurring. The end-user sees "Read Error. Connection reset by peer". My cache log shows: 2010/11/25 16:39:52\| TCP connection to 127.0.0.2/9999 failed 2010/11/25 16:39:52\| Detected DEAD Parent: content2 I have now disabled antivirus scans in my content filters to see if that is the cause...

(0005264) lorenzo-endian (manager) 2010-12-02 09:26	Hi bortol and gmar_87, does you efw work after disabling the antivirus scan? This info would be useful to troubleshoot the problem! Thanks in advance! Lo

(0005265) gmar_87 (reporter) 2010-12-02 09:34	Hi Lo, Proxy seems to be stable after disabling anti-virus scanning under proxy content filter settings. Uptime = 6d 22h 44m so far.. Cheers, John

(0005266) gmar_87 (reporter) 2010-12-02 09:36	also seemed to only occur under heavy load/traffic.

(0005303) gmar_87 (reporter) 2010-12-05 23:23	Definitely related to having Anti virus scanning enabled under content filter. EFW has been up for 10d 12h 20m after disabling this option.

(0005369) bortol (reporter) 2010-12-14 09:27	I've reinstalled all with release 2.4.0 and I don't have any problem from 31d 9h 53m (with antivirus scan actived).

(0005383) gmar_87 (reporter) 2010-12-15 22:24	I can confirm the this issue only occurs on release 2.4.1

(0005493) lorenzo-endian (manager) 2011-01-18 11:54	Hi bordol, can you provide please the version of you efw-clamav package? You can get it with the command rpm -q efw-clamav Thanks in advance! Lo

(0005501) bortol (reporter) 2011-01-18 20:37	Now I have reinstalled version 2.4.0 and the efw-clamav is efw-clamav-2.3.17-0.endian5 I don't kwow the version when of efw-clamav in 2.4.1 ... sorry Bye Bortol p.s. in italiano non conosco bene l'inglese dunque fatico a scrivere in quella lingua... Ho visto che ti chiami Lorenzo: non è che sei italiano? Ho deciso, non essendo riuscito a fare funzionare senza blocchi la 2.4.1, di riinstallare la 2.4.0 con cui non ho problemi se non nello scaricamento di alcuni file pdf di grosse dimensioni.

(0005502) lorenzo-endian (manager) 2011-01-18 20:47	Hey, yes, I am italian :-P we try to use english on the bugtrack so that the information of a ticket are useful for all the people around the world :) Today I tried to replicate the problem on a 2.4.1 but without success, but I think I have discovered something interesting and the fact that you are using the package efw-clamav-2.3.17-0.endian5 is a great help for me! Thanks a lot Lo --- TRANSLATED --- Ciao! sisi, sono italiano :-P cerchiamo di tenere l'inglese sul bagtracker perchè cosi le informazioni servono a tutti quelli che nel mondo hanno problemi. Io oggi ho provato a replicare il problema con una 2.4.1 ma non ci sono riuscito. Ad ogni modo credo di aver scoperto qualche cosa ed il fatto che stai usando il pacchetto efw-clamav-2.3.17-0.endian5 mi aiuta un sacco! Grazie mille davvero Lo

(0005510) gmar_87 (reporter) 2011-01-20 00:30	My EFW 2.4.1 shows efw-clamav-2.4.4-0.endian8

(0005512) claurita (reporter) 2011-01-20 08:25	Hi everybody, inserting myself in the thread because I have identical problem and made some tests I'd like to report. Running 2.4.1, efw-clamav-2.4.4-0.endian8 The problem arises after a couple of hours since proxy reset, (under low traffic conditions). In my case, it's not related just to clam called from dansguardian. Using havp alone has about the same final effect, but squid reports a different error: -------------- 2011/01/18 23:57:00\| helperOpenServers: Starting 20 'ncsa_auth' processes 2011/01/19 08:22:01\| parseHttpRequest: Unsupported method '<D1><BC>Sp<D4><C1><D1><C6><AB><DD>^NY^R<89>^X<E3><E6><BA>^V^V=^Q^K<FC><D4><96>dx^S<93>bN^E<A8>KRi <DF><99><8E>Wvh' 2011/01/19 08:22:01\| clientReadRequest: FD 43 (192.168.18.54:1068) Invalid Request -------------- I tried clamav updates from stellarcore.net (I've been using them since endian 2.0), actually clamav 0.95.5 and havp 0.91 Nothing seems changed, but I noticed that havp log claims an error in clamav: ------------------- Jan 19 22:05:38 efw havp[32295]: Detected crashed ClamAV Library Scanner process (getanswer, pid: 32296, lasturl: http://www.google.it/search? [^]) Jan 19 22:05:38 efw havp[32295]: Scanner errors: ClamAV: Scanner crashed (lastur l: http://www.google.it/search? [^]) Jan 19 22:14:25 efw havp[537]: Detected crashed ClamAV Library Scanner process ( getanswer, pid: 539, lasturl: http://suggestqueries.google.com/complete/search? [^]) Jan 19 22:14:25 efw havp[537]: Scanner errors: ClamAV: Scanner crashed (lasturl: http://suggestqueries.google.com/complete/search? [^]) --------------- If I could help with other tests, ask me. Claudio

(0005651) lorenzo-endian (manager) 2011-02-09 22:19	Hi everybody, I have tested a lot havp and clamav and they don't freeze the system on my side. Can I kindly ask to you which version of HAVP are your systems running? You can get it using rpm -q efw-havp Thanks to all in advance! Lo

(0005652) lorenzo-endian (manager) 2011-02-09 22:28	ps: On my system: root@efw-lo-ce-2:~ # rpm -q efw-clamav efw-clamav-2.4.4-0.endian8 root@efw-lo-ce-2:~ # rpm -q efw-havp efw-havp-2.3.19-0.endian3 root@efw-lo-ce-2:~ # rpm -q squid squid-2.6.STABLE22-6.endian10 root@efw-lo-ce-2:~ #

(0005653) gmar_87 (reporter) 2011-02-10 06:13	My system: root@PROXY1:~ # rpm -q efw-clamav efw-clamav-2.4.4-0.endian8 root@PROXY1:~ # rpm -q efw-havp efw-havp-2.3.19-0.endian3 root@PROXY1:~ # rpm -q squid squid-2.6.STABLE22-6.endian10 root@PROXY1:~ #

(0005655) claurita (reporter) 2011-02-10 08:25	This is my "official" efw machine: root@efw:~ # rpm -q efw-clamav efw-clamav-2.4.4-0.endian8 root@efw:~ # rpm -q efw-havp efw-havp-2.3.19-0.endian3 root@efw:~ # rpm -q squid squid-2.6.STABLE22-6.endian10 Don't know if this could help: I'm also actually testing efw on a pc without hd (I'm using a 2GB SD, tmpfs for /tmp and /var, 2 GB ram, NO swap). It's a fresh 2.4.1 install with the same patches applied as my "official" one, and works very well. Focusing, of course, on ram usage, I noticed that sometimes it starts rising and reaches 98% in few hours (normally is about 50%, low load, many days of working). At that point, havp crashes in a way much similar to the one focused in this thread. Haven't yet found the event which triggers this behaviour, but I suspect it could be exactly the same problem we're investigating here. Claudio

(0005799) ardit-endian (developer) 2011-03-03 10:01 edited on: 2011-03-03 10:54	Hi, the problem is related to dansguardian, for some reason "the guy" goes down :) http://pastie.org/1627966 [^] The problem with dansguardian now is that it have no debug options [at least for this issue], if you want dansguardian in debug mode we need to compile the "debug version" of dansguardian: http://contentfilter.futuragts.com/wiki/doku.php?id=using_a_debug_version [^] As the wiki says, this version is used for "Unexplained frequent crashes when not even a stack backtrace identifies a resolution. " I think this is the case. In past we have several reports of the same problem and unfortunately dansguardian doesn't "say" much om the logs. Regards, Ardit.

(0005879) diwoda (reporter) 2011-03-08 12:15	Something new about this? I had a similar problem today, http proxy just stopped working, believing that it is the same issue. Flushing the Cache made it work again...Logs look quite the same as above... greets Johann

(0005889) lorenzo-endian (manager) 2011-03-08 15:43	Hello diwoda, we are working on it! I will keep you updated about the progress :) Thanks for the patience Lo

(0007087) gvecchi (reporter) 2011-07-26 07:01 edited on: 2011-07-28 09:29	Hi all! I think I have the problem me too. root@PROXY:~ # rpm -q efw-clamav efw-clamav-2.4.4-0.endian8 root@PROXY:~ # rpm -q efw-havp efw-havp-2.3.19-0.endian3 root@PROXY:~ # rpm -q squid squid-2.6.STABLE22-6.endian10 root@PROXY:~ # rpm -q efw-dansguardian efw-dansguardian-2.4.1-2.endian15 after weekly/dayly automatic backup, dansguardian fails to start: root@PROXY:~ # /etc/init.d/dansguardian restart Stopping dansguardian: [FAILED] Starting dansguardian: [FAILED] Any workaround? Any news about solution? Setting squid to allow traffic when dansguardian goes down may be a right workaround, isn't it? Thanks!

(0008352) victorhugops (reporter) 2012-12-10 15:49	Hello, here, we have the same problem (with the last endian version) !!! :-(

(0008368) rbianchi (reporter) 2013-02-08 15:15	We have the same problem with Endian Community ed. 2.5.1 Trying disable HAVP.

(0008415) jejethx (reporter) 2013-04-06 20:27	Hello, Do you find issue to this probleme? I'am increase MAXSERVERS & SERVERNUMBER in /var/efw/havp/settings : MAXSERVERS=500 SERVERNUMBER=200 Set 1Mb of Squid cache but it not resolv. Regards

(0008418) riaanjvr (reporter) 2013-04-16 19:29 edited on: 2013-04-16 19:38	Hallo This happens in the commercial Endian as well. I have the latest version 2.5.1 Endian appliance. In the Web IF one can see HAVP is not running. It broke after a while from setting it up, and I changed the P.I.C.S score from 50 to 100 in the content filter. Flushing the cache, rebooting, en/disabling the proxy doesnt help Forcing an update of Dansguardian rules, doesnt help

Issue History
Date Modified	Username	Field	Change
2010-10-25 17:41	bortol	New Issue
2010-11-25 10:14	gmar_87	Note Added: 0005204
2010-12-02 09:26	lorenzo-endian	Note Added: 0005264
2010-12-02 09:26	lorenzo-endian	Assigned To	=> lorenzo-endian
2010-12-02 09:26	lorenzo-endian	Status	new => feedback
2010-12-02 09:34	gmar_87	Note Added: 0005265
2010-12-02 09:36	gmar_87	Note Added: 0005266
2010-12-05 23:23	gmar_87	Note Added: 0005303
2010-12-14 09:27	bortol	Note Added: 0005369
2010-12-15 22:24	gmar_87	Note Added: 0005383
2011-01-18 11:54	lorenzo-endian	Note Added: 0005493
2011-01-18 20:37	bortol	Note Added: 0005501
2011-01-18 20:47	lorenzo-endian	Note Added: 0005502
2011-01-20 00:30	gmar_87	Note Added: 0005510
2011-01-20 08:25	claurita	Note Added: 0005512
2011-02-09 22:19	lorenzo-endian	Note Added: 0005651
2011-02-09 22:28	lorenzo-endian	Note Added: 0005652
2011-02-10 06:13	gmar_87	Note Added: 0005653
2011-02-10 08:25	claurita	Note Added: 0005655
2011-03-03 10:01	ardit-endian	Note Added: 0005799
2011-03-03 10:02	ardit-endian	Tag Attached: purple
2011-03-03 10:54	ardit-endian	Note Edited: 0005799
2011-03-08 12:15	diwoda	Note Added: 0005879
2011-03-08 15:43	lorenzo-endian	Note Added: 0005889
2011-03-09 12:02	lorenzo-endian	Relationship added	related to 0003528
2011-07-26 07:01	gvecchi	Note Added: 0007087
2011-07-26 07:04	gvecchi	Note Edited: 0007087
2011-07-26 07:07	gvecchi	Note Edited: 0007087
2011-07-26 13:42	gvecchi	Note Edited: 0007087
2011-07-28 09:29	gvecchi	Note Edited: 0007087
2012-12-10 15:49	victorhugops	Note Added: 0008352
2013-02-08 15:15	rbianchi	Note Added: 0008368
2013-04-06 20:27	jejethx	Note Added: 0008415
2013-04-16 19:29	riaanjvr	Note Added: 0008418
2013-04-16 19:38	riaanjvr	Note Edited: 0008418	View Revisions

Please see now our new Bugtracker system: JIRA