SYSTEM WARNING: 'date_default_timezone_get(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone.' in '/usr/share/mantis/www/core.php' line 264

0003494: Enabling IDS kills throughput on red. EFW 2.4.1 CE - MantisBT
MantisBT - Endian Firewall
View Issue Details
0003494Endian FirewallIntrusion Preventionpublic2011-02-23 19:052011-03-10 21:19
baldy 
lorenzo-endian 
normalmajoralways
feedbackopen 
2.4.1 
 
0003494: Enabling IDS kills throughput on red. EFW 2.4.1 CE
When enabling IDS/Snort throughput is severely affected.

I have previously reported the issue (http://bugs.endian.it/view.php?id=3274 [^]) as having to do with torrents.

After further testing the issue is not just related to torrent downloads.
When running a speedtest from my provider the difference is enormous.

The same speedtest, performed within 5 minutes of each other, shows an eightfold increase in speed with IDS disabled.
Issue was already reported for earlier EFW versions.

http://efwsupport.com/index.php?action=printpage;topic=998.0 [^]

No tags attached.
has duplicate 0003645new  snort - performance 
jpg Speedtest IDS enabled.jpg (22,965) 2011-02-23 19:05
https://bugs.endian.com/file_download.php?file_id=617&type=bug
jpg

jpg Speedtest IDS disabled.jpg (22,787) 2011-02-23 19:05
https://bugs.endian.com/file_download.php?file_id=618&type=bug
jpg

jpg Snort CPU usage 1 of 2.jpg (214,720) 2011-03-03 11:36
https://bugs.endian.com/file_download.php?file_id=629&type=bug
jpg

jpg Snort CPU usage 2 of 2.jpg (265,556) 2011-03-03 11:39
https://bugs.endian.com/file_download.php?file_id=630&type=bug
jpg Snort disabled CPU.jpg (264,942) 2011-03-03 11:43
https://bugs.endian.com/file_download.php?file_id=631&type=bug
Issue History
2011-02-23 19:05baldyNew Issue
2011-02-23 19:05baldyFile Added: Speedtest IDS enabled.jpg
2011-02-23 19:05baldyFile Added: Speedtest IDS disabled.jpg
2011-02-23 21:19lorenzo-endianNote Added: 0005748
2011-02-23 21:19lorenzo-endianAssigned To => lorenzo-endian
2011-02-23 21:19lorenzo-endianStatusnew => feedback
2011-02-24 08:11baldyNote Added: 0005749
2011-02-24 08:40lorenzo-endianNote Added: 0005750
2011-02-25 13:25baldyNote Added: 0005760
2011-03-02 11:11lorenzo-endianNote Added: 0005795
2011-03-03 11:35baldyNote Added: 0005802
2011-03-03 11:36baldyFile Added: Snort CPU usage 1 of 2.jpg
2011-03-03 11:39baldyFile Added: Snort CPU usage 2 of 2.jpg
2011-03-03 11:43baldyNote Added: 0005803
2011-03-03 11:43baldyFile Added: Snort disabled CPU.jpg
2011-03-03 17:09lorenzo-endianNote Added: 0005808
2011-03-04 08:31ardit-endianNote Added: 0005810
2011-03-04 08:33baldyNote Added: 0005811
2011-03-04 08:37ardit-endianNote Added: 0005812
2011-03-04 09:03ardit-endianNote Added: 0005813
2011-03-04 12:44baldyNote Added: 0005821
2011-03-04 15:54ardit-endianNote Edited: 0005810
2011-03-04 15:55luca-endianNote Added: 0005826
2011-03-04 15:57ardit-endianNote Deleted: 0005813
2011-03-04 16:07ardit-endianNote Deleted: 0005812
2011-03-08 15:27lorenzo-endianNote Added: 0005888
2011-03-10 19:40baldyNote Added: 0005928
2011-03-10 21:19baldyNote Added: 0005929
2011-04-27 13:37lorenzo-endianRelationship addedhas duplicate 0003645

Notes
(0005748)
lorenzo-endian   
2011-02-23 21:19   
Hi Baldy,

I think the problem is related, as the last time, to the rules.

Did you use the rules from emergingthreats or did you create the rules manually?

Thanks in advance!

Lo
(0005749)
baldy   
2011-02-24 08:11   
Hi Lorenzo,

just the default rules, with p2p disabled.

Regards,

Baldy
(0005750)
lorenzo-endian   
2011-02-24 08:40   
Hi Baldy,

thanks a lot for the quick reply :)

I don't know the type of traffic generated by the speedtest and it could be that that specific traffic makes snort crazy ... could you try to download one (or more, in parallel) .iso file(s) or something similar (which generate high traffic condition) and check if the bandwidth is still decreased as before?

Thanks in advance!

Lo
(0005760)
baldy   
2011-02-25 13:25   
Hi Lorenzo,

After stopping and starting snort and testing with normal file downloads (large files from Microsoft Network) the difference is not as big as it was.

However there is still about a 33% drop in throughput with IDS enabled.

Testing the internet connection without Endian, with a laptop connected directly to the modem the speed is exactly as it should be, 60 Mbps down and 6 Mbps up.

With Endian in place and without IDS I get almost the same values, although this differs from time to time, probably due to other services in the LAN creating additional load.

When I reported this issue my download was around 1.1/1.2 MB/s and immediately after disabling the IDS download went up to 6.7/6.8 MB/s.

The speedtest itself is just simple file transfer from server to client and client to server.

Details can be found here : http://wiki.ookla.com/test_flow [^]

Regards,

Baldy
(0005795)
lorenzo-endian   
2011-03-02 11:11   
Hi Baldy,

which is the load of your system when the IDS is enabled? and when it is disable? can you post the output of "top" in both the situations?

Thanks in advance!

Lo
(0005802)
baldy   
2011-03-03 11:35   
Hi Lorenzo,

I have re enabled IDS for testing.

When I initially reported the issue CPU 1 (Dual Core system) was shown at 92% in the GUI. CPU 2 only between 20-30%

Top showed snort using around 95% CPU leaving the rest for the other processes.

Image added is showing cpu usage within less than 10 minutes of enabling snort and only having 2 torrent downloads with the p2p rules disabled.

regards,

baldy
(0005803)
baldy   
2011-03-03 11:43   
Also added cpu usage after disabling snort/ids.

Download speed in uTorrent went straight from around 750KB/s with IDS enabled to 5.0MB/s with IDS disabled.
(0005808)
lorenzo-endian   
2011-03-03 17:09   
Hi Baldy,

your support is super as all the other times :)

I still suspect that the problem is related to a rule, or to a set of rules ... in Services >> Intrusion Prevention >> Rules , as you know, there is the list of the rule files ... can you try to play with them starting with all the rules disabled and trying to enable one file ad a time?

Probably it will take some time but I think this is the only way to reduce (and finally find) the source of the problem...

Please let me know if you can do these test, otherwise I will prepare a system and I will try it myself

Thanks again

Lo
(0005810)
ardit-endian   
2011-03-04 08:31   
(edited on: 2011-03-04 15:54)
I remember that on one system the IDS was UP and the internet was really slow I saw that the snort chain in iptables was full and the system was dropping the packets, this because all "QUEUE" packets, were packets processed by snort (allot of traffic) and of course if you have most of the traffic passed through snort this will cause slow downs (with slow processors *more*).

I think can help in debugging this situation.

(0005811)
baldy   
2011-03-04 08:33   
Hi Ardit,

How can I verify this on my system ?

Regards,

Baldy
(0005821)
baldy   
2011-03-04 12:44   
Hi Ardit/Lorenzo,

Re-enabled IDS and started testing.

Queue increases with about 1MB/s, this will be a problem when the system is running over a prolonged period of time.

High CPU usage is not due to rules, but due to the auto-update feature.
If I leave this disabled CPU usage is normal (20-25%)

Checked this several times with auto-update enabled and disabled.

I think http://bugs.endian.it/view.php?id=3274 [^] can be reopened.

Regards,

Baldy
(0005826)
luca-endian   
2011-03-04 15:55   
actually this is the right way to check your queue status:
root@kenny:~ # cat /proc/net/ip_queue
Peer PID : 25507
Copy mode : 2
Copy range : 65535
Queue length : 0
Queue max. length : 1024
Queue dropped : 0
Netlink dropped : 0
(0005888)
lorenzo-endian   
2011-03-08 15:27   
Hy baldy,

I did some tests today and it seems that the rule which was causing the trouble has been eliminated.

Can you try to update your rules and check if the performances still decreases as before?

Thanks in advance

Lo
(0005928)
baldy   
2011-03-10 19:40   
Hi Lo,

This morning I have started IDS again.

When started it seemed to be okay, CPU usage 3-10%.

With the autoupdate enabled CPU usage is 23%-30%, which is strange as an update feature should not have such an impact.

When monitoring CPU usage still spikes to over 90%, but not continuous as before.

I also noticed that snort is using only 1 core, while it should be multi-core/processor aware.

I will keep monitoring for a couple of days.

Regards,

Baldy
(0005929)
baldy   
2011-03-10 21:19   
Still the same.

After just 1,5 hours snort cpu usage was a consistent 48-55% and throughput limited to 800-900 KB/s.

Disabling IDS resulted in an increased download speed, went up to 2.2MB/s.

When I am downloading a lot (started 85 HD, about 900GB, movies last week for testing) I am limited to around 1.1 MB/s with IDS enabled. With IDS disabled my internet connection maxes out around 6.8-6.9 MB/s.

Regards,

Baldy