SYSTEM WARNING: 'date_default_timezone_get(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone.' in '/usr/share/mantis/www/core.php' line 264

0003494: Enabling IDS kills throughput on red. EFW 2.4.1 CE - MantisBT Endian Bugtracker
Endian Issue Tracker





Please see now our new Bugtracker system: JIRA








View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0003494Endian FirewallIntrusion Preventionpublic2011-02-23 19:052011-03-10 21:19
Reporterbaldy 
Assigned Tolorenzo-endian 
PrioritynormalSeveritymajorReproducibilityalways
StatusfeedbackResolutionopen 
PlatformOSOS Version
Product Version2.4.1 
Target VersionFixed in Version 
Summary0003494: Enabling IDS kills throughput on red. EFW 2.4.1 CE
DescriptionWhen enabling IDS/Snort throughput is severely affected.

I have previously reported the issue (http://bugs.endian.it/view.php?id=3274 [^]) as having to do with torrents.

After further testing the issue is not just related to torrent downloads.
When running a speedtest from my provider the difference is enormous.

The same speedtest, performed within 5 minutes of each other, shows an eightfold increase in speed with IDS disabled.
Additional InformationIssue was already reported for earlier EFW versions.

http://efwsupport.com/index.php?action=printpage;topic=998.0 [^]

TagsNo tags attached.
Attached Filesjpg file icon Speedtest IDS enabled.jpg [^] (22,965 bytes) 2011-02-23 19:05


jpg file icon Speedtest IDS disabled.jpg [^] (22,787 bytes) 2011-02-23 19:05


jpg file icon Snort CPU usage 1 of 2.jpg [^] (214,720 bytes) 2011-03-03 11:36


jpg file icon Snort CPU usage 2 of 2.jpg [^] (265,556 bytes) 2011-03-03 11:39
jpg file icon Snort disabled CPU.jpg [^] (264,942 bytes) 2011-03-03 11:43

- Relationships
has duplicate 0003645new snort - performance 

-  Notes
(0005748)
lorenzo-endian (manager)
2011-02-23 21:19

Hi Baldy,

I think the problem is related, as the last time, to the rules.

Did you use the rules from emergingthreats or did you create the rules manually?

Thanks in advance!

Lo
(0005749)
baldy (reporter)
2011-02-24 08:11

Hi Lorenzo,

just the default rules, with p2p disabled.

Regards,

Baldy
(0005750)
lorenzo-endian (manager)
2011-02-24 08:40

Hi Baldy,

thanks a lot for the quick reply :)

I don't know the type of traffic generated by the speedtest and it could be that that specific traffic makes snort crazy ... could you try to download one (or more, in parallel) .iso file(s) or something similar (which generate high traffic condition) and check if the bandwidth is still decreased as before?

Thanks in advance!

Lo
(0005760)
baldy (reporter)
2011-02-25 13:25

Hi Lorenzo,

After stopping and starting snort and testing with normal file downloads (large files from Microsoft Network) the difference is not as big as it was.

However there is still about a 33% drop in throughput with IDS enabled.

Testing the internet connection without Endian, with a laptop connected directly to the modem the speed is exactly as it should be, 60 Mbps down and 6 Mbps up.

With Endian in place and without IDS I get almost the same values, although this differs from time to time, probably due to other services in the LAN creating additional load.

When I reported this issue my download was around 1.1/1.2 MB/s and immediately after disabling the IDS download went up to 6.7/6.8 MB/s.

The speedtest itself is just simple file transfer from server to client and client to server.

Details can be found here : http://wiki.ookla.com/test_flow [^]

Regards,

Baldy
(0005795)
lorenzo-endian (manager)
2011-03-02 11:11

Hi Baldy,

which is the load of your system when the IDS is enabled? and when it is disable? can you post the output of "top" in both the situations?

Thanks in advance!

Lo
(0005802)
baldy (reporter)
2011-03-03 11:35

Hi Lorenzo,

I have re enabled IDS for testing.

When I initially reported the issue CPU 1 (Dual Core system) was shown at 92% in the GUI. CPU 2 only between 20-30%

Top showed snort using around 95% CPU leaving the rest for the other processes.

Image added is showing cpu usage within less than 10 minutes of enabling snort and only having 2 torrent downloads with the p2p rules disabled.

regards,

baldy
(0005803)
baldy (reporter)
2011-03-03 11:43

Also added cpu usage after disabling snort/ids.

Download speed in uTorrent went straight from around 750KB/s with IDS enabled to 5.0MB/s with IDS disabled.
(0005808)
lorenzo-endian (manager)
2011-03-03 17:09

Hi Baldy,

your support is super as all the other times :)

I still suspect that the problem is related to a rule, or to a set of rules ... in Services >> Intrusion Prevention >> Rules , as you know, there is the list of the rule files ... can you try to play with them starting with all the rules disabled and trying to enable one file ad a time?

Probably it will take some time but I think this is the only way to reduce (and finally find) the source of the problem...

Please let me know if you can do these test, otherwise I will prepare a system and I will try it myself

Thanks again

Lo
(0005810)
ardit-endian (developer)
2011-03-04 08:31
edited on: 2011-03-04 15:54

I remember that on one system the IDS was UP and the internet was really slow I saw that the snort chain in iptables was full and the system was dropping the packets, this because all "QUEUE" packets, were packets processed by snort (allot of traffic) and of course if you have most of the traffic passed through snort this will cause slow downs (with slow processors *more*).

I think can help in debugging this situation.

(0005811)
baldy (reporter)
2011-03-04 08:33

Hi Ardit,

How can I verify this on my system ?

Regards,

Baldy
(0005821)
baldy (reporter)
2011-03-04 12:44

Hi Ardit/Lorenzo,

Re-enabled IDS and started testing.

Queue increases with about 1MB/s, this will be a problem when the system is running over a prolonged period of time.

High CPU usage is not due to rules, but due to the auto-update feature.
If I leave this disabled CPU usage is normal (20-25%)

Checked this several times with auto-update enabled and disabled.

I think http://bugs.endian.it/view.php?id=3274 [^] can be reopened.

Regards,

Baldy
(0005826)
luca-endian (developer)
2011-03-04 15:55

actually this is the right way to check your queue status:
root@kenny:~ # cat /proc/net/ip_queue
Peer PID : 25507
Copy mode : 2
Copy range : 65535
Queue length : 0
Queue max. length : 1024
Queue dropped : 0
Netlink dropped : 0
(0005888)
lorenzo-endian (manager)
2011-03-08 15:27

Hy baldy,

I did some tests today and it seems that the rule which was causing the trouble has been eliminated.

Can you try to update your rules and check if the performances still decreases as before?

Thanks in advance

Lo
(0005928)
baldy (reporter)
2011-03-10 19:40

Hi Lo,

This morning I have started IDS again.

When started it seemed to be okay, CPU usage 3-10%.

With the autoupdate enabled CPU usage is 23%-30%, which is strange as an update feature should not have such an impact.

When monitoring CPU usage still spikes to over 90%, but not continuous as before.

I also noticed that snort is using only 1 core, while it should be multi-core/processor aware.

I will keep monitoring for a couple of days.

Regards,

Baldy
(0005929)
baldy (reporter)
2011-03-10 21:19

Still the same.

After just 1,5 hours snort cpu usage was a consistent 48-55% and throughput limited to 800-900 KB/s.

Disabling IDS resulted in an increased download speed, went up to 2.2MB/s.

When I am downloading a lot (started 85 HD, about 900GB, movies last week for testing) I am limited to around 1.1 MB/s with IDS enabled. With IDS disabled my internet connection maxes out around 6.8-6.9 MB/s.

Regards,

Baldy

- Issue History
Date Modified Username Field Change
2011-02-23 19:05 baldy New Issue
2011-02-23 19:05 baldy File Added: Speedtest IDS enabled.jpg
2011-02-23 19:05 baldy File Added: Speedtest IDS disabled.jpg
2011-02-23 21:19 lorenzo-endian Note Added: 0005748
2011-02-23 21:19 lorenzo-endian Assigned To => lorenzo-endian
2011-02-23 21:19 lorenzo-endian Status new => feedback
2011-02-24 08:11 baldy Note Added: 0005749
2011-02-24 08:40 lorenzo-endian Note Added: 0005750
2011-02-25 13:25 baldy Note Added: 0005760
2011-03-02 11:11 lorenzo-endian Note Added: 0005795
2011-03-03 11:35 baldy Note Added: 0005802
2011-03-03 11:36 baldy File Added: Snort CPU usage 1 of 2.jpg
2011-03-03 11:39 baldy File Added: Snort CPU usage 2 of 2.jpg
2011-03-03 11:43 baldy Note Added: 0005803
2011-03-03 11:43 baldy File Added: Snort disabled CPU.jpg
2011-03-03 17:09 lorenzo-endian Note Added: 0005808
2011-03-04 08:31 ardit-endian Note Added: 0005810
2011-03-04 08:33 baldy Note Added: 0005811
2011-03-04 08:37 ardit-endian Note Added: 0005812
2011-03-04 09:03 ardit-endian Note Added: 0005813
2011-03-04 12:44 baldy Note Added: 0005821
2011-03-04 15:54 ardit-endian Note Edited: 0005810
2011-03-04 15:55 luca-endian Note Added: 0005826
2011-03-04 15:57 ardit-endian Note Deleted: 0005813
2011-03-04 16:07 ardit-endian Note Deleted: 0005812
2011-03-08 15:27 lorenzo-endian Note Added: 0005888
2011-03-10 19:40 baldy Note Added: 0005928
2011-03-10 21:19 baldy Note Added: 0005929
2011-04-27 13:37 lorenzo-endian Relationship added has duplicate 0003645

Copyright © 2005-2008 Endian, SRL. All rights reserved.


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker