0003494: Enabling IDS kills throughput on red. EFW 2.4.1 CE

(0005748)
lorenzo-endian (manager)
2011-02-23 21:19

Hi Baldy,

I think the problem is related, as the last time, to the rules.

Did you use the rules from emergingthreats or did you create the rules manually?

Thanks in advance!

Lo

(0005749)
baldy (reporter)
2011-02-24 08:11

Hi Lorenzo,

just the default rules, with p2p disabled.

Regards,

Baldy

(0005750)
lorenzo-endian (manager)
2011-02-24 08:40

Hi Baldy,

thanks a lot for the quick reply :)

I don't know the type of traffic generated by the speedtest and it could be that that specific traffic makes snort crazy ... could you try to download one (or more, in parallel) .iso file(s) or something similar (which generate high traffic condition) and check if the bandwidth is still decreased as before?

Thanks in advance!

Lo

(0005760)
baldy (reporter)
2011-02-25 13:25

Hi Lorenzo,

After stopping and starting snort and testing with normal file downloads (large files from Microsoft Network) the difference is not as big as it was.

However there is still about a 33% drop in throughput with IDS enabled.

Testing the internet connection without Endian, with a laptop connected directly to the modem the speed is exactly as it should be, 60 Mbps down and 6 Mbps up.

With Endian in place and without IDS I get almost the same values, although this differs from time to time, probably due to other services in the LAN creating additional load.

When I reported this issue my download was around 1.1/1.2 MB/s and immediately after disabling the IDS download went up to 6.7/6.8 MB/s.

The speedtest itself is just simple file transfer from server to client and client to server.

Details can be found here : http://wiki.ookla.com/test_flow [^]

Regards,

Baldy

(0005795)
lorenzo-endian (manager)
2011-03-02 11:11

Hi Baldy,

which is the load of your system when the IDS is enabled? and when it is disable? can you post the output of "top" in both the situations?

Thanks in advance!

Lo

(0005802)
baldy (reporter)
2011-03-03 11:35

Hi Lorenzo,

I have re enabled IDS for testing.

When I initially reported the issue CPU 1 (Dual Core system) was shown at 92% in the GUI. CPU 2 only between 20-30%

Top showed snort using around 95% CPU leaving the rest for the other processes.

Image added is showing cpu usage within less than 10 minutes of enabling snort and only having 2 torrent downloads with the p2p rules disabled.

regards,

baldy

(0005803)
baldy (reporter)
2011-03-03 11:43

Also added cpu usage after disabling snort/ids.

Download speed in uTorrent went straight from around 750KB/s with IDS enabled to 5.0MB/s with IDS disabled.

(0005808)
lorenzo-endian (manager)
2011-03-03 17:09

Hi Baldy,

your support is super as all the other times :)

I still suspect that the problem is related to a rule, or to a set of rules ... in Services >> Intrusion Prevention >> Rules , as you know, there is the list of the rule files ... can you try to play with them starting with all the rules disabled and trying to enable one file ad a time?

Probably it will take some time but I think this is the only way to reduce (and finally find) the source of the problem...

Please let me know if you can do these test, otherwise I will prepare a system and I will try it myself

Thanks again

Lo

(0005810)
ardit-endian (developer)
2011-03-04 08:31
edited on: 2011-03-04 15:54

I remember that on one system the IDS was UP and the internet was really slow I saw that the snort chain in iptables was full and the system was dropping the packets, this because all "QUEUE" packets, were packets processed by snort (allot of traffic) and of course if you have most of the traffic passed through snort this will cause slow downs (with slow processors *more*).

I think can help in debugging this situation.

(0005811)
baldy (reporter)
2011-03-04 08:33

Hi Ardit,

How can I verify this on my system ?

Regards,

Baldy

(0005821)
baldy (reporter)
2011-03-04 12:44

Hi Ardit/Lorenzo,

Re-enabled IDS and started testing.

Queue increases with about 1MB/s, this will be a problem when the system is running over a prolonged period of time.

High CPU usage is not due to rules, but due to the auto-update feature.
If I leave this disabled CPU usage is normal (20-25%)

Checked this several times with auto-update enabled and disabled.

I think http://bugs.endian.it/view.php?id=3274 [^] can be reopened.

Regards,

Baldy

(0005826)
luca-endian (developer)
2011-03-04 15:55

actually this is the right way to check your queue status:
root@kenny:~ # cat /proc/net/ip_queue
Peer PID : 25507
Copy mode : 2
Copy range : 65535
Queue length : 0
Queue max. length : 1024
Queue dropped : 0
Netlink dropped : 0

(0005888)
lorenzo-endian (manager)
2011-03-08 15:27

Hy baldy,

I did some tests today and it seems that the rule which was causing the trouble has been eliminated.

Can you try to update your rules and check if the performances still decreases as before?

Thanks in advance

Lo

(0005928)
baldy (reporter)
2011-03-10 19:40

Hi Lo,

This morning I have started IDS again.

When started it seemed to be okay, CPU usage 3-10%.

With the autoupdate enabled CPU usage is 23%-30%, which is strange as an update feature should not have such an impact.

When monitoring CPU usage still spikes to over 90%, but not continuous as before.

I also noticed that snort is using only 1 core, while it should be multi-core/processor aware.

I will keep monitoring for a couple of days.

Regards,

Baldy

(0005929)
baldy (reporter)
2011-03-10 21:19

Still the same.

After just 1,5 hours snort cpu usage was a consistent 48-55% and throughput limited to 800-900 KB/s.

Disabling IDS resulted in an increased download speed, went up to 2.2MB/s.

When I am downloading a lot (started 85 HD, about 900GB, movies last week for testing) I am limited to around 1.1 MB/s with IDS enabled. With IDS disabled my internet connection maxes out around 6.8-6.9 MB/s.

Regards,

Baldy

Notes
(0005748) lorenzo-endian (manager) 2011-02-23 21:19	Hi Baldy, I think the problem is related, as the last time, to the rules. Did you use the rules from emergingthreats or did you create the rules manually? Thanks in advance! Lo

(0005749) baldy (reporter) 2011-02-24 08:11	Hi Lorenzo, just the default rules, with p2p disabled. Regards, Baldy

(0005750) lorenzo-endian (manager) 2011-02-24 08:40	Hi Baldy, thanks a lot for the quick reply :) I don't know the type of traffic generated by the speedtest and it could be that that specific traffic makes snort crazy ... could you try to download one (or more, in parallel) .iso file(s) or something similar (which generate high traffic condition) and check if the bandwidth is still decreased as before? Thanks in advance! Lo

(0005760) baldy (reporter) 2011-02-25 13:25	Hi Lorenzo, After stopping and starting snort and testing with normal file downloads (large files from Microsoft Network) the difference is not as big as it was. However there is still about a 33% drop in throughput with IDS enabled. Testing the internet connection without Endian, with a laptop connected directly to the modem the speed is exactly as it should be, 60 Mbps down and 6 Mbps up. With Endian in place and without IDS I get almost the same values, although this differs from time to time, probably due to other services in the LAN creating additional load. When I reported this issue my download was around 1.1/1.2 MB/s and immediately after disabling the IDS download went up to 6.7/6.8 MB/s. The speedtest itself is just simple file transfer from server to client and client to server. Details can be found here : http://wiki.ookla.com/test_flow [^] Regards, Baldy

(0005795) lorenzo-endian (manager) 2011-03-02 11:11	Hi Baldy, which is the load of your system when the IDS is enabled? and when it is disable? can you post the output of "top" in both the situations? Thanks in advance! Lo

(0005802) baldy (reporter) 2011-03-03 11:35	Hi Lorenzo, I have re enabled IDS for testing. When I initially reported the issue CPU 1 (Dual Core system) was shown at 92% in the GUI. CPU 2 only between 20-30% Top showed snort using around 95% CPU leaving the rest for the other processes. Image added is showing cpu usage within less than 10 minutes of enabling snort and only having 2 torrent downloads with the p2p rules disabled. regards, baldy

(0005803) baldy (reporter) 2011-03-03 11:43	Also added cpu usage after disabling snort/ids. Download speed in uTorrent went straight from around 750KB/s with IDS enabled to 5.0MB/s with IDS disabled.

(0005808) lorenzo-endian (manager) 2011-03-03 17:09	Hi Baldy, your support is super as all the other times :) I still suspect that the problem is related to a rule, or to a set of rules ... in Services >> Intrusion Prevention >> Rules , as you know, there is the list of the rule files ... can you try to play with them starting with all the rules disabled and trying to enable one file ad a time? Probably it will take some time but I think this is the only way to reduce (and finally find) the source of the problem... Please let me know if you can do these test, otherwise I will prepare a system and I will try it myself Thanks again Lo

(0005810) ardit-endian (developer) 2011-03-04 08:31 edited on: 2011-03-04 15:54	I remember that on one system the IDS was UP and the internet was really slow I saw that the snort chain in iptables was full and the system was dropping the packets, this because all "QUEUE" packets, were packets processed by snort (allot of traffic) and of course if you have most of the traffic passed through snort this will cause slow downs (with slow processors more). I think can help in debugging this situation.

(0005811) baldy (reporter) 2011-03-04 08:33	Hi Ardit, How can I verify this on my system ? Regards, Baldy

(0005821) baldy (reporter) 2011-03-04 12:44	Hi Ardit/Lorenzo, Re-enabled IDS and started testing. Queue increases with about 1MB/s, this will be a problem when the system is running over a prolonged period of time. High CPU usage is not due to rules, but due to the auto-update feature. If I leave this disabled CPU usage is normal (20-25%) Checked this several times with auto-update enabled and disabled. I think http://bugs.endian.it/view.php?id=3274 [^] can be reopened. Regards, Baldy

(0005826) luca-endian (developer) 2011-03-04 15:55	actually this is the right way to check your queue status: root@kenny:~ # cat /proc/net/ip_queue Peer PID : 25507 Copy mode : 2 Copy range : 65535 Queue length : 0 Queue max. length : 1024 Queue dropped : 0 Netlink dropped : 0

(0005888) lorenzo-endian (manager) 2011-03-08 15:27	Hy baldy, I did some tests today and it seems that the rule which was causing the trouble has been eliminated. Can you try to update your rules and check if the performances still decreases as before? Thanks in advance Lo

(0005928) baldy (reporter) 2011-03-10 19:40	Hi Lo, This morning I have started IDS again. When started it seemed to be okay, CPU usage 3-10%. With the autoupdate enabled CPU usage is 23%-30%, which is strange as an update feature should not have such an impact. When monitoring CPU usage still spikes to over 90%, but not continuous as before. I also noticed that snort is using only 1 core, while it should be multi-core/processor aware. I will keep monitoring for a couple of days. Regards, Baldy

(0005929) baldy (reporter) 2011-03-10 21:19	Still the same. After just 1,5 hours snort cpu usage was a consistent 48-55% and throughput limited to 800-900 KB/s. Disabling IDS resulted in an increased download speed, went up to 2.2MB/s. When I am downloading a lot (started 85 HD, about 900GB, movies last week for testing) I am limited to around 1.1 MB/s with IDS enabled. With IDS disabled my internet connection maxes out around 6.8-6.9 MB/s. Regards, Baldy

Issue History
Date Modified	Username	Field	Change
2011-02-23 19:05	baldy	New Issue
2011-02-23 19:05	baldy	File Added: Speedtest IDS enabled.jpg
2011-02-23 19:05	baldy	File Added: Speedtest IDS disabled.jpg
2011-02-23 21:19	lorenzo-endian	Note Added: 0005748
2011-02-23 21:19	lorenzo-endian	Assigned To	=> lorenzo-endian
2011-02-23 21:19	lorenzo-endian	Status	new => feedback
2011-02-24 08:11	baldy	Note Added: 0005749
2011-02-24 08:40	lorenzo-endian	Note Added: 0005750
2011-02-25 13:25	baldy	Note Added: 0005760
2011-03-02 11:11	lorenzo-endian	Note Added: 0005795
2011-03-03 11:35	baldy	Note Added: 0005802
2011-03-03 11:36	baldy	File Added: Snort CPU usage 1 of 2.jpg
2011-03-03 11:39	baldy	File Added: Snort CPU usage 2 of 2.jpg
2011-03-03 11:43	baldy	Note Added: 0005803
2011-03-03 11:43	baldy	File Added: Snort disabled CPU.jpg
2011-03-03 17:09	lorenzo-endian	Note Added: 0005808
2011-03-04 08:31	ardit-endian	Note Added: 0005810
2011-03-04 08:33	baldy	Note Added: 0005811
2011-03-04 08:37	ardit-endian	Note Added: 0005812
2011-03-04 09:03	ardit-endian	Note Added: 0005813
2011-03-04 12:44	baldy	Note Added: 0005821
2011-03-04 15:54	ardit-endian	Note Edited: 0005810
2011-03-04 15:55	luca-endian	Note Added: 0005826
2011-03-04 15:57	ardit-endian	Note Deleted: 0005813
2011-03-04 16:07	ardit-endian	Note Deleted: 0005812
2011-03-08 15:27	lorenzo-endian	Note Added: 0005888
2011-03-10 19:40	baldy	Note Added: 0005928
2011-03-10 21:19	baldy	Note Added: 0005929
2011-04-27 13:37	lorenzo-endian	Relationship added	has duplicate 0003645

Please see now our new Bugtracker system: JIRA