Where to look for Untangle/Sophos UTM performance bottleneck?

[Viper GTS]

I've been playing with the idea of adding inline AV and content filtering behind my EdgeRouter fronted home network. Over the holiday weekend I installed and configured Untangle's pre-built OVA, installed a newly built VM using Untangle's installer, and installed Sophos UTM from their latest installer.

In every case the performance is terrible. I have a 300/300 FIOS connection that in reality is more like 300/350. With a single client running through the appliance VMs that drops to ~120/120 territory. Obviously this isn't acceptable. I started turning off features one by one in Untangle and saw zero difference. I expected inline AV to be the bottleneck but even when off no change.

I ran speedtest from the gui console of Untangle and saw the same 120/120 territory so it's obviously something inherent to that machine. Other machines on the same ESXi host see full 300/350 so it's not a host/vswitch related issue.

I understand that this stuff is CPU intensive, a little research suggests that a 2011 era CPU should be able to do 200-500 MB/s of inline intrusion detection. I'm running these on a Xeon 5660 based system, with two cores configured. CPU load never goes above ~20% on the appliance VM but performance is still terrible.

Where should I be looking? I'm guessing the reasonably priced physical appliances aren't going to be any better (and likely much worse), you need to go quite a few levels up the Sophos model line before you get one that is rated at 300+ mb/s AV throughput. Although Sophos doesn't make their CPU specs readily available I'm guessing they are far less powerful than two Xeon 5660 cores.

Where is the bottleneck if not CPU?

Viper GTS

 

[ylin0811]

Is it a free version of Untangle? I wonder if the free version of Untangle has a bandwidth policing that caps the ingress/egress rate. I've seen this with other vendors' demo appliances. You might want to ask their support.

 

[XavierMace]

Just to clarify, you mentioned Sophos and Untangle but all the details only mention Untangle. If you eliminate Untangle and run everything through Sophos what do the results look like? Also, what modules do you have enabled on the UTM? Keep in mind (at least with Sophos) the IPS module is single threaded so it's going to be limited by clock speed. I'm getting 120mb down at home right now through the UTM, but I'm limited by my ancient modem. I've been planning on replacing that, at which point I could see if I'm still getting limited to 120mb down.

It's worth mentioning however, that in my case, that 120mb down is with all the clients in the house running through it. 3 desktops, a handful of wireless devices, plus the servers (which admittedly aren't really doing anything on the net).

 

[Viper GTS]

I put the most effort into untangle given the simple UI - I just went down the rack turning things off one by one. Sophos was my last attempt, when I got the same results there I called it a night.

I never ran them in series I set them up in parallel and just changed the gateway on my desktop to test each one. I tested untangle twice due to the ancient ova they provided. It's so old VMware warns you that the AMD vnics are no longer supported by the Linux version specified. I rebuilt it with a modern hardware version and e1000 vnics but got the exact same results.

Viper GTS

 

[PliotronX]

Just curious, have you tried the appliance on the bare metal in place of ESXi?

 

[Viper GTS]

Just curious, have you tried the appliance on the bare metal in place of ESXi?

No I haven't. I don't really have hardware to use, I do everything but my desktop/laptop in vSphere. If I prove the concept and I am going to use it long term I may buy dedicated hardware but for now I have none.

ESXi virtualization is officially supported by both platforms, and Untangle even provides an OVA. It really shouldn't be a problem.

Viper GTS

 

[Genx87]

Is the Sophos also running AV\IPS\App control?

I will let you know in some instances Sophos IPS will flag sites like speedtest.net and destroy the benchmark. Have you tried to download a file from a known good site that can max your bandwidth to verify?

In your setup is the Sophos in front or behind the Untangle box?

 

[Viper GTS]

Yeah Sophos UTM had all the same stuff in place. They weren't sequential, my test setups were:

Ubiquiti EdgeRouter - Untangle VM - Client
Ubiquiti EdgeRouter - Sophos UTM VM - Client

It hadn't occurred to me that they might outright throttle speedtest sites, if that's the case I'm not sure how I can properly evaluate them. I have an SFTP site that can saturate my downstream bandwidth but upstream would be problematic I think.

I'll give it another shot on something other than speedtest, if I can get 300mb down through them I'll assume upload is fine as well.

Viper GTS

 

[XavierMace]

I had also found this over at Spiceworks a while back:


Chris Shipley Nov 17, 2015 at 7:33 AM

A couple pieces of advice I've received from Sophos Support regarding IPS (please feel free to add any):

• Don't enable patterns for services you don't actually have on your network. For example, if you don't run Linux on your network, don't inspect packets for linux attack patterns. IPS will inspect packets against the entire "enabled" list every time.
• 6 mo aging is a lot less intensive. If you keep systems and software up to date, you shouldn't need to protect against vulnerabilities from a year old anyway, and if you choose 12, 24 or all then you will really bog yourself down

 

[ylin0811]

Try a simple iperf test to see if you can get close to the wire speed over the untange vm. Make sure you have vmware tools installed.

If the speed looks good on iperf, that tells me the untange is doing an application inspection for your sftp traffic. Check to see if you can bypass protocol inspection.

 

[XavierMace]

Did some testing tonight so you have some comparisons. Sophos UTM running as a VM, 4 vCPU (L5640's), 8Gb RAM, SSD datastore, ESXI 6.0. VMXNET3 for the guest, QLogic adapter on the host. Speedtest runs from my desktop:

All IPS Attack Patterns Enabled, 12 months: 135Mbps down.
All IPS Attack Patterns Enabled, 6 months: 130Mbps down.
Selective Attack Patterns Enabled, 6 months: 128Mbps down.
IPS Off: 135Mbps down.

Tested with Speedtest.net and Fast.com. Enabling/Disabling web filtering had no effect. AV on and client installed on desktop. Sandbox VM hooked directly to modem (bypassing Sophos) produced same general results.

 

[xbliss]

Nice report on the tests. Would love to see updates on that

 

[XavierMace]

I'm running a slightly faster host now and ESXI 6.5, but the UTM is the same. I replaced my aging modem as well. Selective Attack Patterns Enabled, 6 months, 210Mbps down stable, bursts as high as 400Mbps. Turning IPS off seems to have very little effect. I'm paying for 300Mbps service, but the wiring on the outside of the house leaves a lot to be desired. That's with all traffic at the house running through the UTM, plus the UTM is managing a Sophos WAP.

 

[Viper GTS]

This is with 2670 v2?

I gave up and now have gigabit service so I think I may still be screwed.

Viper GTS

 

[XavierMace]

Correct. Given turning the IPS off has little to no effect, I don't think the UTM is the reason I'm not getting a consistent 300Mbps. The wiring going into the house is a complete mess. IIRC, there's a splitter installed in the line outside even though we're only using one drop. I keep meaning to poke around and see if it could be cleaned up.

CPU load on the VM is almost non-existent and the ONLY semi-noticeable difference I've seen from turning the IPS off is when playing slither.io. LOL.

 

[XavierMace]

Necro'ing this thread as we've now got ghetto-gigabit here in AZ and some of my wiring issues were resolved. With IPS on I get about 350Mbps through the Sophos. With IPS turned off, I get closer to 600Mbps. Bypassing the Sophos completely I get about 800Mbps. May end up moving it to a dedicated box to see if I can get any better, however there's no indication of it being starved for resources with the IPS off.

 

[Viper GTS]

Necro'ing this thread as we've now got ghetto-gigabit here in AZ and some of my wiring issues were resolved. With IPS on I get about 350Mbps through the Sophos. With IPS turned off, I get closer to 600Mbps. Bypassing the Sophos completely I get about 800Mbps. May end up moving it to a dedicated box to see if I can get any better, however there's no indication of it being starved for resources with the IPS off.

I'd be curious to see your results. With the spread of stupidly fast home connections the price tag for line rate firewall AV is way outside of most people's budgets. It's coming down fast, the Fortigate 60E I am running these days does a pretty decent job for not very much money (~$350 initial purchase though the ongoing support costs are painful). It seems to be somewhat intelligent in what it does and doesn't scan so even with a fair amount turned on I don't really notice the hit on gigabit.

Their true gigabit capable box though is much more than $350 and there's simply too many other places I'd rather spend it.

Viper GTS

 

[XavierMace]

Yeah, I could live with the current performance if the alternative is spending $1k on hardware. That said, the sticking point for me is I can't find a reason it's not doing better. Especially with the IPS is disabled. I know Snort's single threaded, but I'm not seeing any indication of CPU contention. A 200mbps hit with the IPS disabled seems quite excessive but livable. However since I wasn't seeing a change between IPS/no IPS before (at 300mbps service), it would seem to indicate that I am running into a limitation on the IPS module.

I've seen some people reporting they've had trouble getting good bandwidth performance running it as a VM regardless how much CPU they throw at it. I've still got plenty of spare hardware laying around including some higher clock speed, lower core count processors. That would at least let me determine if it's a limitation of the Sophos itself or if there's some sort of issue when you're running it as a VM.

What kind of throughput are you getting on your Fortigate?

 

[Hoober]

Have you guys tried Untangle's NG setup for home? $50 or $60/year (can't remember the exact amount), and you get access to all of their plugins. I'm currently running it on an old HTPC with an i3 and 4GB RAM.

 

[XavierMace]

I see no point to give them $50/year for something I can get for free.