Performance of VB.NET, C with various compilers

xtknight

Elite Member
Oct 15, 2004
12,974
0
71
Edit: See my reply below this post.

-------------------------------------------------
Well the results I got were nothing short of shocking. VB.NET beat C VS8.0 by a large margin. What I did was x=cos(sin(tan(i))), increasing i from 0 to 4194303 (4194304 times). I was about to say you could amputate my foot should .NET ever be faster than C. It looks like I just lost my foot.

C (VS6.0) with math.h library: Operation completed in 891 ticks (5430734-5431625).
C (VS8.0) with math.h library: Operation completed in 1579 ticks (3954046-3955625).
VB.NET 2005 Express: Operation completed in 875 ticks (4422468-4423343).
Maybe VB.NET is faster than assembler. :evil: This is work of the devil.

It could be VS2005 C++ Express's C compiler blows chunks, so I'll do it with VS6 also.

Perhaps someone can point out a major flaw in my code, but if there is, I couldn't begin to think what it would be.

Code attached.

Edit: added VS6 results. They seem a little closer now. .NET is still faster by a hair, which is frankly amazing. I thought VC8.0 was supposed to be the most optimized C compiler?
 

singh

Golden Member
Jul 5, 2001
1,449
0
0
The Express edition lacks the optimizing compiler -- which is what gets you the speed.
 

singh

Golden Member
Jul 5, 2001
1,449
0
0
Also, there are some issues with your code/testing methods:
- Using GetTickCount() which returns the number of milliseconds since the OS started. You need to either use a high performance timer API's or increase the time interval so that the imprecision of GetTickCount() will not give you inaccurate results
- Using a floating point number as a counter
- Using a float in the C/C++ code while using a double in VB.NET (which may tilt the results in C/C++'s favor)
 

DaveSimmons

Elite Member
Aug 12, 2001
40,730
670
126
What optimization settings for the unmanaged C ?

A very good compiler could throw away the loop entirely since you're only calling standard functions (no side effects) and never actually using x in or outside of the loop.

You might want to use x += ( calc) for C and x = x + (calc) for VB. Then try different C optimization levels (speed, size, none)
 

kamper

Diamond Member
Mar 18, 2003
5,513
0
0
Originally posted by: DaveSimmons
A very good compiler could throw away the loop entirely since you're only calling standard functions (no side effects) and never actually using x in or outside of the loop.
Just curious, wouldn't that require rolling out ~4million copies of the loop? Would it actually do that for such a big loop?

I'd also be curious to see the execution profiled, particularly to see how much of the time is spent in the actual math methods. It wouldn't surprise me if the .net versions were actually implemented natively anyways.
 

xtknight

Elite Member
Oct 15, 2004
12,974
0
71
I have AMD CodeAnalyst, I'll see what I can do with it in terms of profiling. Off to try VS2003 C++ Toolkit. One thing that was different between the VC6 and VC8 code was the VC8 code was Unicode (TCHAR) (for printf). That shouldn't make any difference shouldn't it? They were being stubborn so I had to use Unicode for VC8 and ANSI for VC6

What are the types for a 32-bit floating point number in C and VB.NET? Short and Single should be identical?

Also, the reason I used the number 4194304 it's 2^22. (I went up on the powers of 2 until I found a decent benchmark duration). Maybe I'll try it even higher per singh's suggestion. What's a high performance timer API function? Such as Windows's QueryPerformanceCounter/QueryPerformanceFrequency or do I need to avoid Windows altogether and there's something else I should be looking for?

- Using a floating point number as a counter

DWORD is a floating point? Well that's what they had in the MSDN example. What's the integer equivalent of DWORD?

Right now I'm using RDTSC x86 instruction for which I found some C/ASM code. That should be as accurate as possible.

This a site to have in your bookmarks: http://www.geisswerks.com/ryan/FAQS/timing.html
 

igowerf

Diamond Member
Jun 27, 2000
7,697
1
76
In VS 6.0, did you remember to set it to Release mode? Debug mode is significantly slower. The same probably goes for VS 8.0.

EDIT: Ok. Apparently, debug is slightly faster than release in this case on my machine. I tried it using VC++ 6.0 SP6. Also, why is "i" also a float when it can be an int? I tried changing it to an int and it ran a bit faster.
 

Mark R

Diamond Member
Oct 9, 1999
8,513
16
81
What's a high performance timer API function? Such as Windows's QueryPerformanceCounter/QueryPerformanceFrequency or do I need to avoid Windows altogether and there's something else I should be looking for?

QueryPerformanceCounter will do what you want. It's basically a CPU cycle counter (actual frequency depends on CPU/chipset), so gives very high resolution timing.

If you want to use it from VB.NET, instructions are here.
 

mattsaccount

Member
Nov 30, 2003
87
0
0
Results:
VC++: Operation completed in 0.043429 seconds (121426600.000000, 2795980000).
VB.NET: Operation completed in 0.883538664797316 seconds.

Not sure if I did it right since I don't know VB.NET but if anyone would care to point out any errors I'd be happy to try again.
 

mattsaccount

Member
Nov 30, 2003
87
0
0
If I use the Intel C++ Compiler, and go out of my way to enable all optimizations, this is the result:
Operation completed in 0.026619 seconds (74426264.000000, 2795980000).

And yes, the VB.NET version was set to "Release mode"
 

xtknight

Elite Member
Oct 15, 2004
12,974
0
71
Whoa, holy crap, what's going on here?

Compiling in Debug the attached code gets:
PerfTest32-VC6
Timer method: High-resolution OS timer.
Overhead Test: 6 cycles (3806453963-3806453969)
Performance Test: 3369413 cycles (3806454058-3809823471)
Release:
PerfTest32-VC6
Timer method: High-resolution OS timer.
Overhead Test: 6 cycles (3762190465-3762190471)
Performance Test: 6 cycles (3762190583-3762190589)

I'm using QueryPerformanceCounter. What's going on here? I haven't implemented the overhead code yet so just disregard the first part of the code. If I counted in overhead, the code would have taken 0 cycles in Release. Yeah right...:confused: It's not a problem with storing my variables because I can obviously tell the results come out instantly. Is my CPU really this fast? I can't fathom why Release would make it ignore the i loop?

WOW, that's damn fast in Release. I added printf("x: %f\n",x); and determined it WAS calculating it. 0 f**king cycles? Even with 2^32 (4294967296) it still gets 0 cycles...hmm I don't believe it.
 

mattsaccount

Member
Nov 30, 2003
87
0
0
You have some pretty bizarre casting going on in that printf. Look into the code I used--you want 64 integers, not 32 bit. Not sure if that's the problem though.

Upon further inspection, I think the optimizer might be doing a better job than we're giving credit for... brb.

Ok ok: The optimizer was getting rid of most of the code. Here's a tweak that forces the code in the loop to actually run:

double x = -1;
int i;

for (i=0;i<10000000;i++)
{
// 10,000,000 times
x=cos(sin(tan(x))); // <----- NOTE tan (x) not tan (i)
}


Then, at the end, make sure to print the value of x.

New results with tweak:
VC++ 0.719272 Operation completed in 1.260617 seconds (3524660740.000000, 2795980000).

VB.NET: Operation completed in 2.48770755584804 seconds. 0.719271520859417

So, VB is only a bit slower. Not too bad.
 

xtknight

Elite Member
Oct 15, 2004
12,974
0
71
Originally posted by: mattsaccount
You have some pretty bizarre casting going on in that printf. Look into the code I used--you want 64 integers, not 32 bit. Not sure if that's the problem though.

Upon further inspection, I think the optimizer might be doing a better job than we're giving credit for... brb.

Yeah I wrote this before I saw your code. I'll have a look at your code now. Thanks.
 

xtknight

Elite Member
Oct 15, 2004
12,974
0
71
OK I attached all of the results.

Winner: VC 2003 Release, No Opt: Operation completed in 0.023115 seconds (82742.000000, 3579545).
 

singh

Golden Member
Jul 5, 2001
1,449
0
0
I should mention that a simple loop with a single mathematical operation is not even a decent indicator of performance. I would rather suggest running a sorting algorithm on a somewhat large array. That should be a much better test.
 

StormRider

Diamond Member
Mar 12, 2000
8,324
2
0
Too bad -- you got my hopes up really high but I should have known better than to think .NET can really be as fast as regular C/C++ in certain areas.
 

xtknight

Elite Member
Oct 15, 2004
12,974
0
71
Originally posted by: StormRider
Too bad -- you got my hopes up really high but I should have known better than to think .NET can really be as fast as regular C/C++ in certain areas.

Well it is, in the VS2005 Express suite using Debug NoOpt C, but that's about it. .NET wasn't as bad as I thought though.
 

mattsaccount

Member
Nov 30, 2003
87
0
0
Hehe, it was only a factor of 2 in my version of the benchmark. :) With all these high speed processors, I don't know why John and Gabe don't write their games in VB.NET! :)

singh is certainly right though, performance differences will vary depending on the benchmark used. I'm not qualified to try anything more sophisticated since I have no experience with VB.NET whatsoever, which, by the looks of this one example, is just as well :)
 

DaveSimmons

Elite Member
Aug 12, 2001
40,730
670
126
Originally posted by: kamper
Originally posted by: DaveSimmons
A very good compiler could throw away the loop entirely since you're only calling standard functions (no side effects) and never actually using x in or outside of the loop.
Just curious, wouldn't that require rolling out ~4million copies of the loop? Would it actually do that for such a big loop?

I'd also be curious to see the execution profiled, particularly to see how much of the time is spent in the actual math methods. It wouldn't surprise me if the .net versions were actually implemented natively anyways.

A good compiler could see that the calculation was never used, and so could throw the code away.

consider this:

x = 5;
x = 3;
x = 2;

A smart compiler would say the = 5 and = 3 lines have no effect on program state and are useless code that can be optimized away to just

x = 2;


that's why I suggested changing the x = assignment into a x += summation. To be safe also use the x outside of the loop in something like a cout or sprintf.
 

Kilrsat

Golden Member
Jul 16, 2001
1,072
0
0
Because of items like Dave suggested, I put together a sample that requires user input for the loop control, and does a summation inside the loop. This way no compiler can eliminate it completely.

VS .NET 2003 was used for both the C++ (standard Win32 console project) and VB.NET tests.

Both projects had full optimizations enabled, run in release mode, etc.

Using an iteration control of 123456789, I got the following numbers on my machine:
C++ - "Operation completed in 22.912992 seconds (82018087.000000, 3579545)."
VB.NET - "Operation completed in 23.2866814078326 seconds."


The VB.NET result was also more accurate ;)

 

DaveSimmons

Elite Member
Aug 12, 2001
40,730
670
126
Nice job Kilrsat , and it looks like there is no penalty to using VB.Net these days, unlike older versions of VB where you gave up speed for the rapid development.

I'd be curious to see if switching to doubles in C++ would make a slight difference (+ or -), but am too lazy to try it myself :)
 

xtknight

Elite Member
Oct 15, 2004
12,974
0
71
Kilrsat's code, Release config:

C .NET 2003 float: Operation completed in 19.344938 seconds (69246075.000000, 3579545).
C .NET 2003 double: Operation completed in 19.011113 seconds (68051134.000000, 3579545).
VB.NET 2003: Operation completed in 23.4601702171645 seconds.

I switched float i and float x,y to doubles. Is that what you meant?

I also have XP 64-bit VC.NET 2003 (native 64-bit) compile set up so I can try that as well. But only if somebody cares, because I'm lazy. :p
 

Kilrsat

Golden Member
Jul 16, 2001
1,072
0
0
I think one of the things we're really seeing is the common trend of code you believe to be the bottleneck really isn't. This is why its important to profile the application (and if you happent o be using .NET you could just write the parts that benefit from it in C++) to verify just what really needs to be optimized. Often its a fault more in your code than a limitation in the language used.
 

DaveSimmons

Elite Member
Aug 12, 2001
40,730
670
126
Originally posted by: Kilrsat
I think one of the things we're really seeing is the common trend of code you believe to be the bottleneck really isn't. This is why its important to profile the application (and if you happent o be using .NET you could just write the parts that benefit from it in C++) to verify just what really needs to be optimized. Often its a fault more in your code than a limitation in the language used.
Exaclty, a quicksort in VB will pull way ahead of a bubblesort in C++ for large lists no matter how much you tweak the C++ code.

At work we don't worry about optimizing most code for speed because it is doing tasks that are waiting on a user or a server or the operating system anyway.

Our rule is to wait until something is actually slow enough to notice, then look and see if any of that slowness is under our control (usually not).