Is this an impossible programming approach?

[AtlantaBob]

I'm currently in grad school and have to develop some small proof of concept programs for some work that I'm doing. I don't have a CS background, nor do I really need one for this work, however, I do need to make sure that this program works.

The general idea involves manipulating large grids of numbers (some integers, some floating point) with simple operations (+,/, ^) where a cell in the grid represents a different aspects of space in a buildings (e.g. how far away one cell is from another cell, etc.). I've created a prototype application using VBA and MS Excel, but a.) it's slow, and b.) I'm running into limitations on the size of the worksheet (especially the 255 column limit).

I'm wondering if I can create a similar program in Visual Basic that would use arrays to manipulate the data without incurring the overhead of Excel. However, the arrays I'm thinking about are much larger than I've seen in any programming examples. Would something like a (500)x(500)x(13) array be out of the question? If so, how would you go about manipulating a data set this size? If not, what's the maximum functional size of an array that you might use in a program of this sort?

FYI, most of the development machines that I'm working on have a gig of RAM and are running XP. I'm using Visual Basic Express Edition 'cause it's free, but if it would help, I could shift over into C#.

Thanks.

 

[igowerf]

I actually just looked this up recently. I think the largest array dimension in VB can be up to (2^64)-1, which is the size of a Long data type. A 500x500x13 array should be fine.

 

[AtlantaBob]

Thanks--I came across that earlier today... just wanted to make sure that I wasn't making a bizzare error and trying to do something in a complicated way that could be done much easier

(For instance, I was earlier trying to work on an access database and eliminate white space at the end of fields, which I was able to do with some if...then, len() and right() statements... then I asked a question about it, and someone pointed me to the trim() function.)

 

[igowerf]

Originally posted by: AtlantaBob
Thanks--I came across that earlier today... just wanted to make sure that I wasn't making a bizzare error and trying to do something in a complicated way that could be done much easier

(For instance, I was earlier trying to work on an access database and eliminate white space at the end of fields, which I was able to do with some if...then, len() and right() statements... then I asked a question about it, and someone pointed me to the trim() function.)

There might be more elegant solutions, but since you just need a proof of concept program, a quick and dirty Windows application in VB is a good option, especially since you don't have a background in CS.

 

[AtlantaBob]

Just out of curiosity...

It took 196 seconds (on a 1.70 Pentium M at full speed) to populate an 500x500 array. Does this sound about right as far as speed goes?
Does anyone have an idea of what kind of time I might except if I wrote this in C? Or would the pain of doing that outweigh the benefits?
I could always write a front end in VB and then call a C program to do the calculations if it would be faster, right?

Although it is just proof of concept, something slightly faster would be nice if possible.

 

[EagleKeeper]

Much will depend on how you prepare the data when populating the array.

As much recalculation that can be done outside the loops the better.

Without knowing the algorithms used to prep the data; it is difficult to provide an estimation.

However, 3 minutes sounds very excessive.

I have a sim that takes 60 seconds on a laptop for prep. It builds initial arrrays of 4K*4K and runs calculations of spherical distances between each reference point. to populate the arrays.

 

[AtlantaBob]

EagleKeeper,

In this case, it was just assigning the value 1 to each member of the array. No computation involved.

Woops! - I just realized that I was using debug.print to print the value of each cell -- more of just so I had an idea of where it was at any given time. Running the program with that line commented out made the execution time ~ .02 seconds. I didn't realize that debug.print could cost that much overhead.

 

[sciencewhiz]

This sounds like a perfect problem for Matlab, if you have access to it.