VB6 Array limitations

[kevinthenerd]

I know, VB6 isn't the greatest thing there is, but I have to collaborate with others who use it for my new job as a research assistant.

I have a complex calculation to make, but thankfully it has a finite set of inputs: RGB data with 8 bits per channel. I wanted to make a look-up table that would store the answer as a Single precision (4 byte) number for each of the 16777216 inputs. The 64 megs of RAM this would take is well worth the effort.

As it is now, the program runs well once the table has been calculated, but I've been looking for a fast way to load the lookup table as a startup data file to speed up the process. Doing file access in 4-byte chunks is very slow, and doing it as a 64MB chunk is impossible.

Can anyone help me do this? I don't care how long it takes to store the file; it only has to be done once.

I'll show you some code so you can see what I'm talking about.

 

[EagleKeeper]

Determine the maximum chunk that the I/O system can handle;

Load that size chunk, process it load the next chunk andrepeat until EOF detected.

You also may look into starting the second I/O while processing the first chunk using an I/O thread.

 

[xtknight]

Maybe you shouldn't use a lookup table? Right now I can't see any way of making that faster...
What I suggest is you load that whole text file in at once to a huge string variable. Then, each time the program needs the data, call a function which seeks in that string variable using an equation based off of how you wrote the text file. I'm still confused about what you're using the array for though. Depending on when you "call" the array this could be faster or a lot slower. What does RGB_Function do? If this was a LUT, shouldn't you be setting an array to the data of a file? I'm really confused...

In any case, making a C DLL (and calling it from VB) is the ideal solution here. I can't understand why this wouldn't be feasible.

 

[kevinthenerd]

To clear this up for xtknight and possibly many others... The lookup table's function is to convert RGB values to hue values using the arctangent function, which we deemed more accurate than the linear method found on Wikipedia. We have a LOT of pixels to convert in this manner. Rather than performing this function on every pixel, the purpose of the lookup table is to pre-calculate every possible RGB combination (all 16777216 of them) beforehand, and to simply look it up when it's needed. Reading a piece of memory, even one in an array, is a lot quicker than performing a series of trigonometric calculations.

There is no string searching necessary in a binary lookup table. In VB6, a single-precision floating point variable requires 4 bytes of memory. Looking up a single entry simply requires multiplying that 4 by the number of entries you want to skip along. For example, if your first entry is at position 1, the third entry will be at position 9. (because 1 + (4*2)=9 ).

Loading a lookup table as a string is not a bad idea, as long as you Lset it back to a single when you're done with it. The trouble is, however, that I can't declare a custom user data structure (Type nameoftype) with a fixed length that uses more than 64k of RAM. I'm beginning to wonder if this is a limitation of VB6 or a limitation of Winblows.

 

[EagleKeeper]

Break the lookup table into an array of tables. then choose which table to use based on some key.

 

[kevinthenerd]

Originally posted by: EagleKeeper
Break the lookup table into an array of tables. then choose which table to use based on some key.

like instead of...

RGB_Array(0 to 255, 0 to 255, 0 to 255)

... I should do something like...

Public Type Red_Table
Red(0 to 255) As Single
End Type

Public Type Green_Table
Green(0 to 255) As Red_Table
End Type

Public Type Blue_Table
Blue(0 to 255) As Green_Table
End Type


Public RGB_Array As Blue_Table

RGB_Array.Blue(Blue).Green(Green).Red(Red) = Element_Stuff


Is that what you had in mind? Is that legal? How else can I do this?

 

[xtknight]

OK...what I'm trying to say is maybe just calculating them real-time will take less time? Are you sure a lookup table is the ideal solution?

 

[Thyme]

Can you use a database with an index and let the database software handle all of the searching?

 

[EagleKeeper]

Originally posted by: kevinthenerd

Originally posted by: EagleKeeper
Break the lookup table into an array of tables. then choose which table to use based on some key.

like instead of...

RGB_Array(0 to 255, 0 to 255, 0 to 255)

... I should do something like...

Public Type Red_Table
Red(0 to 255) As Single
End Type

Public Type Green_Table
Green(0 to 255) As Red_Table
End Type

Public Type Blue_Table
Blue(0 to 255) As Green_Table
End Type


Public RGB_Array As Blue_Table

RGB_Array.Blue(Blue).Green(Green).Red(Red) = Element_Stuff


Is that what you had in mind? Is that legal? How else can I do this?

Not a VB person however:
Setup a array of 256 elements.
Each element is another array of 256x256 that is the calcuations of the value using the base color as the index.

ie.
element 0 is actually a pointer to an array of all values calculated from (0,0->255,0->255)
element 1 is actually a pointer to an array of all values calculated from (1,0->255,0->255)
etc

if VB will not allow you to define an array of 256x256 elements, then follow the same logic for the second tier.

 

[xtknight]

Originally posted by: EagleKeeper

Originally posted by: kevinthenerd

Originally posted by: EagleKeeper
Break the lookup table into an array of tables. then choose which table to use based on some key.

like instead of...

RGB_Array(0 to 255, 0 to 255, 0 to 255)

... I should do something like...

Public Type Red_Table
Red(0 to 255) As Single
End Type

Public Type Green_Table
Green(0 to 255) As Red_Table
End Type

Public Type Blue_Table
Blue(0 to 255) As Green_Table
End Type


Public RGB_Array As Blue_Table

RGB_Array.Blue(Blue).Green(Green).Red(Red) = Element_Stuff


Is that what you had in mind? Is that legal? How else can I do this?

Not a VB person however:
Setup a array of 256 elements.
Each element is another array of 256x256 that is the calcuations of the value using the base color as the index.

ie.
element 0 is actually a pointer to an array of all values calculated from (0,0->255,0->255)
element 1 is actually a pointer to an array of all values calculated from (1,0->255,0->255)
etc

if VB will not allow you to define an array of 256x256 elements, then follow the same logic for the second tier.

Linked list?

 

[EagleKeeper]

Not really a linked list. He knows the size of the arrays being used; therefore can just be a static memory block. Saves the overhead of linking and stepping through. Allows direct access into the proper memory location based on the known RGB components.

The initial array elements are just pointers into another array.
You then will have a 256 pointer array with each pointer referencing a 256x256 static aeeay of values which is the result of his calculation.

 

[kevinthenerd]

Originally posted by: xtknight
OK...what I'm trying to say is maybe just calculating them real-time will take less time? Are you sure a lookup table is the ideal solution?

I'm certain. I've already tested in a very scientific manner. (I'm working on a brief paper about it along with other calculation methods I've tried.) Now I'm just looking for a fast way to initialize the lookup table so I don't have to wait too long for the program to load each time I need it.

 

[kevinthenerd]

Originally posted by: Thyme
Can you use a database with an index and let the database software handle all of the searching?

I've tried that with VB random-access file routines. It's too slow... it involves too much file access. Go ahead and add the VB-to-database overhead and you might as well calculate it every time.

 

[kevinthenerd]

Originally posted by: EagleKeeper
Not really a linked list. He knows the size of the arrays being used; therefore can just be a static memory block. Saves the overhead of linking and stepping through. Allows direct access into the proper memory location based on the known RGB components.

The initial array elements are just pointers into another array.
You then will have a 256 pointer array with each pointer referencing a 256x256 static aeeay of values which is the result of his calculation.

I have C/C++ experience. How do I use pointers in VB?

 

[Thyme]

Even with transactions?

 

[Thyme]

http://www.codeproject.com/vbscript/how_to_do_pointers_in_visual_basic.asp

 

[EagleKeeper]

Originally posted by: Thyme
http://www.codeproject.com/vbscript/how_to_do_pointers_in_visual_basic.asp

Thanks Thyme - I have no real VB experience.

My comments were based on general S/W design and experience with other languages that are more real-time.

 

[Thyme]

Originally posted by: EagleKeeper

Originally posted by: Thyme
http://www.codeproject.com/vbscript/how_to_do_pointers_in_visual_basic.asp

Thanks Thyme - I have no real VB experience.

My comments were based on general S/W design and experience with other languages that are more real-time.

Well, how about using pointers to make a binary tree?

 

[EagleKeeper]

The use of a binary tree would be three levels of branches.

If VB will allow the use of only two branches, that will speed up the access and reduce the overhead of walking the tree.

 

[EagleKeeper]

dp

Darn dial-up

 

[kevinthenerd]

Resolved! I managed to get the 64MB lookup table loaded in my VB code in a mere 0.2 seconds on this particular system using a method that actually appears quick and dirty:

ReDim RGB_Array(0 to 255, 0 to 255, 0 to 255) As Single
Open "lut.dat" For Binary as #1
Get #1, 1, RGB_Array
Close #1


and it checks out, too. I went back and verified the lookup table against the original function.

Thanks!

 

[xtknight]

Originally posted by: kevinthenerd
Resolved! I managed to get the 64MB lookup table loaded in my VB code in a mere 0.2 seconds on this particular system using a method that actually appears quick and dirty:

ReDim RGB_Array(0 to 255, 0 to 255, 0 to 255) As Single
Open "lut.dat" For Binary as #1
Get #1, 1, RGB_Array
Close #1


and it checks out, too. I went back and verified the lookup table against the original function.

Thanks!

Thank you for posting your solution, unlike the so many people here who just change their topic to 'resolved' and never report back. Cool, very useful. What format is your dat file in? How does VB know how to fill in RGB_Array? I was going to suggest just a huge block of memory, but I didn't know that was possible in VB...

 

[kmmatney]

Originally posted by: kevinthenerd
Resolved! I managed to get the 64MB lookup table loaded in my VB code in a mere 0.2 seconds on this particular system using a method that actually appears quick and dirty:

ReDim RGB_Array(0 to 255, 0 to 255, 0 to 255) As Single
Open "lut.dat" For Binary as #1
Get #1, 1, RGB_Array
Close #1


and it checks out, too. I went back and verified the lookup table against the original function.

Thanks!

Well, I saw this too late, but that what I would have suggested...using a binary file. It should also make the file much smaller.

I would do one slight change:

Open Filename For Binary Access Read As #1 ...

 

[EagleKeeper]

Looks like he created the arrays in memory and the wrote them out into a binary file.

Then just read the binary file back into memory.

 

[kevinthenerd]

Originally posted by: kmmatney

Originally posted by: kevinthenerd
Resolved! I managed to get the 64MB lookup table loaded in my VB code in a mere 0.2 seconds on this particular system using a method that actually appears quick and dirty:

ReDim RGB_Array(0 to 255, 0 to 255, 0 to 255) As Single
Open "lut.dat" For Binary as #1
Get #1, 1, RGB_Array
Close #1


and it checks out, too. I went back and verified the lookup table against the original function.

Thanks!

Well, I saw this too late, but that what I would have suggested...using a binary file. It should also make the file much smaller.

I would do one slight change:

Open Filename For Binary Access Read As #1 ...

Before, I was using random access with a Single-type 4-byte element. I believe the files are identical.