Producer/Consumer Problem

RedArmy

Platinum Member
Mar 1, 2005
2,648
0
0
This is my last assignment for the semester and I'm having a bit of a problem going about it. I know what I want to do, just not 100% sure how to implement it. There's a lot more involved then what I'll mention, but that stuff I know I can do.

My basic problem is that I have a single reader thread and single writer thread. The reader thread constantly grabs data from a text file and dumps it into a buffer of some arbitrary size. The writer thread will constantly be taking the data from the buffer and writing lines of it out to a separate text file.

The only major conditions that has to be considered is that if the buffer is maxed out, the reader thread has to cease operating until some of the existing data has been moved out of the buffer. Also, if the buffer is empty, the writer thread has to halt until data is in the buffer again.

I want to implement a semaphore to tell the respective thread to either wait or continue based on the condition of the buffer. I just don't know how to go about checking the current size of the buffer to know what to do.

If you need any more information just let me know and I'll try to give an answer as best I can. It would be preferable if I could do this in C++ but I can also use Java if it turns out to be easier (which it is in some cases for these sort of things).

Thanks in advance for any help.
 

degibson

Golden Member
Mar 21, 2008
1,389
0
0
I want to implement a semaphore to tell the respective thread to either wait or continue based on the condition of the buffer. I just don't know how to go about checking the current size of the buffer to know what to do.

Sounds like a queue<Comsumable> to me.
Wrap the queue<> with a lock and a condition variable.

Assuming Posix threads:
man pthread_cond_wait
 

Crusty

Lifer
Sep 30, 2001
12,684
2
81
Yep, a blocking Queue is really all you need. When the consumer tries to read from the Queue you block until there is an item available and then return, when the producer is ready to queue more data you block until there is space in the queue and then submit.
 

Kirby

Lifer
Apr 10, 2006
12,028
2
0
Get ready to pull your hair out when it deadlocks when you don't get something exactly right. :p
 

Cogman

Lifer
Sep 19, 2000
10,283
134
106
Having implemented a semiphore, you do NOT want to do this yourself. Your OS has one of the best mutexes available, use it.

As has already been stated, your best bet is to wrap up a queue and just use that.

If you can't use a queue, Then I would suggest using a zero terminated char*. Have the read thread write at the first zero it sees (if your data stream has 0's in it, you are going to need to add some sort of escape charactor handling code). The write thread should just grab the full string (all the way up to the first null charactor) and process that, it should set the buffer to zero when it gets a hold of it.

So your flow should look something like this (with a char string)

...Readthread...
Lock the buffer
find first zero
replace zero with read in charactor
unlock buffer
do other stuff

...WriteThread...
Lock the buffer
find the first zero
copy all data up to the first zero into personal buffer
zero all charactors in buffer
Unlock buffer
process data.
do other stuff

You should note that I wuold have you copy the data into your own personal buffer and then unlock the buffer. That is purely because disk IO stuff is slow, there is no point making the read thread wait for the write thread to write to the disk. As well, writing chunks to the disk is much faster then writing one char at a time.
 

degibson

Golden Member
Mar 21, 2008
1,389
0
0
Get ready to pull your hair out when it deadlocks when you don't get something exactly right. :p

Haha, I think that is the point of the assignment!

If you can't use a queue, Then I would suggest using a zero terminated char*.

Encoding the length into the data itself works fine for ASCIIz's, but solely by convention. If for some reason a queue<Whatever> cannot be used, it would be better to do something like this:

struct Blob {
size_t len;
char data[];
};

In other words, spend four bytes encoding a length instead of CPU time to pack/unpack data with escape characters.

Then make your entire 'queue' a byte buffer with a head and tail pointer. Depending on your architecture, you may have to make sure that the size_t is properly aligned.
 

Cogman

Lifer
Sep 19, 2000
10,283
134
106
Haha, I think that is the point of the assignment!



Encoding the length into the data itself works fine for ASCIIz's, but solely by convention. If for some reason a queue<Whatever> cannot be used, it would be better to do something like this:

struct Blob {
size_t len;
char data[];
};

In other words, spend four bytes encoding a length instead of CPU time to pack/unpack data with escape characters.

Then make your entire 'queue' a byte buffer with a head and tail pointer. Depending on your architecture, you may have to make sure that the size_t is properly aligned.

lol, duh, and here I thought I was being clever :). Yes, use degibson's method, not mine. It will be faster and less prone to errors.
 

RedArmy

Platinum Member
Mar 1, 2005
2,648
0
0
I feel like punching myself in the face. I'm so confused at this point that I'm not sure what to do. I was able to create this short program here: http://pastebin.org/61227 in Java that outlined the basics of what I need to do.

I then tried to take that and implement it using threads and what not but I suck so hard at Java it's not even funny. There's like 3 billion errors in http://pastebin.org/61230 this version. I commented out the sections that I have no clue about. Hopefully the rest of it is close to what I want =/.

Basically I have a buffer that reads in 1KB (exact size of one line in the text file) and then that get's past into a larger buffer that's 10KB. I have a counter that keeps track of how many lines gets read into that large buffer. If count reaches 10, it halts.

For the program where I use threads I plan to implement this count variable to determine if my buffer is empty or full and do the appropriate tasks (e.g. read more lines into the buffer if it's empty while halting the write thread or halt the read thread if the buffer is full and let it write some out first) of which I have no clue how to do =(.

I hope someone can point me in the right direction as I understand clearly what I want to do, just not how to actually go about doing it...plus I fail at teh syntax.

Basically you can think of that second link as a really shitty outline. Thanks for any help.
 
Last edited:

degibson

Golden Member
Mar 21, 2008
1,389
0
0
I hope someone can point me in the right direction as I understand clearly what I want to do, just not how to actually go about doing it...plus I fail at teh syntax.

1. If you're not comfortable in Java syntax, don't use Java. You mentioned in a previous post that you want to use C++. So use C++.

2. If you enumerate here in the forum "what you want to do", we'll be glad to help you actually go about doing it. However, its not clear to me exactly what you want to do... so I hesitate to fill you mind with potentially misleading advice.

For the program where I use threads I plan to implement this count variable to determine if my buffer is empty or full and do the appropriate tasks (e.g. read more lines into the buffer if it's empty while halting the write thread or halt the read thread if the buffer is full and let it write some out first) of which I have no clue how to do =(.

3. Look into condition variables. If you continue to insist on using Java, a google of "java condition variable" will get you the reference you need to halt your threads on empty/full condtions.
 

RedArmy

Platinum Member
Mar 1, 2005
2,648
0
0
Ok, I'm making progress. I can read and write strings by themselves, but now they have to come from files. My question is, how can I make it so I'm not getting an IOException where I have to throw or catch on these highlighted lines http://pastebin.com/m598c96b2 here. I've tried putting them in a try block with a catch afterwards, but then it would say out1 or fstream aren't recognized when I call them.

Is there a way I can keep all these in the main method and just pass by reference? Sorry if it seems like a n00b question, thanks!
 
Last edited:

Borealis7

Platinum Member
Oct 19, 2006
2,901
205
106
whats the point if you only have one reader thread and one writer thread?
just have one wait for the other.

maybe i misunderstood your assignment.
 

Cogman

Lifer
Sep 19, 2000
10,283
134
106
whats the point if you only have one reader thread and one writer thread?
just have one wait for the other.

maybe i misunderstood your assignment.
It is called arbitrary homework to teach a point, in this case, threading.
 

EagleKeeper

Discussion Club Moderator<br>Elite Member
Staff member
Oct 30, 2000
42,589
5
0
I remember doing this with a continious paper punch/reader in assembly.

Read to fast and you would rip the tape from the punch, read to slow and the paper tape would hit the floor, collect dust and foul up the reader.

Interesting class assignment