SETI TeAm Proxy Problems! - Let's fix it!!

[IJump]

I am now off of OKs queue and on to my home queue. That should be over 10% of the workload on that queue.

It is over 100 WUs a day out of his 915 WUs a day that go through his queue.


We have now made a start......

 

[Smoke]

I would like to know if there are some settings or adjustments to cfg or inf files that would allow my Q to download without the long pause between WUs?

I have thus far:

Disabled "Look for sweet WUs"
Inter-WU download gap = 0
Inter-WU VLAR gap = 0

Max Delays between retries = 10 (it won't take anything smaller)

My User cfg file has: Insane = TRUE

When there is great bandwidth at Berk I can still only download 17 - 24 WUs @ hour. There seems to be a 2 - 4 minute delay between each WU download where the Q just sits there. That means the upper limit on WUs that my Q could download would be just under 500 @ day (assuming good bandwidth for 24 straight hours - ha).

I am asking about this because I may be able to contribute (in a good way ) to the Q situation "IF" I can download more WUs in a 24 hour period.

 

[Adul]

isnt there some way we can use a load balancer to direct seti q traffic ?

 

[JWMiddleton]

Everyone is really trying to help and it is greatly appreciated! Currently OK's Proxy has 3,566 results waiting to be transmitted. So, things are still getting worse. Let's hope that the many recent changes start to show some positive benefit!

I prefer that we don't assign users to the other proxies; I would hope that they do it on their own. If you have a public proxy you might review the clients at OK's site and invite some to join you. Start small and build as you see how your system handles the added load. I have a private queue which handles 12 machines on a LAN and 11 remote system that dial-up an ISP for connectivity. I don't notice any ill effects on the system running the queue or to my Internet through-put.

Smokeball

Disabled "Look for sweet WUs"

I have this feature enabled and don't see a problem with it. Besides who doesn't want Sweet WUs!!

Adul

isn?t there some way we can use a load balancer to direct seti q traffic?

The way SetiQueue works it would not be efficient to do so. Each queue you connect to would have to have WUs waiting for you. It doesn?t just take them out of a big cache of WUs as with RC5.

 

[networkman]

When I get home tonight, I'll see if I can figure out what's going on with my own queue; if I can get it to connect directly to Berkeley, then that will free up roughly 70 WUs/day from going to the Federation queue.

 

[Ken g6]

I'm using the TA Cube proxy, and it doesn't seem to get clogged much. But I couldn't get to my computers to change any ports until August.

I've been wondering if our "spy" inside SSL - what's his name again? - could set up a Q on a port other than 80? That way the WUs might not be counted as SETI bandwidth and we could get more through.

 

[IJump]

Originally posted by: Ken_g6
I've been wondering if our "spy" inside SSL - what's his name again? - could set up a Q on a port other than 80? That way the WUs might not be counted as SETI bandwidth and we could get more through.

Is that MereMortal?

 

[micron]

Ok, I got all of my remote machines going through their own queue, which goes straight to berkeley!


Thanks for letting me use your queue while I needed it, Orange Kid!

 

[Smoke]

JWM

I only disabled "Look for Sweet WUs" trying to figure out why there is that pause before downloading the next WU. It didn't seem to help solve the situation and I will definitely turn that back on first when I get this finally resolved.

I'm trying to download as many WUs as possible because I have offered Cobra the use of my Q. But I will have to be sure I can download enough WUs on a daily basis to meet his needs. Thus my inquiry about getting rid of that 2 - 4 minute delay between downloading WUs.

Anyone have any ideas?

 

[micron]

Originally posted by: Smokeball
Thus my inquiry about getting rid of that 2 - 4 minute delay between downloading WUs.

Anyone have any ideas?

Can't you just set:

Inter-WU download gap 0 minutes
Inter-WU VLAR gap 0 minutes

 

[CADsortaGUY]

I'm trying to download as many WUs as possible because I have offered Cobra the use of my Q. But I will have to be sure I can download enough WUs on a daily basis to meet his needs. Thus my inquiry about getting rid of that 2 - 4 minute delay between downloading WUs.

Anyone have any ideas?

What if a few of us also allowed Cobra to use our Qs? Say getting 100-200 WUs /day or what ever we could handle. Then the "load" would be divided among a couple Queues thus not bottlenecking 1 Queue.

I'd be willing to have him connect and grab 100-200/day(maybe more once I see how well I can restock). Would this help? It would create a little more work for him cuz he'd have to make sure he didn't grab all my WU's

Is this a viable option?

CADkindaGUY

 

[Smoke]

micron, I posted this a little further back in the thread:

I have thus far:

Disabled "Look for sweet WUs"
Inter-WU download gap = 0
Inter-WU VLAR gap = 0

Max Delays between retries = 10 (it won't take anything smaller)

My User cfg file has: Insane = TRUE

CADkindaGUY, Thanks for the offer. I'll mention that to Cobra. It is my understanding that he only attempts to upload/download for a two - three hour period each day (or every few days) and I do not know if he can easily or conveniently setup different computers (clusters) to go to different Qs.

I've been studying my LOGS and it appears my Q is trying to connect to Berk quite often but gets a lot of these:

"Download WU: SendRequest to shserver2.ssl.berkeley.edu failed: Connection Timeout"

And this is during a relatively good bandwidth period averaging 30.40 MBits! :Q

Are there different servers that can be tried? Other than the above, "shserver2.ssl.berkeley.edu"? If there are, where and what would you change to direct a Q to an alternate location?

 

[Poof]

Smokeball - I think in your case, you're running up against the 1500 simultaneous connections limit on the SETI data server. I wouldn't be surprised that if your queue gets a connection and takes say 5 minutes to download 5 WUs, that's considered an "idle" 5 minutes (non-interactive) by the connection monitoring protocols and suddenly, the connection is reset. At that point, your queue is thrown back out there among the masses and must compete again to try to re-establish a connection to get more WUs. I think of it like how some FTP software works when you're downloading a big file. Some client proggies are dumb and after x number of minutes, they don't send some sort of ACK back to the server and then your FTP connection is closed (while you're in the middle of the download! :Q). It seems like this is what is happening. I don't think there's any setting in the queue that would fix that. Either the queue program would need to be rewritten to try to maintain the connection (which I doubt would happen cause we have to share that data pipe and share the connections) or SETI needs to increase the number of simultaneous connections permitted and/or get more data servers.

 

[CADsortaGUY]

Anyone know how to have 1 SetiQ connect to another SetiQ? Is it in the "Server Settings" and "connection Type" then "Proxy"?

Thanks

CADkindaGUY

 

[Rattledagger]

CADkindaGUY, yes, it's just to change connection-type to ...Proxy and type in a proxy-server and port-number on the line below. The recommended type is WinSockProxy.

 

[Migroo]

Yes CADkindaGUY. I used to connect my work SetiQ to OrangeKid's.

Exactly as you describe 'Winsock Proxy'

 

[micron]

Originally posted by: CADkindaGUY
Anyone know how to have 1 SetiQ connect to another SetiQ? Is it in the "Server Settings" and "connection Type" then "Proxy"?

Thanks

CADkindaGUY

Set your connection type to either "WinINetProxy" or WinSockProxy", then enter the "Proxy Server" and "Port" of the queue you want to flush to.

 

[CADsortaGUY]

Thanks M & M I haven't done that before so I wanted to make sure it was possible.

Thanks!

Smokeball - YGPM

CADkindaGUY

 

[Woodie]

Still have my queue prob, 11 WUs ready to go, and none waiting.

How do I repoint SetiQueue? I don't see it in the INI files...what am I missing?

 

[Orange Kid]

The main problem is that I cannot get enough connects with Berkeley.
Have beeen getting less that 500 good connections a day lately.....................If the pipe they had was big enough this would not be a problem
My best day in the past i have had close to 6500 connects with them.....easily enough to handle it all
Till they get thier new pipe, we will just have to struggle along..............to bad they couldn't let some of us have some of that internet2 bandwidth

 

[MaxSiren]

After looking at the problem and some comments people have made, I don't think doing a whole reorganizing of who connects to what proxy is feasible. It looks like the only way to do this is on an individual basis, with each person deciding on a proxy server him/herself.

So, I'm gonna offer my SetiQueue as a full alternate to OK's. At peak traffic a few months ago my server was handling about 300WUs/day, with no adverse effects to system or network performance. I'm gonna go out on a limb and predict that I can handle at least 3 times that much traffic, or 1000WUs/day. I keep very large Q-sizes (20 days worth/user) but I actively delete waiting and pending WUs older than 30 days (except when Qs are running low). I also tend to shuffle WUs around to make sure that all users are stocked as much as possible.

All I ask in return is:

• New users PM me within a few days of beginning to use the server

• Users return the results of WUs back to my proxy (as much as possible)

• Users no longer planning to use the server PM me with that info

• Users PM me ahead of time if they expect a drastic increase/decrease in production

This is simply to help me better manage the resources available. One more thing I should mention is that I frown on users connecting to my proxy just to refill their own stockpile without notifying me. When people do this, they only get WUs as quickly as my machine can download them from Berkeley, and it prevents all the other users' Q's from getting any more WUs. When people do do this, I set their account to low priority and if it continues I'm forced to ban them from my proxy.

I may be strict, but it's the only way to make sure all my users are served properly. Anyone on the TeAm is welcome.

 

[networkman]

/CS radio on

Reporting in!

Mission accomplished!

My home fleet and queue are now set to fetch/flush directly to Berkeley; hopefully, since I have a cable modem, I'll be able to access Berkeley without too much difficulty.

So anyone wanting to use the Federation queue should PM Engineer - 'cuz he won't have to set 70+ units aside for me anymore.

 

[SoulAssassin]

Noticed a few more users in my queue tonight...nice...keep 'em coming there's plenty left in the keg for everyone (we're almost out of cups but that's another story).

cwcenterprises.myip.org:5517



Side question: What if OK temp. pointed his queue at one of the other TeAm ones, dumped say 500 w/u's...dumped 500 on the next...etc. Since most of the users don't have an established queue would it cause hell or would it be a quick way for him to clear the plate?

 

[Polo]

Done. I'm not using OK proxy anymore. I've pointed my home SETIQueue directly to Berkeley. I've an ADSL connection so it's not a prob for me trying all day to get WUs.
My queue at OK was empty so it's not a prob; no WU will be lost.

If there isn't enough queues, I could set another one for the team. (ADSL, 128 up, 512 down). Just let me know.

Hope this helps a little.

 

[JWMiddleton]

Well, it looks like some nice progress is being made! OK's proxy is down to 3,411 results, which is down about 150 from yesterday's high.

Those of you who have pointed their SetiQueue directly to Berkley may want to check the option to "Report Stats to SetiQueue Server." It can be found on the Server Settings screen.