SETI TeAm Proxy Problems! - Let's fix it!!

[JWMiddleton]

I have watched OK proxy clinging to life for the past few weeks. He has over 3,000 WUs results queued for submission. WUs that would help our daily numbers!

His proxy is set to upload to Berkley between 11:00 AM and 10:00 AM daily. That is a 23-hour window for a total of 1,380 minutes per day. The Inter-WU download gap is set to 1 minute (the minimum allowed without defeating the feature.) So, if it took 15 seconds to get a ~340k WU then, with the wait time, OK could only retrieve 1,104 WUs per day (1,380 minutes divided by 1.25 minutes.) So, just to replace the ~3,000 WUs that are waiting to be returned it would take nearly 3 days! This does not allow for returning results or having you download your WUs and upload results. You would hope that the latter functions could happen during that one minute gap. We must realize that going through a Proxy requires twice the bandwidth as each piece of work is handled twice.

The bottom line is that OK?s Proxy is oversaturated! TeAm members need to establish their own proxy or spread the work to other proxies. Poof started a thread many months ago letting you us know what Public Proxies are available. We need to use that info, but it needs to be managed.

I would like to suggest a few guidelines:

1. Limit each proxy to 100 to 200 WUs (RPD) per day
   
2. Assign (dedicate) a Port to each user to better manage incoming traffic. This will get rid of those horrible dialup addresses that change each time you call in. For example my Mother uses port 5522 and my Dell notebook uses 5521. I can then name them anyway I prefer and they don't change.
   
3. Setup your own SetiQueue and go directly to Berkley if you have a full-time connection
   
4. Set the Priority to ?Upload Results?. Results are very small and when one fails to send then SetiQueue will Get a New WU, rather than sit idle.
   
5. Set ?Inter-WU download gap? to 0 minutes and set ?Inter-WU VLAR gap? to a higher number. Mine is set to 5 minutes. Many times WUs come in groups of like kind. So, if you get one VLAR, then you are likely to get anther one right behind it! (Let?s spread those buggers around!)
   
6. Have fun and let?s get those results returned where they should be!!!

I hope I have not stepped on any toes! Please believe that I am willing to help in this process anyway that I can!

 

[RaySun2Be]

I'm already helping the bottleneck. I found out during the first major Berkeley bottleneck that my personal and work Qs were able to upload/download better than OKs, so I haven't been using his Q since then.

Too bad the team public SETIQs can't be setup in a Round Robin like the RC5 Proxy Servers. That way if one gets clogged, you can take it offline, and the other Qs would take up the slack.

 

[Crazee]

I think everything you have stated is great advice JW and I don't think you stepped on anyone's toes

I have a couple of remotes that I am in the process of moving off his queue and on to another.

 

[IJump]

You stepped on my toes, man you have big feet....



I am in the process of setting up a Setiqueue at my house. Hopefully that will aloow me to get my 100 or so WUs a day off of OKs queue. That should help some (help him, me, and the TeAm).


Good suggestion JW.........

Too bad the team public SETIQs can't be setup in a Round Robin like the RC5 Proxy Servers. That way if one gets clogged, you can take it offline, and the other Qs would take up the slack

Maybe time for me to try to write another program....the "The Advanced SETI Queue Manager and Round Robin Connection System" (or ASQMARRCS for short. , that is pronounced ask-marks..... )....... I just might look into that..... Anyone can feel free to PM me with suggestions.....

 

[LastKnight]

With JW's help, i was able to set one up at work, and at home, hopefully offloading 30-40 a day off OK's server. SQ is not difficult to set up at all (that is, unless you have a multi-subnet network, as I found out, and even then, it's not too bad) and work wonders for storing WUs for those long berkeley winters ahead.

As it is, OK's server worked great for me while I was a newbie to DC, easy to set up and use. I suppose it's just time to make room for a few more newbies *sniff. Thanks for all your help, JW and OK.

 

[Smoke]

Very good advice, JWM. Long ago I setup my mini-team with its own Q. There are times I am barely able to keep up. I started to offer Cobra a Q service since I'm on 24/7 and he only tries to contact Berk for 2 - 3 hours but I'm not sure I could handle 500+ WUs @ day so I have held off doing that.

BTW, how do you establish a 2nd Queue? I only have Q0001 for TeAm Smokeball and I wouldn't want to mix a lot of other people up with the team members. I need to monitor them to keep things flowing ... if you know what I mean.

I believe all of this will be a thing of the past when Berk gets that 100 MBit pipe working in the next month or so. /fingers crossed/

 

[conjur]

Originally posted by: IJump
Maybe time for me to try to write another program....the "The Advanced SETI Queue Manager and Round Robin Connection System" (or ASQMARRCS for short. , that is pronounced ask-marks..... )....... I just might look into that..... Anyone can feel free to PM me with suggestions.....

I'd be a bit squeamish

Seti
QUeue
Efficiency
Manager
for
Information
for
Seti@
Home

SQUEMISH (I know...missing the A)

 

[Confused]

Greg (and others),

you can set up two separate Seti Queues to run simultaneously, say have port 5517 for your Team's queue, and then say port 5001 for a public queue. That way all your own team ones can be kept separate and you can keep track of all of them

Running another copy of SetiQueue in another directory is very simple, and requires hardly any overhead on the computer


Confused

 

[JWMiddleton]

Smokeball,

SetiQueue automatically creates a new queue under the following conditions:

• A new user sends to the queue - using a different SETI ID than Smokeball's
• A Smokeball user has a different platform - such as Linux, Sun OS, Mac, etc.

After posting I saw Confused's response. Now not sure which you meant, but one SetiQueue could handle it all. You can have a range of ports for the users on the same SetiQueue. You would only need to dedicate a port for dialup users who do not have thier own SetiQueue. With two SetiQueues talking to each other you can get the name correct without a dedicated port.

 

[dmcowen674]

Looks great JWM. The spreading out of Users and Queues should alleviate the bottlenecks wherever they may be. Wish I had the problem of saturation , no where near even on my pathetic 24K connection and 2 queues. The connection does stay up 24/7 though.

 

[MaxSiren]

Returning results is important, but let's not forget that so long as everyone is supplied an adequate supply of WUs, all the results will eventually be returned. That's why I tend to favour setting my public Q to focus on downloading new WUs, and keeping all users well stocked.

The problem we're seeing is that OK's Q has so many users that HIS bandwidth runs low. This isn't a criticism...if everyone used my Q, I'd have the same problem. When Berkeley starts showing problems, even MORE people start connecting to OK's Q, and the problem gets even worse. Also, when Berkeley is dropping connections, there's no difference between downloading a WU and returning a result, cause the difficulty is in establishing the connection. I know we're a big successful team, but we still account for a very small portion of all the traffic Berkeley sees...so refraining from connecting to them won't really help.

The Backup Public SetiQueues were meant to support the team when times were tough and OK's Q is overrun, but we've gotten so large and productive that I don't think a single Q can support the TeAm even when the pipes are wide open. I like JWM's suggestion of spreading the load. I'd go a step further and suggest that we start assigning some of OK's users permanently to other Qs, so that the TeAm is more evenly distributed.

We should talk to the various owners of our present Qs, and find out which would commit themselves to providing this service in a reliable, and longer term way.

Edit: I just took a look at OK's Q stats, and there are only 299 WUs left in all the Qs...unfortunately this means that a lot/most of the remote systems that connect to our TeAm SetiQueue are going idle. This means a real loss in production, not just a delay in when the results are returned. We have to do something about this NOW!

 

[SoulAssassin]

I wholeheartedly agree on what everyone has said...I've mentioned numerous times in the past that we have all our eggs in one basket. Anyone is welcome to join my queue at cwcenterprises.myip.org:5517 or one of the other many public queues. I took a moment to browse the queues at OK's the other day and noticed many, many empty.

Long story short, if you're on OK's machine then you should consider moving at least some machines to another public queue. Now.

 

[m2kewl]

Most of my WUs have been diverted to Engineer's Federation Q for a while, thanks Engineer!

 

[onebadv6]

exactly why I posted last night. I have a very modest production of 7-11 units a day, didn't think much of jumping on OK's queue when I couldn't direct connect to Berkeley. Now after running out of work units and seeing the problem of OK overload I learned very quickly how to set up setiqueue.

I love the idea of assigned queue and think that could work out really well. That is if some one has the time to catalog the users that are currently on public queues and relocating them to even out the work load.

I'm willing to take on some of the burden.

www.nebra.org:5001
onebadv6 setiqueue

Scott

 

[networkman]

Most of my WUs have been diverted to Engineer's Federation Q for a while, thanks Engineer!

Same here! In fact, I often find that I'm out-producing the Federation itself!

 

[CADsortaGUY]

I'm willing to take on some of the burden.

Same here. I think that this may all be a mute point in a few weeks but for now lets get those WUs spread out.


MysteryMachine Public Queue
mysterymachine.hn.org Port:443
CADkindaGUY

 

[ViRGE]

If someone wants to run another full-time proxy like what OK does, I could see dividing up the proxies to serve people by name. Everyone A-M would use OK's(a-m-setiproxy@teamanandtech.com) and N-Z would use the new volunteer's(n-z-setiproxy@teamanandtech.com). While it would be voluntary of course, I figure we could easily divide up the users to the point that we can alieviate things.

 

[onebadv6]

I think what would be best if we can get avg rpd for each user for all the public queues. Then devide everyone up so that each public queue is balanced. Currently there are 16 listed public queues so each one would get 1/16 of the total output. The only problem is it is going to be a bitch to organize

:Q

 

[Woodie]

My home SetiQueue (not public, I'm on dial) is out of WUs. This is the first time, and it appears that the pass-through is failing:
Failed Connect to shserver2.ssl.berkeley.edu:80 (Host not found)

Can I point my setiqueue to another queue? Or do I have to wait for Berkeley? I'm only doing 1-4 wu/day, which just recently went up, when I fixed a heat problem on one machine.

Regards,

 

[micron]

It looks like it's time to take my remote clients off of OK's queue and start my own queue for them. I'll try tommorow!

 

[CADsortaGUY]

Originally posted by: Woodie
My home SetiQueue (not public, I'm on dial) is out of WUs. This is the first time, and it appears that the pass-through is failing:
Failed Connect to shserver2.ssl.berkeley.edu:80 (Host not found)

Can I point my setiqueue to another queue? Or do I have to wait for Berkeley? I'm only doing 1-4 wu/day, which just recently went up, when I fixed a heat problem on one machine.

Regards,

Yeah - just point them to a Queue and if the pass-through for the Queue doesn't work just PM or email the "Queue Master" and ask them to put some WUs in your Queue

CADkindaGUY

 

[IJump]

As one of the biggest users of OK queue, about 150 WUs a day on my good days, probably higher this summer, I will probably be off of his queue in the very near future. At last count, I had about 10% of the back logged WUS that came from my work. I have already taken my home machines off of his queue and will be taking my work client SETIQueue off tomorrow if all goes well. That will at least be a step in relieving his queue.

 

[networkman]

Oops.. I guess I shouldn't've bragged.. can't connect to the Federation queue now.

 

[Confused]

I am willing for people to use my queue, i do not use most of my bandwidth when i am here, and i'm not here all day due to college (soonish to be work), and overnight in the UK, which is day time/evening in the US


teamanandtech.d2g.com port 80


Confused

 

[sduguid]

I mostly agree with what people are saying, but it should be kept in mind that some folks will not be able to access remote machines to change queue addresses. I moved my remote machines from OK's to MaxSiren's queue several months ago to help(in a small way) alleviate OK's bandwidth problems. However, I cannot access these machines easily, if at all. This would make it nearly impossible for me to participate in a scheme where I had to move based on alphabetic order or some other determinant.

I'm not saying that this isn't a good idea - i'm just trying to note that it will be logistically tough for some.

Since we're talking about queues, i'd like to thank MaxSiren for the continued use of his SetiQueue!