Webbox Update and a New Pproxy Plan

[ViRGE]

Well, I've talked to Mika today(actually, it was yesterday, but its today for me), and have an update for all of you about the webbox status. The box itself is put together, and is running Win2K Srver trial version, as it awaits it final copy of Win2K Server from Steve. The design and software are almost complete too, with the content(will be emailing those who know what I'm talking about) and high-end features the only things needed to be added. We're also working on setting up some remote control software, so that if the pproxy fails in the future, it can easily be restarted(see note below).

As for the pproxy problems, Mika's narrowed those down to the Dnet keyserverfails to give it any blocks and/or fails to release the connection. We're working on trying to fix that too, so any problems should be even more unnoticeable(see below).

And that in turn, brings us to the next, and mose important point, Bphantom's round-robin. After discussing the idea with Mika, he's agreed that it's a great idea, and that a modified version of it would, and should be used on the pproxy. Under this tenitive plan, the pproxy would be the "keyserver" for TA, and the Baby Bovines(pproxies such as clueless and Jays) would be the "proxies". As such, the "keyserver" would be off-limits(all connections RRed unless to a certain subdomain) to normal users(only pproxies and high-volume users would get access), in order to take advantage of this plan, which is redundancy, and smaller logs for the "keyserver"(which translates into quicker statsruns).

Now, since this is a tentive idea involving many people, it needs to be discussed with BPhantom(should get email shortly), and the Baby Bovine operators first(BB OPs, please email me), so that any problems can be first worked out. However, if any of you non-operators have an objections/comments, please feel free to post them, so that they can quickly be answered.

But for the time being, that's the current status of TA's new webbox. If you have anything to add/say/complain about, give a hoot(or a woot), because as we're nearing completion quickly, any and all opinions will help.

 

[Lord Demios]

Man, I don't want my proxy touching that IOD crap.

Just giving you a hard time Brad, You know that you are now forcing me to open my proxy publicly (was going to do that anyway) and see if I can get permission to have it dump to mika's. I wonder how well TMRFIRE would run a PProxy with it's great uptime.

LD

 

[LeBlatt]

Restricting access to Mika's is a neat idea, but how about cows ppl have scattered all around, of which they might not change the settings ?
Some will falllback to dnet, others will crack randoms forever, or what ?

 

[Beefcake]

I agree with Leblatt, Lots of my herd I no longer have access to, I think the only thing you can do, is ask people to use the other proxies from now on and just leave the rest to use the proxy, this would still cut the hit count for Mikas!

 

[Jator]

I believe they will be redirecting mika.dhs.org, proxy.teamanandtech.com and a few of the others to the round robin address.

Jay

 

[Beefcake]

Yeh, I thought about that after I had posted, but I didnt want to log back in just to edit it

 

[Viztech]

I will lament the loss of the "hostnames working for this participant" feature, but I heartily agree that this looks like the best overall solution.

Is there any way that the baby bovines can retain the "hostnames working for this participant" function so we can manually check all of them?
That would be a small price to pay for those of us who really would like/need to know.

Would direct access to Mika be allowed by any Pproxy?

viz

 

[bphantom]

Since mika.dhs.org (204.209.128.157) is hard coded into a lot of clients/pproxies all over, Mika could possibly run two pproxies. One would be the master TA keyserver, possibly bound to special port# (or bound to a "secret" IP). The second pproxy could handle all public mika.dhs.org (204.209.128.157) traffic. The second pproxy would, just like the other RR pproxies, connect to the master TA keyserver.

A possible general rule of thumb would be:

All clients connecting directly to Mika's, should (if feasible) be redirected to the TA RR pproxies. This would guarantee all the clients stay in the TA pproxy realm and not have to fall back to D.Net.

A private pproxy would best be pointed at the TA RR pproxies, but as ViRGE said, if the pproxy is high volume, then access to the keymaster should be made available.

With a core set of TA public pproxies already available, setting up of a DNS RR under the *.teamanandtech.com domain should be easy to do very soon (I'm contacting Mika momentarily).

----

In another thread I mentioned I had created a perl script which can automatically switch a pproxy to a different keyserver and preserve the outgoing blocks. The script is finished in it's current revision and I'm currently running it on my pproxy without any problems. Before I fully release it into the wild, I would like to find a couple TA members who run a Unix/Linux pproxy that wouldn't mind testing it. My email is located in my profile.

LD, I was planning on setting up a PayPal account and transferring the $10 to the project, but now I will have to reconsider. Even so, your just jealous my domain is fully functional, while yours is parked out by the curb getting rusted.

Brad..

 

[Jator]

Brad,

If I had more time to work with Linux, I'd love to move my server over to a Linux setup and run the pProxy and stats off of that. As it stands, I will be sticking with NT (Currently using 2K Pro, but will go back to NT4 Server once LD purchases the Dualeron). If you need a volunteer to test the script on a NT machine, I would be more than willing to volunteer.

Jay

 

[bphantom]

Viztech, I'm not sure about all of the sub-pproxies (Jator already does and TheCool1 sounds like he wanted too), but I'm about to make my ppstats public. The URL will be http://stats.iodbbs.com and I will have Hostnames working for this participant enabled.

Brad..

 

[Viztech]

Stats are good... To bad I don't have them up yet... Brad..

Cool

Ummm, how do you run a port 80 proxy and a webserver on the same site? Separate IPs?

viz

 

[bphantom]

Jator, the reason I stated Unix/Linux was because I have no idea how to send a kill -HUP and kill -ALRM to a Win32 pproxy (similiar to a CTRL-BREAK). The kill sequence works great in a Unix/Linux environment.

Since I said I'd give you a copy as soon as I finished it, I will email you the perl script. I'm still impressed almost all of the code is all mine (I used the PPStats nice_gmtime subroutine), even if there are better ways to do what I did. This is my second large perl script I have made. WooMoo!

I also made a pproxy log compressor script (which I sent an early revision to Mika, whether or not he uses it is another matter ), which is automatically running now. It can compress a single days log or multiple logs at one time. I will send you that as well.

Brad..

 

[bphantom]

Viztech, yeah, I have separate IP's. I dedicated an IP specifically for the pproxy and then setup a virtual hostname for the stats on the web server. Works fairly decently.

BTW, the stats are now active. Current update is once an hour from 6am - 10pm MST/MDT (EST -2 and UTC -6). Enjoy!

Brad..

 

[mechBgon]

I don't really like the sound of that. Along with many other teammates, I contributed money and hardware to update the TA pProxy so that it will have the power to handle the load, and now it's going to be "off-limits" to small herds like mine? :| Yes, I understand that the RR pProxies will dump through Mika's. Is there going to be a problem with insufficient power at Mika's after we've added Ultra2 SCSI, another 1100MHz of P3's, and 576Mb of RAM to the task? I know I'm slow sometimes, so maybe I'm overlooking something... but I too would miss the "Hostnames working for this participant" feature, for starters.

Just my 2 cents' worth...

 

[BoberFett]

While I understand the benefits of spreading out the load, it does seem a bit heavy handed to simply deny access to Mika's proxy. I guess I'd just make the other proxies known and request that people use them. If we explain that with our current throughput, Mika's may routinely run into problems like we experienced recently, perhaps enough people would volutarily use the round robin to alleviate some of the stress on the team server.

 

[Viztech]

Mech-

I appreciate what you are saying, but another part of the mix is that Mika's proxy breaks down on occasion (as well as everybody else's at one time or another) so the redundancy of the round robin makes a lot of sense.
Some of the baby bovines will be reset to fetch work from elsewhere in case of emergency, and that alone would be worthwhile to set your herds to flush to the round robin.
I like what Jay has been doing to his stats, and if everyone in the round robin does 'hostnames' we can just look at the baby bovine stats to get our info. Heck, if you could, just point your whole herd to flush to just one of the BBs to get your hostname stats there.

I contributed to the effort as well, and I certainly have gotten and will continue to get my money's worth out of it.

As for access to Mika, I would not vote to do a 'deny access,' if for no other reason that there are a few clients/proxies out there that are looking for the IP address as opposed to the FQDN. If we point the FQDNs of mika.dhs.org and teamanandtech.dhs.org to the round robin, 99% of the traffic will go to the RR.

In conclusion, it is best not to put all your eggs in one basket, except when you need to. We will be saving Mika's server for the webserver and stats server that we all contributed money for.

viz

 

[Russ]

Even though I don't use Mika's, I have to agree with mechbgon and Boberfett. To limit the pproxy to only the "big dogs" is unfair to the real backbone of the team; the guys that have been slogging along, day after day, week after week to get us to where we are.

I don't think the argument about load and problems makes much sense either. The machine has plenty of horsepower to handle FAR more than we'll be throwing at it. When, and if, those rare times occur that there is a problem, the team still has the other pproxies to use. Nothing there has changed.

While I understand the reasoning, I don't think it's compelling enough to risk alienating the smaller herds which, to my way of thinking, are the most important element in the effort.

Russ, NCNE

 

[Lord Demios]

After a good night of sleep I felt that it was time to correct my post and my position on this mess.


First rule of thumb, keep it SIMPLE. You talk about mika's breaking, I know Brad has done a lot of work, and has gotten this stuff to work. But it still has a much higher chance of breaking!!! What ever happened with the idea of asking D.net for one of there KeyServer software programs.? The only reason we are doing this is because we run out of open connections to the keyserver. Well, why not ask Moose, or some of the D.net guys to give us a different version of the PProxy that can handle more connections. They have the source, would it be that hard to change a VAR to = 128 instead of 32? I think this idea has some neat factor to it, but it's TOTALLY beyond what we really should be doing. Mika's is for stats! We would lose half of the stats because of this. I need to see what domain is doing what blocks, that's how I know if someone has booted one of my machines. And your right, I could point it through my own proxy, but why, when we got this great machine that much hard work was put into to get it to handle our team.

This ideas has a BIG break factor to it and I think that it will be MUCH worse for many people to make many machines act like one, when One machine working as one will be fine.

Let's talk to D.net before we decide anymore RADICAL changes to the way we do things.

LD

 

[Jator]

I just want to make sure everyone realizes that they are not being cut off from Mika's, the suggestion is to add another level inbetween the client and Mika's cow. To some this is not good as they track production based on host name.

I think for now we should make it an elective participation on the round robin. The next time Mika's proxy goes ont he fritz(not the machine, just the proxy) and people are scrabling to change all their machines to another proxy, they will begin to realize the value of having a round robin proxy setup. I can't speak for the other 'baby' proxies, but I will maintain stats for every blocks that comes through my proxy.

Jay

 

[Viztech]

Russ-

You are correct in saying that the new box will have the horsepower for the present load. The problem is the proxy software and the number of connections that it allows.

To expand on the 'connection' theme, there is a definite difference in how long a connection takes when on a dial-up, cable modem, DSL, etc. The other day I dumped 63,000 blocks (with a bunch of little ones) from my cable modem proxy in ~15-20 minutes. On my dial-up clients at the shelter, it takes that long to flush and fetch 5000. Each one is a connection, and another thread on Mika's computer, even though the cable modem is 12 times faster.

The round robin, baby bovine solution will split up this load over several different proxies, so we can grow this team some more.

viz

 

[Russ]

viztech,

I understand the connection situation. Reading LD's post, I get the impression that this is a simple matter of beefier proxy software.

My concern is that any team growth that might occur will be offset by those members who may just say "the hell with it" when faced with the prospect of resetting their clients because they can no longer use Mika's. I'm also concerned that this sends the message that the little guy isn't important to the effort.

Mika's is a fabulous resource, but I don't want it to become perceived as being available to only the "elite few". I'm afraid that's what will happen if this approach is implemented.

Russ, NCNE

 

[Jator]

Russ,

Mika's would still be available for stats. All the baby proxies would still flush to Mika's to allow for a master Stats box. As for host names on Mika's, that poses a problem. I rely on my stats box to supply me with the host names, but I still flush to Mika to maintain my stats on his machine as well.

Jay

 

[Wolfie]

Maybe I am coming on a little strong here, but if I get cut off from Mika's for not having more then 600 blocks a day. Then I feel that I mean nothing. I would feel as if I was getting pushed out of the "team". As for the problem, I wouldn't know anything about it. I don't know anything about proxies or the sort. The main reason for people sending in money and parts is because everyone wants to use mika because they like it. And I would like to see the machine work for everyone. If its come down to the amount of people accessing the server, wherever Mika got the proxies software, didn't think that there would be such a demand on the server. I hope there is a way to get more people to be able to use the server at one given time. This could really hurt the team.

I know this is a lot of blah blah. But this is what I think of what I have read. Please excuse the moronic idiot typing this.


Wolf

 

[ViRGE]

Humm, I really should sleep less, so I have more time to reply, but anyways, here it goes:

LD: We do have a beefier pproxy. However, it's a beta, which makes Mika a litle hesitent to use it. We'll have to see if he can be coaxed into installing it when the webbox goes up though...

Mech, Bober, Russ, and the others: Remember guys, this is still a drawing board idea, and needs to have a few kinks worked out. Since you guys don't like the idea of being "cut-off" how's this for a compromise: proxy.ta.com will be the round-robin, and all connections to www.ta.com, mika.dhs.org, ect. will be forwarded to that. However, we will make the "master" address public, and people can use it, altough it will be highly reccomended that they don't.

In all honesty, this is more of a future-proofing and redundancy idea than anything else. By setting up the round-robin, we can hopefully cut down on CPU power needed, which may make it so we can do some more stats(like daylies), at least until OGR happens.

So, ideally, the Baby Bovine idea is the best bet to keeping ahead of the curve. But if enough problems exist, we can stick with the current setup.

 

[ViRGE]

Wolfie: It really is just an idea to try to slightly improve things at the moment. It's a little more efficent, a little more redundant, and feeds our overclocking habbits nicely.