Webbox Update and a New Pproxy Plan

[Mika]

I'll add my thoughts to the comments here (I'll try to organize them, but they may come across as being quite random)

There's lots of really great ideas in this thread!

The only time the pproxy actually runs out of connections is when there are no blocks available (usually due to a failed connection upstream). When the pproxy runs out of blocks, clients are still connecting, trying to download blocks. The problem is that they hold their connection for 30 seconds (I think), and then release the connection, then attempt to connect again (if they are not set to fallback to d.net servers). The connection pool gets used up fairly quickly in this situation.

When the pproxy has a full buffer to supply clients, the number of connections used is rarely more than 6 or 7. The average number of connections used over long period of time is 3 (according to the status line in the pproxy's conlog). Basically, it seems like the problems are not related to the number of connections the pproxy software can handle. The problem appears to be more the way the pproxy handles upstream connections, and when problems with some upstream keyservers occur. Another problem with the pproxy software appears to be the amount of memory used when you have large buffout files, and the amount of cpu consumed (often spikes to consume 1 cpu).

I can actually think of a reason to not have a higher number of connections allowed. The more active connections, the more resources used - cpu, memory, etc.

I really like the idea of a round-robin pproxy, but I certainly think it should be optional for people. Does anyone know definatively how a RR dns works? For example: if I look up a RR record, and my machine fails to resolves it to an IP, what happens? Does my machine even try to resolve it again, or just keep trying to connect to the first IP it resolved even though it is not available. Also, if it does try to resolve the IP again, what keeps it from resolving to the bad IP again? The reason I am curious about this, is because even if I have the d.net RR keyserver set in the pproxy, there have still been problems connecting to an upstream keyserver. There was times when the d.net RR would resolve to the same (unavailable) IP regardless of how many times I pinged it.

Sorry about the length here.

Mika

update: my speeling sucks

 

[Ken g6]

What if we had each of several proxies automatically ping each other at regular intervals? Then if any proxy was down, its dns entry could automatically be changed to another proxy. Basically, this is what Jator has been doing with his proxy, but automated.

It might not solve all the problems a Round-Robin dns would, but maybe it could be set to also adjust the RR to only those proxies responding to pings. And it would probably prevent almost all fallbacks to D.Net.

 

[Jator]

Please, if there is a way to automate it, tell me. This always seems to happen while I am away from my computer and I have to point my name to Mika's server until I can get home and fix it.

Jay

 

[Lord Demios]

Jay, in linux, there is always a way. Are you using W2K?

LD

 

[CLL Sr]

Iwould be glad to change the small herd I have access to to point at the RR proxy. I have set one up (pproxy) for my home herd after the last outage and know what a problem it can be. I still need a way to track my loose cows however. I want to support this team in any way possible, so if that would help then point me in the right direction and I'll be there.
Chris aka ll6@uswest.net

 

[Moose]

Well let me offer some words of advice and some general d.net information.

We provided a custom built pproxy that had more connects. We are sad to see that Mika did not use it when the only thing different about it was the client connections. It was given in hopes that it would solve the problems Mika was having at the time.

Personal Proxies buffer blocks completely in memory hince the large memory sizes you are seeing. No matter what you do to provide flushes through 1 pproxy you will always have the memory problems you have always had due to the memory requirements for buffering as many blocks and sending as many as you do.

Put simply the pproxy was designed to simply provide a solution for individual users or companies who need to send. We are simply afraid that you have reached the limit of a single pproxy for doing stats for the load that you are pushing through that proxy. It was never intended for this purpose. If you can allocate more memory to the box you may be better off but for the most part you don't have many options.

Your best option would be to provide a RR of proxies that all flush to our keyserver network not to one single pproxy that then flushes to us. Then hack ppstats to combine multiple proxy logs and write some scripts that will transfer hourly/daily logs to one box that runs stats.

If your pproxy is not switching between IP's in our RoundRobin you might want to talk you your ISP and see if their DNS server is up to date or at least properly working. Or try another DNS server from someplace. The pproxy assuming your DNS server properly working will rotate between IPs as needed if the proxy can not be reached or has no work available.

We could still provide you with a current pproxy with more connections if you would like to try that.

I hope this is helpful.
moose

 

[mechBgon]

Moose, let me be the first to thank you and the rest at D.net for your advice, and for taking the trouble to cook up a custom pproxy for TA. I guess I suspected what you said... that we have taken the pproxy software farther than it was intended to go. I notice you aren't offering us the Keyserver software but that's your prerogative.

As for making it "taboo" for the smaller herds to fetch and flush directly to Mika's, Russ put the right wording on it: it is likely to alienate some members. Whether technically-correct or not, the solution leaves a sour taste in some teammates' mouths. Proceed with caution.

If Mika's Team pProxy is really that stressed, I may just resume fetching/flushing directly to the Dnet keyservers.

 

[LeBlatt]

If all TA heavy hitters turn to use big packets (2^33 is 16 blocks packets, i believe ?) the proxy *should* require much less memory/cpu because of the smallest numer of blocks to handle.
If this is right, it would solve most of our problems.
just a guess

Oh, and thanx Moose for taking time caring for us !

 

[Jator]

Another thought.....

.....if Mika's is being stressed by too many blocks at one time, why not set the baby proxies to flush at a certain time, each about an hour apart from the others. This gives Mika's a chance to flush out his system and hopefully reduce the amount of memory being used by the pproxy software.

However, I do not know if having the pproxy flush at timed intervals is possible (hint hint Moose ), so it might be a moot point.

LD...Yes, I am using Win2k, but if you take the DUaleron from me, I'll probably go back to NT 4 Server.

Jay

 

[bphantom]

Jator, *hint* Linux *hint*. My pproxy only connects once a hour at a set time to Mika's (via a cronjob). Otherwise it will run a few hours before it's maxkeysdone setting is reached and it automatically dumps. The way to do this is the same reason my buffercheck perl script really is only helpful in a Unix/Linux environment.

Brad..

 

[Mika]

Moose: Thank you very much for your reply. As others have stated, we highly value your input here and hope you continue to contribute to our discussions. I was under the impression that the pproxy version we had was under beta, but from your comments, apparently this is untrue. I'm not convinced that the enhanced version will solve our problem, but I'm willing to give it a try .

Mika

 

[denali]

Mika, Am I correct in stating that the only time the pproxy has problems is when it has no blocks to give. Is this due to your inablility to get through to dnet based on problems with you upstream Internet provider? If that is the case have you looked into finding a backup Internet provider that uses a different upstream path?

It also looks like part of the problem might be that about 55% of the flushing/fetching occurs in the last 4 hours of the day GMT. If we could get people to spread this out over the entire day or atleast starting around 8 EST this might prevent some of the problems. I never really looked that closely at the stats by hour before but I was suprised that 55% of the activity was in just 4 hours.

 

[BoberFett]

Denali

Part of the reason for the round robin would be to use them as backup block providers. If Mika can't make a connection, there are still several hundred thousand blocks available from the other various pproxies. I know mine has about 80K blocks buffered at any given time.

In the event of an outage, it would help to minimize the random blocks that the team encountered.

I don't think the idea of anybody here to start giving priveleges to the higher power crackers. We're just trying to make things smoother for the team so we don't run into ugly situations like that last outage.

 

[Jator]

I agree with bober....

I keep 80K blocks in my buffer which currently gives me a day and a half for the people who connect to my proxy. What we are trying to do is provide some redundency for the team in case there is another block shortage from Mika's pproxy. The round robin would make it so that people could hook in and continue to get blocks and submitt blocks without fear that those blocks be lost to Mika directly.

I think it should be voluntary, but I think the next time we have Mika's go down for an extended period of time (it always happens on a holiday weekend when Mika is away ), people will come to appreciate the value offered by having the round robin available for there use.

There has been talk before of merging stats from baby proxies before, and maybe that is something us baby proxies should explore so that peoplr can still keep there host option available to them (this would exlude Mika's as his would be the Master Stats box and he would have people connecing directly to him as well as the baby proxies).

Jay

 

[denali]

Actually I like the idea of the RR proxy, I guess what I don't understand is why these BB proxies flush to mika at all if I'm understanding how thay work correctly. As I understand it if I flush to Jator's pp it is then flushed to mika's pp which goes to dnet. The only stats I really care about are the ones on dnet as most of my blocks go directly to dnet now. Although it's fun having more current stats via mika these are not as important to me as it is to others.

What about my second point about 55% of blocks going through mika's pp within 4 hours. This would seem like a problem to me.
http://teamanandtech.dhs.org/dnet/rc5/fullstats/byhour.asp

 

[Mika]

denali: The "by Hour" stats cannot be trusted. The reason is that at the end of each day, I compress the logs. Basically, it takes all the entries from a unique email/ip/cputype/os/clientver, adds together those blocks, and makes 1 entry in the log file. This has the effect of making a 10MB log file into a 100kb log file, which reduces the amount of space the logs consume, and the amount of time it takes to process the logs. The other effect is that all the blocks you submit from a certain client get lumped into the last time that you submit blocks.

The original log files get zipped and occasionaly written to cd before they are compressed.

All the rc5 logs sitting in uncompressed files would consume about 2.5GB!! The conlog files would consume about 15GB!! I can't begin to imagine the amount of log files that the d.net servers generate.

Mika

 

[denali]

Thanks for the answer Mika. It just looked like this might cause you some problems.

 

[ZapZilla]

I just found out today that the new FBA MacOS client was released!
Whoo Hooo! My keyrate should be going up soon!

So starting tomorrow I will begin updating my 100+ comp herd to the new client.

What pproxy address should I point my herd to while I'm updating?

 

[Jator]

Mika,

Do you have to keep the con files? The ppstats as I understand don't use it. I can understand keeping a week or 2 for backup, but after that why not (pardon the pun) flush them?

Jay

 

[Possessed Freak]

of course there is a way to use win2k or NT4 to flush pproxies at a given time, here is how I would do it, open a pproxy to the public, that one will not automatically connect. make a scheduled task (hourly) that is a batch that:
1) edits the ini on the pproxy to fetch/flush. edit( better yet, copy an alternate ini over that is configured to fetch/flush)
2) restarts the pproxy.
3) edits the ini on the pproxy to not fetch and flush. edit... ditto to above
4) restarts the pproxy.

does this make sence?

 

[ViRGE]

Humm, Moose's point does add another varible to the equation though. We are running out of time on how long the current setup will last. The question is, how long can we extend it, and how long will it take to find another way. Right now, we have enough memory to last us for a bit longer, and the additional funds needed to add on more memory if nesssicary. However, once we pass the memory limit(512MB is really the most we can put in that's feesable), we'll more or less be forced to go to a plan like Moose's. So, I'd like to save some of the future trouble, and get a stable RR set up first. Then, when the time arrives when we can't effeicently use the current setup, we can modify a few things with the RR, and Mika's drops to a "proxy", the Baby Bovines start flushing to Dnet, and the logs get auto-merged like Moose said.

So, RR wise, I'm going to have to stick with my compromise plan, and that is that we forward most clinets to the RR(due to technical issues, the roaming cows will probably get to stay), which should help balence things out for the future. As for the log merging, we have the time in the future to worry about that(probably once the website settles down), and put it into implementation as soon as its ready(no real point to waiting until the last second). As usual, are there any objections to this new plan, or is it something we can all agree on?

 

[Jator]

U know me, down with OPP...

...uh, I mean, You know me, I'll do whatever the team wants.

Jay

 

[ZapZilla]

I'm starting my upgrade and since no answer I'm keeping it at mika.dhs.org...If it should be otherwise let my know before I get too far in...

 

[bphantom]

PF, mercy! That sure sounds like a lot of work in Windows. Doing this in Linux is as simple as putting one command line in cronjob. Just giving you a hard time. If thats how to make it work in Windows, then great!

Brad..

 

[bphantom]

Zapzilla, I was hopeing to get you an answer, but I haven't heard back from Mika yet (he's pretty busy). I was going to suggest proxy.teamanandtech.com, except a nslookup is telling me it doesn't exist right now. I'm still not sure what we are planning on calling the RR hostnames (proxy.t*a*.com, proxy23.t*a*.com, etc??).

Brad..