What is causing my site to be so slow at random?

[Red Squirrel]

It only does this from work, but on two completely separate networks.

http://www.uovalor.com/forum/

I run another site on the same server, yet it's fine:

http://www.uogateway.com/

Basically it takes a good 30 seconds before everything loads. The bottom task bar keeps going to "waiting for.." "looking up..." and even "read" and keeps altering back and forth, it's like if it's doing DNS lookups for each and single element even though it's one domain. Basically it will be bloody slow for the initial load (ex: within a half hour period) and then it's fine. But if I close the tab and come back to it later, it's super slow again.

What could be the cause of such sporadic slowdown? Anything I can tweak in Apache, named, or anywhere else?

 

[Gryz]

You only got the problem from work ?
I'd say firewall.

It's slow at first and fast later ? Caching. Firewall is maybe checking stuff, making it slow. But once data is on your own machine, it's fast. Then if you wait a while, the cache is not considered valid again, your PC downloads everything from the server again, through the firewall. And things are slow again.

Both sites seem equally fast to me.

 

[DaveSimmons]

Besides the firewall suggestion:

F12 in borwsers does timing now.

Google has timing tools that will fetch from your site, though I haven't used them myself.

In WHM / CPanel (hosted server software, others may offer this) you can get the top CPU usage and you can also look at the Apache current requests. For one of our sites it was easy to see that one page (that code-signed an executable with OSSLSignCode) was using 100% of one core when it ran.

 

[Red Squirrel]

Yeah I've run the firefox performance test, it does not really tell me much other than confirm that it's slow, like there's nothing that's specifically slower, it's just overall slow. And it's only when I first load it after a while. So I am leaning towards some weird cache thing, but not sure why it only does it to that site. I have other sites hosted on same server and they're fine. So I'm wondering if it's something that could maybe affect other viewers as well, but I'm thinking it may very well be some weird thing at my work. The site used to be websensed under "gaming" actually, but they removed all the restrictions a while back, but do monitor usage (I presume) so maybe that's part of the slow down too?

 

[DaveSimmons]

Client side:

Did you look at the view where it shows timing for every fetch related to the page (scripts, images, etc.)? I think both IE and Chrome offer this, not sure about Firefox.

Then run the same test away from work for comparison?

If your page is fetching 50 elements and you see fetching a large JS file takes 10 seconds at work but is local from cache and 0 seconds at home, there you go.

Also check the F12 console on the work PC, maybe you're getting a bunch of script errors?

Server side:

Like I said, you might see one page is chewing up CPU time for some reason, or that MySQL is the culprit and that is the only site running DB code.

 

[nakedfrog]

Run some traceroutes?

 

[Red Squirrel]

Traceroute is fine, it only does it to that one site, even other sites hosted on same server are fine. I think it's closer to being a layer 7 issue but can't quite figure out what because of how sporadic it is. I might need to see if I can simulate a page load and time it, then have it run at certain intervals. Might be able to coincide it with something.

 

[cytg111]

empty page, jsp / aspx / whatever your boat is floating on.. Does the problem persist? Add half a page. So on and so forth, make a binary search of it.

 

[Red Squirrel]

^ I'm not sure what that post means, I feel like some words are missing... lol. Can you explain further?

I think the issue is a combination of DNS and the actual data transmission being slow. When I look at the status bar it goes to "looking up" constantly... for the main domain. It's like if it keeps forgetting the DNS info and keeps having to requery it for each and every element.

That and sometimes it gets stuck at "read" so I think that's when it's actually loading data (ex: images). A new forum post for example, one smiley will load one at a time instead of all at once. Even the main forum page the icons on the side take forever to load. It only seems to do this at work, but it does it at home once in a while too. I'm leaning towards some kind of DNS issue just not sure what. The DNS server is on the same server as the web server. Maybe some kind of config issue or something?

Server load is not really that bad, either. I do have an app that hammers the SQL server quite a bit, but I'm really leaning towards the issue being on the apache side more then DB since it gets stuck at parts that arn't DB related.

 

[cytg111]

^ I'm not sure what that post means, I feel like some words are missing... lol. Can you explain further?.

- Yea, sure sorry.
Create an empty page on *that* server.
Load that up.
Fast / Slow?
Slow? -> Not the content that is causing the slowdown.
Fast? -> Add half a page (of a slow one).
Repeat.

If that makes sense?

Edit: also, WireShark. Sniff a session and get all the nitty gritty at the tcp level.
WireShark is an awesome tool.
Besides that sysinternals; procmon, tcpview etc may reveal hidden flaws in the system.

 

[Red Squirrel]

I sort of did that, indirectly. I have a lot of other sites hosted on that server but for some reason they arn't slow. For example:

http://www.iceteks.com - not slow
http://www.uovalor.com - slow - forum even slower. (each small gif image loads one at a time, it seems)
http://www.uogateway.com - sometimes slow

One thing though I noticed we have a lot of content that is being hosted on postimage.com, it's the default place for the forum to upload attachements and idealy I should not be relying on a 3rd party server like that, I just never bothered to change it. Maybe that's part of the issue?

I've measured with the firefox "network" tool, but it just confirms what I know - it's slow, it does not nececerily pin point what. Is there some kind of diagnostic program I can run on a site and it spits out all sorts of details?

 

[cytg111]

Block postimage.com and try again? Images wont load of course but ....

 

[John Connor]

You can use Google Chrome's Developer Tools or the addon Firebug in Firefox. Using this tool I see a 404 for the favicon and an AD from secure.img-cdn.mediaplex.com that took a little over 2 seconds. Other than that the site loaded fairly fast. You might also want to try Google page speed. https://developers.google.com/speed/pagespeed/insights/
The above the fold content happens with a lot of sites including mine.

It could very well be AD related.


If you have a cPanel, check the resource usage. Check out uptimerobot as well. It will send a SMS or E-mail on fail. You may also want to ask your host.

Have you updated phpBB? It's at version 3.1.10 now. At least try 3.1.9. Also, there is an extension I'm experimenting with right now that sends attachments to Amazon S3. You can get rid of postimage then. Thing is if you do get rid of post image that may break attachments.

 

[John Connor]

I bet you're using 3.0.x of phpBB which would be a PITA to upgrade, but can be done. Make sure you back up the Public_html folder and your database.

This link seems problematic. http://www.iceteks.com/misc/adserver/banner_728x90.php

 

[Ken g6]

You might also want to try Google page speed. https://developers.google.com/speed/pagespeed/insights/

Running that on your forum site says none of your forum images are being cached. Things like button images have no expiration date specified, so they have to be reloaded every time. Google's suggestions are: https://developers.google.com/speed/docs/insights/LeverageBrowserCaching

 

[Red Squirrel]

Hmmm that would make sense that nothing is being cached, that seems to be part of the issue, all the small gif images that are part of the forum have to load each time, one at a time, very slowly. Like when you used to have to wait for images to draw one line at a time on dialup. This is basically what happens with those.

And yeah that is a super old version of phpbb, I actually have 4 forums, two of which I turned off years ago because they were too old and kept getting hacked. It's a project I keep putting off but goal is to combine all 4 into a new SMF forum, a single forum, that I'll actually work on keeping up to date. With customizations etc it makes updating harder, but SMF actually publishes very specific code change logs which will make it easier for me to update. I can't just plop the new files as I tend to customize stuff quite a bit.

It's not just the forum that's slow though it seems to be the site itself. I feel it's some kind of apache or DNS config issue that is specific to that domain. Just not sure what.

Going to start with that google link you posted and go from there and see if any of that helps.

Oh and the domain TTL is set to 5 minutes, though while the page is loading it actually looks up the domain several times within seconds. So something seems really odd with DNS alone.

 

[John Connor]

kept getting hacked.

I would at least install Ninjafirewall. Check out CloudFlare and maybe look into CIDRAM at Github. If you have a VPS install Fail2Ban.

 

[Red Squirrel]

No it was scripting vulnerabilities that I just never had a chance to fix. SQL injection and crap like that. So it was easier to just turn the forums off until I can fix it, then got busy and next thing you know it just never got done. They were running Invision power board 1.x or something crazy old like that. So goal is to combine all the posts into a brand new forum. That's a whole other story though. I seriously need to just get that done. It's mostly setup in dev just needs to be finalized.

This particular issue it really looks like it's some kind of caching issue. I'm just not sure how to tell the browser to cache the images. Is that something I put in the .htaccess? When I load the forum first page, the small icons take like 30 seconds to load all one by one, and they are good for about 1 minute, then they have to all reload again.

 

[John Connor]

No it was scripting vulnerabilities that I just never had a chance to fix. SQL injection and crap like that.

Ninjafirewall would have prevented that. There is also an htaccess firewall you can use. If you go CloudFlare let me know, I can give some pointers.

I'm just not sure how to tell the browser to cache the images. Is that something I put in the .htaccess? When I load the forum first page, the small icons take like 30 seconds to load all one by one, and they are good for about 1 minute, then they have to all reload again.

https://www.mnot.net/cache_docs/#IMP-SERVER

CloudFlare caches as well.

 

[Red Squirrel]

I'm not sure how hosting on the cloud would do anything, the exploit was in the actual script itself, I can have all the firewalls in the world but as long as port 80 is open the stuff that is being served on it still has to be secure. But all that has been addressed. A lot of the security issues were from stuff I did when I was very new at php and did not know any better. I have a good deal on a dedicated server so I will keep my stuff hosted there as well. It's a Xeon based server with 2TB of disk space for under $100/mo. You don't find deals like that very often. You typically get a Celeron with 512GB of disk space and 1GB of ram for that price. I'd actually host it in my house so I can build any box i want at any specs I want and only pay once, but my ISP does not allow servers nor do they provide static IP blocks. (you want a real static IP if you want to do stuff properly and not rely on these 3rd party dyndns type services) But that's way off topic nor would any of that fix the issue I'm having with one particular site not caching stuff properly.

I'll look into those apache modules for the caching when I get the chance.

 

[John Connor]

Try to find my real IP address. systechforum.net. You can't, and if it ends in 215 that was old. Point is CloudFlare does caching, Ninjafirewall will block SQL injections, etc and Fail2Ban can help. Do you even have mod_security installed?

 

[Red Squirrel]

If someone can connect to your server whether there's some weird trickery like a transparent proxy blocking your IP is kinda irrelevant. Someone can still abuse a script, if there is something in it that can be abusable. At the end of the day it's the site's security itself that really matters. Of course fail2ban is good to have to block SSH brute force and stuff. I put SSH on a non default port (stops the logs from filling up the whole drive from bot attacks) and use fail2ban to block IPs.

But this is kind of off topic anyway, I think my issue has to do with caching. For whatever reason that specific site is telling the browser to not cache anything, and something is slowing down the actual data transmission of small images. (larger ones seem to be fine)

Maybe I'll just wait until I finish the new forum and see if the issue goes away.

Edit: Looks like Etag cashing is default behavior. Did not change anything on the server but this is a typical data transmission of a gif image (those are the ones that seem to be the slowest to load)

GET /forum/templates/Aeolus/images/top_left.gif HTTP/1.1
Host: www.uovalor.com
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Referer: http://www.uovalor.com/forum/
Cookie: ***removed***
Connection: keep-alive

HTTP/1.1 200 OK
Date: Sun, 16 Oct 2016 00:28:21 GMT
Server: Apache/2.2.15 (CentOS)
Last-Modified: Sat, 25 Oct 2014 23:58:38 GMT
ETag: "***removed***"
Accept-Ranges: bytes
Content-Length: 438
Connection: close
Content-Type: image/gif

 

[John Connor]

Someone can still abuse a script

If that happens on my end they have five layers to go through. Three of those are; CloudFlare, Ninjafirewall and mod_security. The others I won't mention. Scrip abuse is possible, but Ninjafirewall will help you GREATLY. It's not that hard to deploy and gives you peace of mind. Why get owned? I had many try to own me and I owned them back...in more than one way. LOL!

Your SSH port wouldn't happen to be leet, would it? LOL! This is why you should consider my CloudFlare suggestion man. Once your behind CF, change the server IP and delete the MX record and use the SMTP option in the phpBB ACP using a third party E-mail service such as from Name Cheap. It's like $20/month. Just looking out.

PS. Do you actually host the mincraft server from OVH SAS as well?

I found all this out with a simple Nmap scan. Once again. CloudFlare yo stuff!

 

[Red Squirrel]

I'm not moving my stuff to the cloud. Let alone a CDN lol. I'm not sure what kind of business you do that can justify that complexity and cost, but for my needs I'm fine with just a single server. That and I like having more control over stuff. Just looking at why one specific site out of 20+ is slow at loading images. I'm sure putting it on Amazon or Cloudflare would maybe fix the issue... but so will finding whatever configuration issue I might have that's causing it to be slow on my own server. I have a gig connection goion to that server and it's barely used, so bandwidth is not the issue. Load is fairly low too.

 22:39:25 up 733 days, 21:18,  1 user,  load average: 0.18, 0.35, 0.35

nmap is not really rocket science, I'm serving services... of course the ports are going to be open.

 

[John Connor]

No, no, no! CloudFlare is free and if you want their WAF it's $20/month. CF is not really like Amazon. Think of CF as a caching proxy only. It's a reverse proxy that uses edge servers that you save on bandwidth, etc. This site uses one called Akamai.

Just remember that if you need to change something in your forum to go into CF and enter Developer Mode and clear its cache along with the cache in your phpBB ACP.