how can I do this?

[nageov3t]

let start from the premise that I know absolutely nothing about programming, scripting, or anything above and beyond "hello world," but I like to think that I'm a fast learner

I'm trying to think of something I can run that would reach out to a couple dozen different servers (mostly either running Redhat or Solaris) and report back either via email or a website output on the free space and oldest file in a specific directory (well, oldest file would be nice but not required)

I manage backups for a hosting company any one of the more tedious parts of my job is checking on free space and verifying retention dates on about 20 different backup enterprises (each with ~100TB storage). we have monitoring setup to send out emails when storage usage reaches a certain threshold, but they're all but useless since all of the highly utilized storage servers are perpetually at 98-99% used (because, naturally, we don't buy storage as quickly as we sell it, but we never want to delete more than we absolutely have to so that we don't compromise retention)

I'd love to come up with a way to automate my morning process of logging into 20 different servers and checking space/retention.

 

[Ka0t1x]

Because I know PHP... if you used pubkeys you could do something similar to..

exec('ssh <ip> "df -hl"');
exec('ssh <ip> "cd /path/to/files; ls -altr | tail"');

But... I see this as a risk for security..

I'm really surprised if you're working with a system that is 100TB+ that most of what you're asking isn't already logged.

 

[trexpesto]

Sure you can't specify 99.9% or 10 Gb or whatnot? It might be worth a call to support, if you have the Bottomless Support Contract for your software.

More information would help.

What is your process of checking that information now?
What is your machine OS, or can you use another machine?
Are you networked with those servers, got a user account & password you'd want to use? Avoid putting your user/pass in a script.
Would you be allowed to put a script on those servers? Shares?
Do those servers have admins? Good or Evil?
How critical/sensitive/time-sensitive is this data?

PS: why is this post so indented compared to Ka0t1x's?

 

[nageov3t]

Sure you can't specify 99.9% or 10 Gb or whatnot? It might be worth a call to support, if you have the Bottomless Support Contract for your software.

definitely something to look into! what we have in place now is monitoring software that was designed in-house, so my support contract is "until the dev's get tired of dealing with me"

What is your process of checking that information now?

currently, my process is logging into each individual machine and checking space (df -h /directoryinquestion/)... this isn't so much a problem during the M-F week because each person in the department has their own cluster of 2-3 enterprises that they're responsible for, but problems arise on weekends when you've got 1-2 people trying to watch over all 20 servers.

What is your machine OS, or can you use another machine?

my machine is windows, the machines in question are a mix of Redhat and Solaris, but I can utilize what I need to... I can't imagine anyone grumbling if I need a 5gb VM slice or something.

Are you networked with those servers, got a user account & password you'd want to use? Avoid putting your user/pass in a script.

networked with the servers, but they're strictly internal. since the backup infrastructure pretty much contains every single file on every server, it's all kept on a private network that's not connected with the outside.

Would you be allowed to put a script on those servers? Shares?
Do those servers have admins? Good or Evil?

my department in general admins these servers (though more specifically, we've got two Sun gurus who tend to take the lion's share of the admin duties).

How critical/sensitive/time-sensitive is this data?

insofar as we're talking about like petabytes of data, it's critical but the network that the servers are on is pretty well insulated from outside traffic.

 

[bhanson]

What's your job title?

Just curious how you are responsible for backups without knowing any programming or scripting.

 

[skrewler2]

first step would be figuring out the commands you type to do the above tasks manually.

probably something like

ssh server1
df -m -P /var/log | grep -v Filesystem | awk '{print "Mount point " $6 " Free Space in mb " $4}'

ls -la -t -r /var/log | tail -1 | awk '{print "File: " $8 " on " $6}'


now you can script it:


#!/bin/bash

LIST_O_MACHINES=(box1 box2 box3 box4)

for i in "#{LIST_O_MACHINES}" ; do
ssh $i df -m -P /var/log | grep -v Filesystem | awk '{print "Mount point " $6 " Free Space in mb " $4}'
ssh $i ls -la -t -r /var/log | tail -1 | awk '{print "File: " $8 " on " $6}'
done

 

[nageov3t]

What's your job title?

Just curious how you are responsible for backups without knowing any programming or scripting.

I'm a grunt; I spent the vast majority of my day troubleshooting stupid windows issues that cause backups to fail, it just so happens that the enterprise is *nix based on the backend.

 

[mosco]

You should look into Nagios. It;s not pretty but its free. since you have no programming skills, you can probably hack together some perl scripts to add any other functionality that you need.

 

[GaryJohnson]

I was doing similar grunt work at one time monitoring backups on 100+ servers. I was a "computer operator" and all the scripting/programming was done by the "system administrators."

The system admins frowned on us eating up CPU cycles with scripts on their servers. The company had a license for a reasonably fancy terminal emulator called Attachmate Reflection for UNIX and OpenVMS which supported VBA macros/scripts. So I ended up automating a lot of stuff with scripts on my client written in VBA.

 

[trexpesto]

I have to admit, Cygwin was my first thought.
If you can connect to them from your machine, you should be able to use a script from your machine which is simpler and less invasive.
Been a while, like I don't recall if the cygwin ssh prompts for login/pass?

Wholeheartedly, I would recommend attempting to engage one of the "Sun gurus" to review your work. Often they have a secret fetish for scripting.(! Don't you?)
Also they should want to make sure you don't do anything ignorant. If there is one you think is super correct in everything the do, and communicates well, that'd be the one.

BUT! and it's a Big One:

I would download Cygwin and play with it at home for a week.
As skrewler2 wisely specifies, work it out first at the bash prompt, then move it to a script. Try not to just crib his suggestions, read a basic bash scripting intro, then a guide for grep, sed, awk, tail, tee, etc. and after you've done your homework, take it back to the guru and sit with them to finish it, test, and deploy to "production." You are taking some initiative, still respecting their authoritah, building relationships and earning respect.

And, I guess watch out for them just whipping it out in 5 seconds and crushing your vibe. Ask a week ahead of time if you could get them to check out a short script you want to run from your machine, they can help you with the ssh part if you haven't got it by then.

Cool man. I went from automating a software build and CM with Make, and other scripting duties all learnt on the job, to Java Dev back in the day.

 

[trexpesto]

Realize there are many different variants of b and c shell, so not all bash documentation will directly relate to your config, but that said, I learnt much from the GNU bash docs and some books. There are lots of tutorials out there by now.
Here's the reference.
http://www.gnu.org/software/bash/manual/bashref.html

 

[nageov3t]

I talked with one of our sun guys today. he said that it should be easy enough to do with a bash script and gave me some links, so I've got some reading to do :thumbsup:

 

[trexpesto]

Cool. Cygwin is free bash and tools for Windows so good at home if you don't have a linux box. It's possible you could use Windows Command Prompt or powershell and .bat files to accomplish the same thing, but back in Win2K/NT it seemed to fall short for some essential tasks so we had to have cygwin anyway. Maybe the ssh.
http://www.cygwin.com/



.

 

[trexpesto]

I see you're still online.. This looks nice like a nice clear and short tutorial:
http://www.codecoffee.com/tipsforlinux/articles2/043.html