Need Help With Bash Script

[Netopia]

Hey Gang,

I've got a situation where an old (non-linux) mail server crashed. I've got mal for a bunch of people, but the way this server worked, it's in all sorts of different directories, and each mail is split into two pieces, a .hdr (with all the header info, like TO: From: and Subject and .msg file that has all the actual message info. The .hdr and .msg both have the same filename, with a different extension and are always both in the same directory.

I've got all the folders, what I'd like to do is to search for an individual's email addres in the .hdr message (grep) then identify the file (find) and copy it and it's counterpart .msg message to another directory (cp). Once that's done, I want to create a new file that is a concatenation of, first the .hdr and then the .msg and have it named the same as the .hdr and .msg, but with a .fin extension.

I know that this should be very doable from the bash command line using just the four tools listed, but I am woefully ignorant of proper syntax to bring it all together.

ANY help would be GREATLY appreciated. Heck... I'm thankful if you've just read this far!



Joe

 

[cleverhandle]

Untested...

for header in `grep -l 'email@address.com' *`; do
NAME=`basename $header .hdr`
cat ${NAME}.hdr > /target/dir/${NAME}.fin
cat ${NAME}.msg >> /target/dir/${NAME}.fin
done

Should get you going in the right direction...

 

[Netopia]

Hmmm... gave me some ideas, but I can't get it to work. Complains about the "do" statement for some reason. (and I changed the things like "email@address.com" to the real deal.

Any other thoughts?

Joe

 

[cleverhandle]

Since this is apparently being used for actual work, I won't play "spot my mistake", though that would obviously be more fun.

My first line is wrong because it will output both foo.hdr and foo.msg, which will muck up the rest. Change the * to *.hdr and that should work. It tests out OK on my machine at least.

 

[Netopia]

Hmmm....

I thank you for your help and patience. I turned the script into this:

for header in `grep -l 'crnagan' *.hdr`; do
NAME=`basename $header .hdr`
cat ${NAME}.hdr > /temp/mailout/${NAME}.fin
cat ${NAME}.msg >> /temp/mailout/${NAME}.fin
done

But when I run it, I still get this:

[root@cranium BACKUP10]# ./grabmail.sh
'/grabmail.sh: line 1: syntax error near unexpected token `do
'/grabmail.sh: line 1: `for header in `grep -l 'crnagan' *.hdr`; do

====
From what I've been able to read about "for....in....do" today, I don't see anything wrong with the syntax.

Joe

 

[Nothinman]

Can't get the f'ing attach code button to work, what's so hard about implemeting proper code tags?

Anyway, I just ran this and it works here although I only had one hdr and one msg since I had to create my own:

for i in *.hdr; do NAME=`basename $i .hdr`; cat $NAME.hdr > $NAME.fin; cat $NAME.msg >> $NAME.fin; done

 

[Netopia]

Thanks Nothinman. It doesn't seem to work for me either.

I think part of the problem is that I am running this in a directory, but it needs to recurse through all the sub-directories.

Here's what I got when I ran it:

[root@cranium BACKUP10]# for i in *.hdr; do NAME=`basename $i .hdr`; cat $NAME.hdr > $NAME.fin; cat $NAME.msg >> $NAME.fin; done

-bash: *.fin: No such file or directory

-bash: *.fin: No such file or directory

I've been trying to work off of something like this:

find . -exec grep -l "username" '{}' \; -print

That shows me where all the files are that have that person's email in it. but if I change it to:

find *.hdr -exec grep -l "username" '{}' \; -print

but then it doesn't find anything because there aren't any files in the root of the working directory. So, I changed it to:

find ./*/*.hdr -exec grep -l "username" '{}' \; -print

but then it errors out after a bit and says that there are too many arguments.

I've also tried just grep, like

grep -l "username" ./*/*.hdr

but it too errors out with too many arguments.

I'm starting to feel like a BASH retard! I know this is doable (or at least it seems like it should be) but I don't know enough and keep running myself into dead ends.

Thanks for any brain cells put into use.

Joe

 

[Nothinman]

Here's what I got when I ran it:

Something else happened then, somehow your shell didn't expand the glob *.hdr. I assumed that the files were in one directory, possibly with multiple directories holding hdr and msg files, but a single directory containing the pairs.

What's the directory structure really look like and where are the files in that structure?

 

[Netopia]

Main Directory (not .hdr or .mgs files loose in this directory)
Lots and lots of directories under the Main Directory
.hdr and .msg files in various numbers in the sub-directories that are under the main directory
pairs of .hdr and .msg are always together in a directory

So... only two levels of directories, with all the files in the lower level.

I'm wondering if the * didn't glob because there were no filenames to expand in that directory.

Joe

 

[cleverhandle]

Originally posted by: Netopia
Main Directory (not .hdr or .mgs files loose in this directory)
Lots and lots of directories under the Main Directory
.hdr and .msg files in various numbers in the sub-directories that are under the main directory

Well, geez... you could have said that earlier. Add an "-R" to my grep then. But the errors you got (unexpected token) don't even look like they're related to that. Maybe you typoed or maybe you need some quotes on some of the strings (to protect spaces or special characters in the filenames).

 

[Nothinman]

So... only two levels of directories, with all the files in the lower level.

So run it in the lower level directories, the directory holding the .hdr is also always holding the .msg, right? If that's true, just run the script once per directory.

I'm wondering if the * didn't glob because there were no filenames to expand in that directory.

Yea, shell globs aren't recursive.

 

[Netopia]

Yeah, I guess I wasn't clear when I said that they were in lots of different directories, that all the directories were together.

Sorry.

Joe

 

[Netopia]

Originally posted by: Nothinman

So... only two levels of directories, with all the files in the lower level.

So run it in the lower level directories, the directory holding the .hdr is also always holding the .msg, right? If that's true, just run the script once per directory.

I'm wondering if the * didn't glob because there were no filenames to expand in that directory.

Yea, shell globs aren't recursive.

Yikes... there are hundreds of directories!

Hmmm.....

 

[cleverhandle]

So, again, just do a grep -R -l if you want to search for a specific address, or do an ls -R if you just want to operate on all the *.hdr files in one run. Provided your OS's tools support those options, those seem like the simplest ways to me.

 

[Netopia]

Originally posted by: cleverhandle
So, again, just do a grep -R -l if you want to search for a specific address, or do an ls -R if you just want to operate on all the *.hdr files in one run. Provided your OS's tools support those options, those seem like the simplest ways to me.

I'll give it a shot, thanks!

Joe

 

[Nothinman]

Yikes... there are hundreds of directories!

So do this:

for dir in *; do echo "Processing $dir"; cd "$dir"; /path/to/script; cd ..; done

The only problem with that is that if there's more than just directories there it'll get confused, so if there are files there in the top directory too let me know and I'll add an 'if [ -d $dir ];' to it.

 

[Netopia]

There are files, but if it would make things easier, I'll just yank them and put them in another directory for holding. These are just "dead" files and directories now, the only thing happening with them is me, feably attempting to use Linux (beyond my abilities) to resurrect people's email for them.

I'm currently reading through a thousand page tome on Bash Scripting... but things that seem to be hard coded for some of you (like shell globs not being recursive) I think are going to take me a lot more repetition to get written into my brain!

Thanks again for all the help guys. It became aparent really quickly that I could not have gotten very far by myself.

Joe

 

[Nothinman]

but things that seem to be hard coded for some of you (like shell globs not being recursive) I think are going to take me a lot more repetition to get written into my brain!

Well most of it makes sense if you think about it. If you do 'ls *' in a directory do you get just the current directory contents or everything below? Imagine how slow it would be if * matched everything in the current directory and everything below it.

 

[Netopia]

AH.... you meant it glob isn't recursive BY DEFAULT or AT ALL? I took it that you meant at all. I knew that many things aren't recursive by default, so I figured you meant globbing had no recursivity at all.

Joe

 

[Nothinman]

At all. I see a few options to change how bash matches globs in the builtins(7) man page, but nothing to do with making them recursive.

 

[Netopia]

Ok... I've got all the .hrd and .msg files in one directory. There are about 50,000 files! Since it took so long to gather them, I wanted to make a copy before I started messing around with them. Now I'm dead ending myself again!

I thought by going into the directory where all the files are, and issuing the following command, that I could copy them all:

for i in *; do cp /mailtemp/

what I end up with is that it just drops down a line and gives me
>

Damn.

When I try to simply do cp * /mailtemp, it complains that there are too many arguements (I presume because of the vast number of files.

What am I doing wrong with the for/in statement?

Joe

 

[Nothinman]

What you would want is 'for i in *; do cp "$i" /mailtemp/' but the * in that isn't going to work with that many files just like 'cp *' won't. You'll need to use find for that. Something like 'find . -type f -print0 | xargs -0 cp /mailtemp/'.

The kicker here is that find is recursive by default, so that'll copy every file in the same directory and all below into /mailtemp/.

 

[Netopia]

Well... I finally just got frustrated and went into a KDE desktop and just dragged a copy of the folder to another location. It BLAZED and took only about 1 minute to copy all 42,700+ files!

Ok... back to trying to cat the .hdr and matching .msg files.

Ugh... I feel like I'm going in circles and teaching myself all the things I can discover that DON'T work!

Now that they are all in one locale, I'll try some of the stuff you guys suggested before.

Joe

 

[Nothinman]

Actually the 'for i in *' stuff won't work if the 'cp *' stuff wouldn't. If you're still having problems let me know and I'll whip something new up tonight.

 

[Netopia]

Hmmmm.... getting farther.

Typed what you said and got this. Any ideas? The files obviously exist!

[root@cranium raw]# for i in *.hdr; do NAME=`basename $i .hdr`; cat $NAME.hdr > $NAME.fin; cat $NAME.msg >> $NAME.fin; done
cat: m0243129.msg: No such file or directory
cat: m0243141.msg: No such file or directory
cat: m0243145.msg: No such file or directory
cat: m0243146.msg: No such file or directory
cat: m0243158.msg: No such file or directory
cat: m0243171.msg: No such file or directory
cat: m0243196.msg: No such file or directory