Bash script and populating arrays. How?

[Resh]

I didin't add this to the FAQ thread 'cause I didn't think it qualified a "most commonly asked" question.

I understand how to call elements of an array in a bash script and how to manually populate the array, but I want to know how to automate the task of populating the array.

For example, suppose I have 10 zip files in a directory. I want my script to start by the creating an array that places the name of each of those files in an array. The array would then feed a while loop.

The size of the array created and populated would change from one execution to another, depending on the number of zip files present in the directory.

In practice, the sequence of events would look like:

- run script
- script places zipfile0.zip in array[0]
- script places zipfile1.zip in array[1]
.
.
.
- script places zipfile(n-1).zip in array[n-1]
- script places zipfile(n).zip in array[n]

- script continues on to next phase

Any ideas? If I need to clarify, just let me know.

Thanks!

N

 

[cleverhandle]

I'm not clear on what you're ultimately trying to accomplish, but I think this will do...

array=(`ls *.zip`)

Bash will automatically initialize the array and index the results of the ls starting at index 0. Note that would make zipfile0.zip element 0, zipfile1.zip element 1, as expected, but zipfile10.zip element 2, due to ascii ordering. So you may want to work a sort in there as well.

 

[drag]

Arrays and stuff are a bit over my head right now.. but I do have a simple way to execute a command on a bunch of files in a directory...


for i in `ls *zip` ; do unzip -oqL $i ; echo "$i has been unzipped"; done

I used that to unzip a hundred of so files I downloaded off of a ftp server last week.
To me the "for i in `xxxx`; do xxxx ;done" thing is the greatest thing ever...
Simple for loop. I am sure that you could adapt to a while loop in a shell script pretty easily.

 

[cleverhandle]

Yeah, I was also thinking that a for loop could probably do whatever you need to do with less fuss. Are you certain you need arrays, here?

 

[Barnaby W. Füi]

I tend to only use bash for simple stuff, if arrays are involved I'd use python or perl.

And as others mentioned.. I'm not sure you need the array.

 

[Resh]

Thanks. I'll try that suggestion cleverhandle.

To satisfy everyone's curiosity..

I have a script that unpacks zip files, performs various operations, logs the files that were in the zips, and puts the files where they need to be. Each zip file contains a core file and other related files. Right now, my log of the files does not identify which dependent files are linked with which core file.

I'm hoping that using an array with while/do loop will allow the process to be done on one file at a time. That way, the log of files will show the core and the dependent files as a package and I'll know which dependents go with which core.

Does that help? If anyone has a better idea on how to accomplish this, I'm certainly open to suggestions as I'm no bash guru, let alone a programmer.

Thanks!

 

[cleverhandle]

I'd stick with for loops, then. How about...

#!/bin/bash

cd ~/zipfiles
for file in *.zip do
mkdir work
cp $file work
cd work
unzip $file
echo "### Results of $file ###" >> ~/zipfiles/log
ls -lR >> ~/zipfiles/log
[do whatever operations and copying from here]
cd ~/zipfiles
rm -rf work
done

 

[Resh]

Cleverhandle,

That's exactly the type of thing I was hoping to do and I understand your proposal except for one line:

for file in *.zip do

Could you elaborate on how this works? It seems to successively assign the variable name $file to each member of a set populated by *.zip. Is that set an array?

As I said, I'm new to these suggestions and explanations from all you are very much appreciated. Always grateful for a chance to learn something.

N

 

[drag]

I like to do:

for i in `ls *zip`
do
unzip -oq $i
echo $i is unzipped
done

you see if you do a "ls *zip" you get a list of files ending in zip. But if you just do a ls *zip it isn't gonna work.
bash scripts have three types of quotes. "double" 'single' and `those slanted ones in the ~ (tithe) button`

try em' out

prompt>> ls *

cow dog poo luck script

prompt>> echo "ls *"

ls *

prompt>> echo 'ls *'

ls *

prompt>> echo `ls *`

cow dog poo luck script

prompt>> echo ls *

ls cow dog poo luck script (this last one is a bit funny, because echo first prints "ls" then it prints "*" which defaults to the contents of the directory, so don't get confused...)

You see the tithe quotes means: pretend the output of the command is what I actually typed in there.
The other two types of quotes means actually used what I type in there literally. The single one is a bit MORE literal than the double quotes for those instances when you want to echo the double quotes in a string.


So the when you do:

for i in `ls *`

it means that for evertime that `ls *` returns a output, instead of directing it to standard output (the terminal) take that output and assign it to the variable i. When you put the $ sign in front of the i it returns it's contents to stnd output which you can naturally use in other commands and redirecters.

So for every output from `ls *` assign it to the variable i. Then do unzip on what is represented in i. Then echo what is stored in i and echo unzipped after it. then repeat for next output until done.

btw other special characters are:
> --- redirect output to a file, over writing what's there, or creating a file if non-existant (ex. echo `ls *` > filename)
>> --- redirecting stnd output to a file, appending to the contents or creating a file if non existant
2> --- same as above, but only for errors, or special output intended to be kept seperate by the tool's programmer.
2>> --- same as the second one but for errors, too
; --- deliminates the end of a command or line and waits for the first one to complete before running the next one
& --- runs a command in the background
&& -- runs the command in the background and do what command comes next (this i am not sure off)
| --- used to pipe the output from one command to another.

That's about the extent of my bash knowledge...

just trying the :
for file in *.zip

I never tried doing that.

 

[cleverhandle]

Originally posted by: Resh
Could you elaborate on how this works? It seems to successively assign the variable name $file to each member of a set populated by *.zip.

Yup - exactly. Drag's method does the same. The index - file in this case - can have whatever name you wish. Just prepend a $ to it in the rest of the script. The *.zip indicates a set of files - you can use "globbing" like I did with the * or use the output of a command between ` ` marks, like in Drag's example. I believe the set has to be a set of files, as opposed to a set of strings or characters, but I'm not sure off the top of my head.

Is that set an array?

Internally? I have no idea. Maybe. But within the loop, I'm pretty sure it's not - i.e. you couldn't use something like ${file[1]} meaningfully.

 

[Barnaby W. Füi]

Originally posted by: cleverhandle
I believe the set has to be a set of files, as opposed to a set of strings or characters, but I'm not sure off the top of my head.

It can be anything:

for i in 1 2 3 4 5 6 7 8 9 10; do echo $i; done
1
2
3
4
5
6
7
8
9
10

for i in `seq 1 10`; do echo $i; done
1
2
3
4
5
6
7
8
9
10

For the hell of it, here's a script I used to back up my mp3's a while ago. My mp3s are in /music, I wanted to copy them to the current directory, in batches of 680MB. I would then run mkisofs in the directory, do an ls, and use the last file in the directory as the argument to the script the next time around, so it would start copying files alphabetically after that file. Dunno if it makes any sense the way I'm explaining it, or if it makes sense at all, but it worked nicely

(hit quote to look at the indentation properly)

STARTWITH=$1

while [ true ]; do

if [[ ! $LASTFILE ]]; then
LASTFILE=$STARTWITH
fi

SIZE=`du -c -m 2>/dev/null | tail -n 1 | awk '{print $1}'`

if [ "$SIZE" -gt "680" ]; then
exit 0
else
FILE=`ls -1 /music/*3 | grep -A 1 "$LASTFILE" | tail -n 1`
cp -p "$FILE" .
fi

LASTFILE=`ls -1 *3 | tail -n 1`

done

Note that "for" loops loop through words (delimited by spaces). If you want to loop through lines, for example, from a file, you can use a "while" loop with "read", like this:

while read line
do
<some stuff>
done < /some/file

Or, you can pipe it:

echo `some magical command here that outputs stuff | while read line; do some_command "$line"; done

 

[Barnaby W. Füi]

&& -- runs the command in the background and do what command comes next (this i am not sure off)

&& is a conditional "and", it will execute the latter command if the former exits with 0 status.

echo foo && echo bar
foo
bar

cat /dev/urandom > /dev/kmem && echo hello
bash: /dev/kmem: Permission denied

Notice that the hello never appears.

You can also use it in the place of "if" in certain situations.

if [ "$TERM" = "xterm" ]; then echo -ne "\033]0;"`tty | sed -e 's,/dev/,,' | sed -e 's,/,,'`"\007"; fi

[ "$TERM" = "xterm" ] && echo -ne "\033]0;"`tty | sed -e 's,/dev/,,' | sed -e 's,/,,'`"\007"

Those both accomplish the same thing.

Take our "echo foo && echo bar", and we can also switch it around.

if echo foo; then echo bar; fi
foo
bar

The opposite of && ("and") is || ("or"). If the first command exits with non-zero status, then the next command gets executed.

echo foo || echo bar
foo

rm -rf /root || echo "dammit!"
dammit!

etc etc.

 

[cleverhandle]

Thanks for the clarification, BBWF.

 

[Resh]

Guess I know where to come when I have bash questions, huh? Thanks everyone.

If anyone is interested, I can post the complete script and you guys can critique. The script works for my purposes, but it could be improved and your comments would surely be informative.

Indicate interest and I'll post it.

Thanks again!

N

[EDIT] - Drag, thanks for the help. Just realized that I missed your original for loop post when I was reading. Some of the credit I gave to Cleverhandle should have gone to you. Sorry if you felt snubbed.

 

[drag]

No credit. I am not realy that good. That's about the only realy usefull trick I know. Got the concept from here

 

[Barnaby W. Füi]

If anyone is interested, I can post the complete script and you guys can critique. The script works for my purposes, but it could be improved and your comments would surely be informative.

Indicate interest and I'll post it.

Yeah, post it, looking at other peoples' code is good practice for everyone

 

[cleverhandle]

Definitely post it - I'd love to see more scripting around here. Can always get some new ideas...

 

[Resh]

You asked for it. Like I said, I'm a beginner so be gentle.

Note: must be run as root so I 'su' first

##############################################
#!/bin/bash
# This script is intended to facilitate the unzipping, compression
# and placement of map files to the appropriate directories of the
# game server.
# linked files include:
# /home/server/.rfe (hidden)
# Define Variables
work="/home/server/working"
ut="/home/server/ut2003_dedicated"
www="/var/www/html"
dl="/home/server"
zip="/maps"
ut2="/Maps"
uz2="/redir"
ogg="/Music"
utx="/Textures"
usx="/StaticMeshes"
uax="/Sounds"
u="/System"
compress="/home/server/ut2003_dedicated/ucc compress -nohomedir"
log="/var/www/html/maps/map_log"
llama="/home/llama/ut2003"
rfe="/home/server/.rfe"
matt="/home/matt/ut2003"
# Set Date in Log
echo "Installed on `date`" >> $log
# Create Directories in llama & matt
mkdir $llama$ut2 $llama$ogg $llama$usx $llama$utx $llama$uax $llama$u
mkdir $matt$ut2 $matt$ogg $matt$usx $matt$utx $matt$uax $matt$u
# make extensions lower case
$rfe ZIP zip
# Set permissions on zip files
# chmod 770 $dl/*.zip
###### Start loop #######
for file in $dl/*.zip do
{
# Create working directory
mkdir $work
mv $file $work/
# Decompress file
unzip -j -o $work/*.zip -d $work
echo Unpacking Zip file
# Change directories
cd $work
# Change case of extensions
$rfe TXT txt
$rfe XML xml
$rfe HTML html
$rfe HTM html
$rfe GIF gif
$rfe OGG ogg
$rfe JPG jpeg
$rfe JPEG jpeg
$rfe jpg jpeg
# Change back directories
cd $dl
# Clean out text & other files
rm $work/*.txt
rm $work/*.xml
rm $work/*.gif
rm $work/*.html
echo Extra Files Cleared
# Inventory files with data so I know what has been installed
echo "********************************* Map File: $file" >> $log
ls $work >> $log
# Compression
echo Beginning ucc compression
$compress $work/*.u*
echo ucc compression complete
# Move compressed map files
echo Moving compressed files to redirect
mv $work/*.uz2 $www$uz2/
# Copy files to matt
echo Moving files to llama directory
cp $work/*.ut2 $matt$ut2/
cp $work/*.utx $matt$utx/
cp $work/*.usx $matt$usx/
cp $work/*.uax $matt$uax/
cp $work/*.u $matt$u/
cp $work/*.int $matt$u/
cp $work/*.ogg $matt$ogg/
cp $work/*.jpeg $matt$ut2/
# Copy files to llama & move oggs
cp $work/*.ut2 $llama$ut2/
cp $work/*.utx $llama$utx/
cp $work/*.usx $llama$usx/
cp $work/*.uax $llama$uax/
cp $work/*.u $llama$u/
cp $work/*.int $llama$u/
mv $work/*.ogg $llama$ogg/
mv $work/*.jpeg $llama$ut2/
# Move files to server
echo Moving files to server directories
mv $work/*.ut2 $ut$ut2/
mv $work/*.utx $ut$utx/
mv $work/*.usx $ut$usx/
mv $work/*.uax $ut$uax/
mv $work/*.u $ut/$u/
mv $work/*.int $ut/$u/
# Move zip file to www
echo Moving zip file to public folder
mv $work/*.zip $www$zip/
# Remove working directory
rm -d -f -R $work
}
###### END LOOP ##################
# Set permissions
echo Changing Permissions
/root/jobs/owner
chown -R llama. $llama/*
chmod -R 770 $llama/*
chown -R matt. $matt/*
chmod -R 770 $matt/*
echo Process Complete & the Anandtech community rules!

################################

Thanks everyone!

N

 

[cleverhandle]

This should be fun. Here's one bit...

mkdir $llama$ut2 $llama$ogg $llama$usx $llama$utx $llama$uax $llama$u
mkdir $matt$ut2 $matt$ogg $matt$usx $matt$utx $matt$uax $matt$u

can be redone as
mkdir {$llama,$matt}/{$ut2,$ogg,$usx,$utx,$uax,$u}
I see some more opportunities, but don't have time to test them out at the moment. I'm sure others will kick in, though...

edit: typo

 

[Resh]

Cool! That would save sooo much typing.

This is exactly the sort of thing I'm looking for!

Keep 'em coming. I'll try your suggestions out and report back.

N