Talkin' 'bout BASH scripting and auto updating Ubuntu/Debian.

[Brazen]

I want to create a script to perform an apt-get upgrade and schedule to run weekly (maybe daily).

The script could be a simple

#!/bin/bash
apt-get update
apt-get upgrade -y
apt-get autoclean

but what about if apt-get is already running (just in case). From what I can tell, apt checks that the file /var/lib/apt/lists/lock is not open. If it is open, then it won't run. I was thinking of using a while statement to have the script sleep until the file is no longer open, but I can't find a test to check if the file is open or not.

 

[Brazen]

Ok, so lsof lists open files. I tried it out and I see two files that are open while apt-get is busy downloading packages: /var/lib/dpkg/lock and /var/cache/apt/archives/lock. I can grep for these. Now I just need to figure out some way to test if the grep returns anything.

Also, does anyone know, is this a good mechanism for testing if apt-get will be able to run? and are these the correct lock files? and all the lock files, that I would need to test for?

 

[Brazen]

Ok, so far, I've come up with this

#!/bin/bash

LOCK_1="/var/lib/dpkg/lock"
LOCK_2="/var/cache/apt/archives/lock"
LOCK_3="/var/lib/apt/lists/lock"

OPEN_1='lsof | grep $LOCK_1'
OPEN_2='lsof | grep $LOCK_2'
OPEN_3='lsof | grep $LOCK_3'

while [ -n $OPEN_1 -o -n $OPEN_2 -o -n $OPEN_3 ]; do
echo "sleeping..."
sleep 10
OPEN_1='lsof | grep $LOCK_1'
OPEN_2='lsof | grep $LOCK_2'
OPEN_3='lsof | grep $LOCK_3'

done

echo "updating..."
apt-get update
apt-get upgrade -y
apt-get autoclean
echo "done."

But I'm not sure I put that 'while' condition correctly. Plus, I'm getting a "too many arguments" for the while condition.

 

[drag]

That's pretty good.
took me a while to figure it out. BTW I found a lockfile-progs package for debian that has a bunch of command line stuff for making and utilizing lock files.

But i found a clever way using 'fuser'. Fuser looks at a file and tells you what proccess is using it. returns non-zero if nobody is using it.


So something like this would work, I think (haven't tested it):

#! /bin/bash

LOCK_1="/var/lib/dpkg/lock"
LOCK_2="/var/cache/apt/archives/lock"
LOCK_3="/var/lib/apt/lists/lock"


while fuser $LOCK_1 $LOCK_2 $LOCK_3
do sleep 10
done

echo "updating..."
apt-get update
apt-get upgrade -y
apt-get autoclean
echo "done."

 

[Childs]

Too many arguments is because you are using a single quote, not a tick "`" for your OPEN_* variables. Also, I dont think you want to use lsof, as that checks for an open file, but dont you just want the existence of a lock file? In lsof it might show up as a hidden file or something.

Just fooling around here is my version of your script:


I dont use apt-get (running OSX) so it doesnt really do anything, but it would work. It all depends what you are trying to do with the lock files.

 

[drag]

btw you may want to be carefull with apt-get like that. It's not designed to be automated so much.

I don't know if aptitude would be better, but you probably want to setup so that it emails you results of what is going on so you don't assume it's been updating and it hasn't.

 

[drag]

Too many arguments is because you are using a single quote, not a tick "`" for your OPEN_* variables. Also, I dont think you want to use lsof, as that checks for an open file, but dont you just want the existence of a lock file? It might show up as a hidden file or something.

I this case the lock files are always there. They are always empty, even when they are locked. The programs basicly test to see if some other proccess is using them.

Linux has support for simply 'locking' files so that other proccesses can't use them. Check out the lockfile_create man file and such. If you have debian or ubuntu there should be a lockfile-progs file for managing them.

 

[Childs]

lockfile_create just seems to create a <file>.lock, if you were writing in C. You should force an update, then do lsof to see if you can see the lock file. fuser looks like a good utility...too bad it hasn't been included in OS X. I've spent many hours working around various file locking test cases when I simply wanted to know if a file was open or not.

 

[drag]

I am not sure about how all the locking work realy. This is the first time I looked at it. All I know is that the files are always there for dpkg/apt-get/etc, they are only locked when they are in use.

If somebody else knows more about it I'd like to hear it.

 

[Brazen]

Yeah, I double checked - the lock files are always there. And lsof does give reliable results. If apt-get is being run, the files show up in lsof, if it is not run, they don't show up. I've checked it manually

Originally posted by: drag
btw you may want to be carefull with apt-get like that. It's not designed to be automated so much.

I don't know if aptitude would be better, but you probably want to setup so that it emails you results of what is going on so you don't assume it's been updating and it hasn't.

Well, I looked on the Ubuntu wiki for how to do automatic updates on a server and basically the script provided there is just

#!/bin/bash
apt-get update
apt-get upgrade -y
apt-get autoclean

I figure I'm at least going a step beyond that and checking to see that it CAN be run. I don't have the link to the wiki article now, but I left it up on my computer at work so I will update this post tomorrow with it, if anybody cares.

And and thanks to drag for the fuser idea, I'll have to check that out tomorrow. And thanks to Childs for the tick thing. I didn't know I could (and _should_ in this case) use ticks, I'll try that out tomorrow.

Edit: Here's that wiki page.

 

[Brazen]

Ok, here is what I got now, but I'm still getting the "too many arguments error" I even tried using ticks and single quotes in my variable definitions, but no luck:

#!/bin/bash

LOCK_1="/var/lib/dpkg/lock"
LOCK_2="/var/cache/apt/archives/lock"
LOCK_3="/var/lib/apt/lists/lock"

COUNTER=0

while [ fuser $LOCK_1 $LOCK_2 $LOCK_3 -a $COUNTER -le 240 ]; do
echo "sleeping..."
sleep 2
COUNTER=$((COUNTER+1))
done

echo "Pretending to update..."
echo "done."

It's probably obvious, but I'll just point at, that what I added to drag's changes is a counter, so that the loop doesn't go on forever. My thought is to limit it to a couple hours, in case some one is on dial-up and apt-get could be downloading for days.

edit: btw, Childs, you might point out that I obviously don't understand how ticks are supposed to be used. I just now find an article that mentions ticks, so now I see why they would have been used up above, but not in this case 🙂

 

[Childs]

The ticks are used when you want to use the output of whats inside. So:

file=`fuser $LOCK_1`

would store the result of fuser $LOCK_1 in variable file. If you use quotes you are just putting the string fuser $LOCK_1 in variable file. You most likely are still getting the "too many arguments" error in your latest version because you need ticks around the fuser part:




I dont have access to fuser so I cant check it.

 

[drag]

Fuser will return 'true' if the files are being used, 'false' if they aren't.

It doesn't output any data to the console either way. So back ticks aren't going to work either way.

You can do something like..

counter=0
while [ $counter -le 32 ]
do if fuser $LOCK_1 $LOCK_2 $LOCK_3
then true
else apt-get update
apt-get upgrade -y
apt-get autoclean
break
fi
sleep 5
((counter++))
done

btw:
To see the return code of commands.. run the command then go
echo $?
and it will display the return code. 0 is true anything else is false. It will often give you a better idea of what is going on also.

 

[Nothinman]

I want to create a script to perform an apt-get upgrade and schedule to run weekly (maybe daily).

Why not just install and use apticron or cron-apt?

 

[Childs]

Originally posted by: drag
Fuser will return 'true' if the files are being used, 'false' if they aren't.

It doesn't output any data to the console either way. So back ticks aren't going to work either way.

You can do something like..

counter=0
while [ $counter -le 32 ]
do if fuser $LOCK_1 $LOCK_2 $LOCK_3
then true
else apt-get update
apt-get upgrade -y
apt-get autoclean
break
fi
sleep 5
((counter++))
done

btw:
To see the return code of commands.. run the command then go
echo $?
and it will display the return code. 0 is true anything else is false. It will often give you a better idea of what is going on also.

If fuser returns true or false then his version can work as well, and you can store the status of fuser in a variable and run tests against it, or use it in a test directly.

 

[drag]

Originally posted by: Nothinman

I want to create a script to perform an apt-get upgrade and schedule to run weekly (maybe daily).

Why not just install and use apticron or cron-apt?

sounds like a plan to me. (apticron sounds better)

 

[Brazen]

Originally posted by: Childs
The ticks are used when you want to use the output of whats inside. So:

file=`fuser $LOCK_1`

would store the result of fuser $LOCK_1 in variable file. If you use quotes you are just putting the string fuser $LOCK_1 in variable file. You most likely are still getting the "too many arguments" error in your latest version because you need ticks around the fuser part:




I dont have access to fuser so I cant check it.

I did try that, but I still got the error.

 

[Brazen]

Originally posted by: drag

Originally posted by: Nothinman

I want to create a script to perform an apt-get upgrade and schedule to run weekly (maybe daily).

Why not just install and use apticron or cron-apt?

sounds like a plan to me. (apticron sounds better)

The Ubuntu wiki page only mentions cron-apt, and says that it will only download packages, not install them. I'll look more into those two though.

 

[Nothinman]

The Ubuntu wiki page only mentions cron-apt, and says that it will only download packages, not install them. I'll look more into those two though.

There might be a reason for that, what happens when one of those packages pops up a debconf question and waits for input?

 

[Brazen]

Originally posted by: Nothinman

The Ubuntu wiki page only mentions cron-apt, and says that it will only download packages, not install them. I'll look more into those two though.

There might be a reason for that, what happens when one of those packages pops up a debconf question and waits for input?

At first glance, I think there is an option for cron-apt to ignore any debconf questions. Why would your config files ever need to be changed? Isn't that the point of freezing the version? Also, I don't plan to do a dist-upgrade, just upgrade. A new kernel would wreak havoc on vmware-tools among other things, so I would rather just do dist-upgrade manually.

 

[Childs]

Originally posted by: Brazen

Originally posted by: Childs
The ticks are used when you want to use the output of whats inside. So:

file=`fuser $LOCK_1`

would store the result of fuser $LOCK_1 in variable file. If you use quotes you are just putting the string fuser $LOCK_1 in variable file. You most likely are still getting the "too many arguments" error in your latest version because you need ticks around the fuser part:




I dont have access to fuser so I cant check it.

I did try that, but I still got the error.

Its your while statement. Rework it like I did in my previous example. You apparently cant do compoound statements in while, but can in if.

 

[drag]

Originally posted by: Brazen

Originally posted by: Nothinman

The Ubuntu wiki page only mentions cron-apt, and says that it will only download packages, not install them. I'll look more into those two though.

There might be a reason for that, what happens when one of those packages pops up a debconf question and waits for input?

At first glance, I think there is an option for cron-apt to ignore any debconf questions. Why would your config files ever need to be changed? Isn't that the point of freezing the version? Also, I don't plan to do a dist-upgrade, just upgrade. A new kernel would wreak havoc on vmware-tools among other things, so I would rather just do dist-upgrade manually.

Beter safe then sorry.

Configuration file `/etc/vim/vimrc'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : background this process to examine the situation
The default action is to keep your current version.
*** vimrc (Y/I/N/O/D/Z) [default=N] ?

Now does that pop up with regular upgrade or only with dist-ugprade?

You probably want to keep your configuration file changes you've worked on. I don't know, but it makes sense that if you choose 'yes' it's going to blow your configuration file away, but I haven't tested this so I don't know for certain.

For other prompts there is options for debconf on how serious the question has to be before you are prompted for it.


See dpkg-reconfigure debconf

the Debian packaging system isn't designed to provide automated updates. It's one of the downsides of it.

Also you miss out on notes. For example there are certain packages which upgraded may interfer with other things, so you have things like it will restart services automaticly after you upgrade that package. You could accidently end up taking down a bunch of services on your box and if it fails to upgrade then they may not come back up.

Other ones may be reconfiguration changes. For instance a package change may end up having a application's setuid bit removed because it was a security problem or something that can end up breaking system scripts you've made.

With Debian stable this isn't going to happen very much, but it may still happen. And if something breaks over the eavening then it may be a few days before you realise what is broken. After that then you may have no clue how to fix it or what went wrong. A huge PITA

(I'd be weary of taking advice from Ubuntu forums and wikis and such. Try to use Debian manuals and documentation as the more reliable source for information. Not that there isn't good information, just that you need to be carefull.)