dd - a pretty slick little utility...

[Kadarin]

So I bought a 120GB drive to replace my 27GB that was home to both my Win2K and my Gentoo linux installs, and the thing I was dreading the most was reinstalling Gentoo. So my coworker asks, why not use dd? So I threw this drive in off another IDE controller, booted up (stupid me doesn't even bother to do this off a live cd, but instead off the regular drive I'm about to image), opened a terminal, did an su to root, and typed "dd if=/dev/hda of=/dev/hdi bs=8192". About a half hour later or so, it's done, I swap out the drives, and here I am. My coworker just saved me a whole buttload's worth of work.

 

[Crusty]

:beer: cheers!

I should remember that, I am about to swap hard drives soon!

 

[Barnaby W. Füi]

Though by directly copying the disk's contents, you either need identical drives, or you need to be copying to a bigger drive and then will have to muck around with your partitioning a bit.

And dd really shows how nice unix's everything-is-a-file concept is. All it's really doing is opening two files, reading from one, and writing to the other. You could write your own utility to do that pretty easily. (but of course dd has a lot of options which make it a bit more complex)

Edit: And dd is by no means the only way of going about it. I usually just use tar. A little more work, but a lot more flexible since it's moving files around instead of the raw data on the drive.

 

[skyking]

That is cool!
With FreeBSD, you have to boot into single-user mode to have that work properly.

 

[drag]

Ya. DD is cool, but for backing up drives and stuff it definately has some issues (other then the file size images their is issues with very low-level aspects of the OS that I don't understand). Tar is good, but it has it's issues too. What I find tar most usefull for is to backup home directories for moving from drive to drive and such.

some examples of using dd (remember everything is a file in unix...)

dd if=/dev/zerp of=/dev/hda

Doing a zero-out wipe of a harddrive.

###########################

dd if=/dev/urandom of=/dev/hda

doing a random junk filling up a harddrive.

##############################

dd if=/dev/fd0 of=/home/drag/cool.img

make a floppy image

#################################3

dd if=/home/drag/cool.img of=/dev/fd0

take a floppy image and make a new floppy that's a copy of the original disk.

#################################3

dd if=/dev/cdrom of=/home/drag/cdrom.iso

make a iso image of a data cdrom. (to bad it doesn't work in reverse. cdrecord app does that...)

#################################3

dd if=/home/drag/sample.wav of=/dev/dsp

play a wave file on your sound card (crappy quality though! It's a good test for troubleshooting cards)

#################################3

mv /home/drag/cdrom.iso /dev/null

deletes the cdrom.iso file. Note realy a dd command thing though...

#################################3

You can use DD for even messing around with the MBR.

To back up a entire MBR

dd if=/dev/hda of=/home/drag/MBR.backup.img bs=512 count=1

To format the portion of the MBR that lilo lives in, without screwing up the partition information on the last 66 bytes of the first sector (512 bytes large)(USE AT YOUR OWN RISK!!!!!!)

dd if=/dev/zero of=/dev/hda bs=446 count=1

#################################3

Something that is fun to do is to make a image and mount it in loopback:

dd if=/dev/cdrom of=./cdrom.iso
mkdir ./isomount
mount -o loop cdrom.iso isomount

(remember that only root can mount stuff unless you make a special exeption in /etc/fstab file)

That way you can edit the contents of a cdrom image and then umount and you have a modified iso image you can burn to a cdrom.

 

[ntrights]

:beer: dd is really neat. i dd when i make copy's of solaris hd's to new ones works great

 

[Barnaby W. Füi]

Originally posted by: skyking
That is cool!
With FreeBSD, you have to boot into single-user mode to have that work properly.

Probably an issue with buffers, and probably applies to linux as well. If you're reading raw data from the hard drive, you won't get changes to files which are still buffered, since that's a layer of abstraction above the raw drive contents.

 

[skyking]

Originally posted by: BingBongWongFooey

Originally posted by: skyking
That is cool!
With FreeBSD, you have to boot into single-user mode to have that work properly.

Probably an issue with buffers, and probably applies to linux as well. If you're reading raw data from the hard drive, you won't get changes to files which are still buffered, since that's a layer of abstraction above the raw drive contents.

that was my understanding of it.
Drag, you outdo yourself with yadda yadda! Nice post!

 

[HKSturboKID]

I see a lot of DD commands, but can anyone put in the syntax and what does DD stands for?
I don't have my box up yet, I guess once I get it up I'll have to do a man dd.

 

[skyking]

Direct Disk?
It copies the input file (if=) to the output destination(of=).
man dd
Look at the bottom of this page, to see how it is used to write a bootable image: "Write the Image Files to the Floppy Disks"


Freebsd install floppies, for a good example.

 

[xcript]

$ dd if=/dev/urandom of=/dev/dsp

 

[Nothinman]

I believe it stands for data dump, the fact that it works on disks is only a nice side affect of the "everything is a file" concept unixes use.

But if you copied it to a bigger disk you will need to use something to resize the filesystem even though the partition is bigger.

 

[Kadarin]

Thanks for the useful info, drag!

I did run into a bit of a snag, though (I'm out of town so it's taken me a couple of days to post here). Turns out, apparently, that I had previously partitioned my old drive into 4 *primary* partitions, so I have all this free space I can't do anything with (save resizing the last primary partition, but I can't do that to expand to encompass the entire remaining volume of dead space.

So... is it possible to non-destructively rewrite the partition table in such a way that I can convert those 86.5 or so GB into a single FAT32 partition?

I *really* don't want to reinstall Gentoo!

 

[Nothinman]

Rewriting the partition table won't help, you still have to resize the filesystem inside the partition.

This is why things like partimage exist, dd just isn't that good of a tool for this use.

 

[CTho9305]

google fips.

 

[skyking]

I've used ghost 2002 or newer to image my UFS drives, as long as the destination drive was at least the same size. Is there any way or tool I can use after imaging, that will allow me to use addtional unpartitioned space in my /usr partition?
I move from a 2 to a 4 gig, for example, and I am left with a smallish /user, and 2 gigs of unpartitioned space I can then partition off for something else, where it would be more convenient to incorporate that space into a larger /user.

 

[drag]

Here is a nice webpage with various examples of Netcat and DD to clone operating systems.


Of course as with all *nix stuff there is almost a infinate different ways to accomplish any task. You can use propriatary tools to accomplish the job, you can build your own programs and scripts, or you can use the veriaty of tools at you displosal. Which is one of the main reasons I enjoy using Linux over Windows.

It all depends on how creative you are and how much you know about the inner workings of the operating systems and it's various applications and tools.

Here is another example of a way to clone drives using "supertars"

Also on that guy's website he has a few other examples of backup technics that you can employ. Also, if you care, check out the other articles for some old school unix knowledge applied to modern operating systems. He seems to have most of his experiance with the SCO unixware type stuff, but I can't hold that against him. (After all it's the company formally known as Caldera that are being the legal butsticks, former linux Distro people even! The original SCO company has changed it's name and has moved on to bigger and better things )

 

[drag]

Originally posted by: skyking
I've used ghost 2002 or newer to image my UFS drives, as long as the destination drive was at least the same size. Is there any way or tool I can use after imaging, that will allow me to use addtional unpartitioned space in my /usr partition?
I move from a 2 to a 4 gig, for example, and I am left with a smallish /user, and 2 gigs of unpartitioned space I can then partition off for something else, where it would be more convenient to incorporate that space into a larger /user.

Probably using Tar instead of outright cloning you could do it. With tar you can copy the files into one big file, copy it over then simply untar the contents of over into the new harddrive.

Or maybe use Tar in combination with netcat command to simply transfer everything over. Pluss this will have the added benifit of degragmenting everything...

Check out the man pages or get a better Tar then what you have inorder to preserve permissions. There are a few different versions with gnu tar being the most common for linux, but not nessicaraly being the best one.

After all tar is Tape ARchive utility, and is pretty much what it was designed for in the first place.

 

[AFB]

Originally posted by: drag
Ya. DD is cool, but for backing up drives and stuff it definately has some issues (other then the file size images their is issues with very low-level aspects of the OS that I don't understand). Tar is good, but it has it's issues too. What I find tar most usefull for is to backup home directories for moving from drive to drive and such.

some examples of using dd (remember everything is a file in unix...)

dd if=/dev/zerp of=/dev/hda

Doing a zero-out wipe of a harddrive.

###########################

dd if=/dev/urandom of=/dev/hda

doing a random junk filling up a harddrive.

##############################

dd if=/dev/fd0 of=/home/drag/cool.img

make a floppy image

#################################3

dd if=/home/drag/cool.img of=/dev/fd0

take a floppy image and make a new floppy that's a copy of the original disk.

#################################3

dd if=/dev/cdrom of=/home/drag/cdrom.iso

make a iso image of a data cdrom. (to bad it doesn't work in reverse. cdrecord app does that...)

#################################3

dd if=/home/drag/sample.wav of=/dev/dsp

play a wave file on your sound card (crappy quality though! It's a good test for troubleshooting cards)

#################################3

mv /home/drag/cdrom.iso /dev/null

deletes the cdrom.iso file. Note realy a dd command thing though...

#################################3

You can use DD for even messing around with the MBR.

To back up a entire MBR

dd if=/dev/hda of=/home/drag/MBR.backup.img bs=512 count=1

To format the portion of the MBR that lilo lives in, without screwing up the partition information on the last 66 bytes of the first sector (512 bytes large)(USE AT YOUR OWN RISK!!!!!!)

dd if=/dev/zero of=/dev/hda bs=446 count=1

#################################3

Something that is fun to do is to make a image and mount it in loopback:

dd if=/dev/cdrom of=./cdrom.iso
mkdir ./isomount
mount -o loop cdrom.iso isomount

(remember that only root can mount stuff unless you make a special exeption in /etc/fstab file)

That way you can edit the contents of a cdrom image and then umount and you have a modified iso image you can burn to a cdrom.

Thanks for another wonderful post.

DD sounds cool.

 

[skyking]

Originally posted by: drag

Originally posted by: skyking
I've used ghost 2002 or newer to image my UFS drives, as long as the destination drive was at least the same size. Is there any way or tool I can use after imaging, that will allow me to use addtional unpartitioned space in my /usr partition?
I move from a 2 to a 4 gig, for example, and I am left with a smallish /user, and 2 gigs of unpartitioned space I can then partition off for something else, where it would be more convenient to incorporate that space into a larger /user.

Probably using Tar instead of outright cloning you could do it. With tar you can copy the files into one big file, copy it over then simply untar the contents of over into the new harddrive.

Or maybe use Tar in combination with netcat command to simply transfer everything over. Pluss this will have the added benifit of degragmenting everything...

Check out the man pages or get a better Tar then what you have inorder to preserve permissions. There are a few different versions with gnu tar being the most common for linux, but not nessicaraly being the best one.

After all tar is Tape ARchive utility, and is pretty much what it was designed for in the first place.

So I'd partition a target drive using another running installation, and mount the source drive there also? Tar and move the files to these larger partitions, and untar?
Or is this something you would do from within the source system, in single user mode?

 

[drag]

Well. I donno, kind of have to depend on the OS itself which I am not sure of the details. I am only familar with linux.

Plus anything that would do would probably be inferior to getting a app that is designed to do this stuff from the get go, but I would use it just so that I wouldn't have to spend any more time or effort then I had to.

I was checking out this page of nice tar tricks

So say you wanted to back up your config or /usr or home files onto a onto a server. So you can login from your machine you want to back up via FTP and do the nice ftp pipe feature that I just discovered:

ftp > put "|tar -cvfp /usr/ |gzip" > usr.tar.gz

Then in order to retreive them you would do the oppisite from a boot disk. Mount the future user partition to /mnt and change into it. The login to the ftp server.

ftp > get usr.tar.gz "|gunzip | tar -xvfp "

Then do the same for the rest of the directories.

Or something like that. Stuff like this usually takes a couple tries in order to get it working correctly, but after I got it and remember to write it down somewhere or make a script, then I can do it pretty reliably.

There are only a few things I would worry about...
Like permissions. Messed up permissions suck. I would have to be very carefull if your OS uses ACL and have to make sure that the tar versions on both the boot and the original machine support it correctly.

Then any "special" files. In linux you have proc and dev that that sometimes don't take kindly to being copied since they don't hold any "real" files.

unfortunatly I don't have anywere to mess around with this stuff right now. Maybe when I get home...

I would like to figure out a way to use Netcat to tar ball a directory pipe it into netcat and then have it untar onto the client computer.

Althought that ftp pipe trick is pretty cool.


Of course the VASTLY EASIER way to do it reliably would to simply stick both drives in one computer, boot up with a boot disk, and copy the entire directory from one drive to another. Unless there some special file permission or low-level whatnot that I am not aware of with your OS.

 

[chsh1ca]

Actually, I rather like the age old "listening to the sounds of your harddrive" that dd if=/dev/hda of=/dev/dsp offers.

A while back a friend and I were working on a perl frontend + perl daemon that would essentially replace apps like Ghost (both networkable and on local drives) using DD. Never got around to finishing it, but I got my part far enough that the interface and daemon were capable of doing local partition mirroring. There is a lot of quite powerful tools included in most linux distributions that many people don't know about. It's good to muck about in /bin, /usr/bin, /usr/sbin and read the manpages on all the binaries you find there.

Simple example of a VERY simple remote disk image server using cat and netcat would be:
while true;
do
cat /path/to/some/disk.img | nc -l -p 7500 -q 2
done

Then on the client you just do:
nc <disk image server hostname> 7500 > disk.img

 

[Barnaby W. Füi]

Here's how I back up my home dir every day

#!/bin/bash
ssh fw 'rm -f two-daily-backup-*.tgz'
tar zcvf - /home/death -X /home/death/.tar-exclude | ssh fw 'cat > "two-daily-backup-`date | sed -e s/\ /_/g\;s/:/-/g`.tgz"; chmod 600 two-daily-backup-*.tgz'

With the key point being that you can pipe data right through ssh.

tar zcvf - . | ssh somemachine 'cat > tarball.tgz'