Java File I/O Question: How to remove a portion of a file?

[statik213]

I have log-files that collect data over a long period of time in my app. The systems I'll be woring with will have very limited storage space. I'd like to specify an upper-bound on the size of the log file and once this size is reached I want to delete some of the old data in the same file. Now, since free space is already limited (if not unavailable) I can't simply copy the 'required' portion of the file to a new file snd delete the old log file.
Can this be done?

 

[bersl2]

What about memory? I mean, you're running a VM, so you must have a lot of RAM available. So copy the contents of the log into a String (StringBuffer?), then perform the changes, then overwrite the file.

 

[kamper]

http://java.sun.com/j2se/1.5.0/docs/api/index.html?java/io/RandomAccessFile.html

If you don't want to do it all in-memory. This would be just like any traditional posix-style file handling:
-seek to position of data you want to keep
-read data you want to keep
-seek to position of data you want to over write
-write data you want to keep
-setLength() to truncate

 

[kamper]

Second suggestion: are you using log4j? (If not you should probably look into it) It's pretty easy to whip up a solution that rolls over a file after a certain size is reached (archive and start a new one) and then to compress the old file. I believe gzip compression does extremely well with text (especially if it's repetitive) and that's all available with the standard jdk.

 

[statik213]

Originally posted by: kamper
Second suggestion: are you using log4j? (If not you should probably look into it) It's pretty easy to whip up a solution that rolls over a file after a certain size is reached (archive and start a new one) and then to compress the old file. I believe gzip compression does extremely well with text (especially if it's repetitive) and that's all available with the standard jdk.

I was thinking of gzip as well, but the fundamental problem still exists. I'm assuming that some users would just set it up to log data and never bother to collect it unless something goes wrong, in which case they'd be interested in the most recent data.

Log4j seems interesting, I've been pondering using it for my own logging requirements... if it's got built in support for this sort of a thing then all the better. But the RandomAccessFile stuff doesn't look too difficult.

Again, thanks kamper!!

 

[statik213]

Originally posted by: bersl2
What about memory? I mean, you're running a VM, so you must have a lot of RAM available. So copy the contents of the log into a String (StringBuffer?), then perform the changes, then overwrite the file.

I don't want to do the whole thing in memory, the log files may end up being pretty big depending on how the user sets things up.

 

[MOCKBA1]

Roll it daily or weekly

 

[bersl2]

Originally posted by: statik213

Originally posted by: bersl2
What about memory? I mean, you're running a VM, so you must have a lot of RAM available. So copy the contents of the log into a String (StringBuffer?), then perform the changes, then overwrite the file.

I don't want to do the whole thing in memory, the log files may end up being pretty big depending on how the user sets things up.

Yeah, forgot about RandomAccessFile..

 

[ys]

Are you using java.util.logging already? It provides the ability to set file size and cycle through a set of log files.

 

[kamper]

Originally posted by: MOCKBA1
Roll it daily or weekly

He's got a good point. Rather than chopping up a single file, it's more efficient and easier to keep a single roll. So, past a threshold (which could be time or size based) just delete the old file, rename the current one to the old one, create a new current file and continue to log to it. Log4j does this without any modification (as opposed to compression, which takes a little extra work).

 

[kamper]

Originally posted by: ys
Are you using java.util.logging already? It provides the ability to set file size and cycle through a set of log files.

Have you used the jdk logging much? I've never heard anything good about it (especially in comparison to log4j) but I've never used it myself.

 

[znaps]

It's not *bad*, just not as good as log4j.

 

[statik213]

Originally posted by: kamper

Originally posted by: MOCKBA1
Roll it daily or weekly

He's got a good point. Rather than chopping up a single file, it's more efficient and easier to keep a single roll. So, past a threshold (which could be time or size based) just delete the old file, rename the current one to the old one, create a new current file and continue to log to it. Log4j does this without any modification (as opposed to compression, which takes a little extra work).

Yeah that was my initial plan but the client want all the data in one file, so I guess I can write a utility to stitch the files back together..... got to think abt this a lil more.