Need an editor for very large text files

[Miramonti]

Any suggestions for a program that has a find/replace for large text files, such as 500mb or even 1gb+ size files?

 

[Phynaz]

Any suggestions for a program that has a find/replace for large text files, such as 500mb or even 1gb+ size files?

Try Ultraedit.

 

[Miramonti]

Try Ultraedit.

Thanks, looks like an awesome package. In looking it up, I also found a free program called PsPad, which might do the trick as well.

 

[bruceb]

Ultraedit should do. Nice program. See info below about how to set it for large files.

http://www.ultraedit.com/support/tutorials_power_tips/ultraedit/large_file_handling.html

 

[Engineer]

I know Notepad2 can edit large files, but it is limited by the amount of memory that you can give it (entire file must be in live memory). Not sure if it would meet your needs or not (I've used it for years but not on files of that size).


http://www.flos-freeware.ch/notepad2.html

 

[Miramonti]

I know Notepad2 can edit large files, but it is limited by the amount of memory that you can give it (entire file must be in live memory). Not sure if it would meet your needs or not (I've used it for years but not on files of that size).


http://www.flos-freeware.ch/notepad2.html

Thanks, but some of the files that I have are closer to 2gb, with for example 13m lines of data, so it would be tough to work with them in memory on my system.

 

[Miramonti]

I just tried PsPad that I mentioned earlier and it needs to load the files into memory as well, so it's not going to work as a free option.

 

[KeypoX]

what are you doing with 1GB text file? That seems to be more, than possible to read in a life time?

 

[Miramonti]

what are you doing with 1GB text file? That seems to be more, than possible to read in a life time?

Each 1.5gb+ file is one month's worth of price data on a few thousand stocks. Sometimes there are wierd header anomalies that can be fixed with a find/replace tool.

 

[KeypoX]

Each 1.5gb+ file is one month's worth of price data on a few thousand stocks. Sometimes there are wierd header anomalies that can be fixed with a find/replace tool.

ah ok thats a ton of data.

 

[Engineer]

Hell, with that size, you might just need to use a full word processor and save in .txt mode. Not sure if OpenOffice and do a text file that size or not. Wow, that's big stuff.

Textpad states it can use all of memory, including virtual memory limits to open a file. Might try that one.

Notepad++ is supposed to have a 2GB file size limit.

 

[tcsenter]

If you have 64-bit OS you can try modifying the executable image to set LARGE_ADDRESS_AWARE flag on the 32-bit text editor application. That should help speed things up as well.

 

[Aluvus]

Do the things you need to change span multiple lines? If not, it would be reasonably straightforward to install Cygwin or and just write a script in Perl/Python/whatever that reads the input line by line. Now that I think about it, you could probably do it with ActivePerl. This approach can handle arbitrarily large files without worrying about how much RAM you have.

Or anyway, it's reasonably straightforward if you're already familiar with a scripting language...

 

[Miramonti]

Hell, with that size, you might just need to use a full word processor and save in .txt mode. Not sure if OpenOffice and do a text file that size or not. Wow, that's big stuff.

Textpad states it can use all of memory, including virtual memory limits to open a file. Might try that one.

Notepad++ is supposed to have a 2GB file size limit.

I want to avoid putting everything into memory because other things just don't run efficiently once all of Windows memory is used up. Usually after something like that, I need to reboot to refresh the memory and that takes a long time with my large ramdisk copying itself to the hd each time (which is also why I don't have enough ram to but one of these files into memory w/o using the pagefile quite a bit.)

If you have 64-bit OS you can try modifying the executable image to set LARGE_ADDRESS_AWARE flag on the 32-bit text editor application. That should help speed things up as well.

32 bit Xp.

Do the things you need to change span multiple lines? If not, it would be reasonably straightforward to install Cygwin or and just write a script in Perl/Python/whatever that reads the input line by line. Now that I think about it, you could probably do it with ActivePerl. This approach can handle arbitrarily large files without worrying about how much RAM you have.

Or anyway, it's reasonably straightforward if you're already familiar with a scripting language...

Everything that I need to change is on single lines. I'm certain a script would be perfectly efficient, altho I don't know any scripting languages (or coding of any language for that matter.)

I use a program called Automate by networkautomation.com which allows for embedded basic scripts in the tasks, and I've used them before for various text manipulation functions on some of my larger files. The Automate program itself is way to slow to loop thru millions of lines in these large text files.

Those scripts I've gotten written for me by their forum guru's on request, and the scripts generally rip thru these large files quickly, but I thought I'd go a different route this time for simple find/replace actions, and not hit people up too much over there for script requests.

 

[Anubis]

try editpad lite

 

[amddude]

I'm a notepad++ fan

 

[Gooberlx2]

I'm a notepad++ fan

yeppers

 

[Modelworks]

Context has no limitation and is free
http://www.contexteditor.org/

 

[Miramonti]

I give up on finding a free version that doesn't put the entire file into memory. All the ones suggested here, and I'm appreciative of everyone's suggestions, appear to do this. (unless I overlooked a program's preference setting)

Ultraedit seems to be the only one that doesn't...the downside of course is the $60 price tag.

When opening a file larger than 45mb (default customizable threshold), it asks whether or not you'd like to disable creation of a Temp file.

After turning off the temp file creation, it doesn't load the file into memory and acts in a find/replace just as a script would, making all changes directly to the file, changes made permanently.

It's impressive. It took roughly 3.5 minutes to edit a 1.8gb file with nearly 34 million lines of data (the actual quantity of lines edited might have been only been ~1 percent), which is as fast as an efficient script would do it, and the memory used never went above 45mb.

 

[bruceb]

Sorry to say, sometimes you need to pay to get a program to do what you want to do. I might suggest the bundle of UltraEdit & Ultracompare (file comparison prog) for $89.95 USD .. Note that both come with 1 year of free upgrades, be it minor or major release, so long as it falls within the 1 calendar year. After that if you want or need to upgrade, there is a fee structure on the web site.

 

[Miramonti]

Sorry to say, sometimes you need to pay to get a program to do what you want to do. I might suggest the bundle of UltraEdit & Ultracompare (file comparison prog) for $89.95 USD .. Note that both come with 1 year of free upgrades, be it minor or major release, so long as it falls within the 1 calendar year. After that if you want or need to upgrade, there is a fee structure on the web site.

I don't have a problem with paying for it (subscription-ware?...that can be a different story...)

At this point I can resolve what I need to within the trial period, but if issues come up more frequently or other issues arise then I'll pony-up...it would be a nice tool to have in my arsenal.

 

[lxskllr]

Is there a reason you can't go 64bit with your O/S? If you don't mind putting out $, That should solve the problem, and alleviate possible future problems with other software.

 

[Miramonti]

Is there a reason you can't go 64bit with your O/S? If you don't mind putting out $, That should solve the problem, and alleviate possible future problems with other software.

An upgrade to Windows 7 64bit will cost me >$2k in software upgrades, including the OS, to upgrade the software I'm dependent on in Xp to be compatible with win7. (Automate alone is now ~$1500 + $300 annual maintenance fee...which is a joke but out of my control.)

Unless I can gain a significant speed increase going to win7, which I can't now, I am probably going to be an xp holdout for a long time.

 

[lxskllr]

Maybe you're going at it the wrong way. How about splitting the files into usable chunks? Here's a Google search of file splitters. I haven't used any, but that might work for you...

http://www.google.com/cse?cx=partne...-8&sa=Search&q=how+to+split+a+text+file&hl=en

 

[Miramonti]

Maybe you're going at it the wrong way. How about splitting the files into usable chunks? Here's a Google search of file splitters. I haven't used any, but that might work for you...

http://www.google.com/cse?cx=partne...-8&sa=Search&q=how+to+split+a+text+file&hl=en

1.8gb files are actually not a problem to use for research, just mass editing, which is solved earlier by using Ultraedit. I'll still need some occasional script assistance for non-standard data manipulating, but the file size doesn't effect those issues.

The biggest thing I'm missing out on with win7 is being able to have a much larger ramdisk to work with all this data in, but it's not as bad on the harddrive as one might think.