Is there a way to allocate a specific amount of ram to act as a write cache for a specific HD? In this case, a p2p app writes 10 different streams @ lets say 100k/s in parrallel. With traditional windows management, the result after 1hr is a very very heavily fragmented drive. Increasing cluster size seems to have some effect, but I'm wondering if there isn't a way to more directly address the issue with otherwise superfluous ram. I assume windows isn't structured to handle this kind of predictable writing. Wouldn't it be substantially more efficient for the HD to write these files in 30MB chunks rather than cluster size chunks?
Simulating HD cache with Ram?
- Thread starter isaacmacdonald
- Start date
you don't think the info could be held in ram for 5 minutes without being corrupted? I haven't had any problems with the more conventional ram disk setup, so I assume it wouldn't be any more risky than that.
The OS caches the file in 256K chunks by default. There are calls you can make to MSVCRT to increase the buffering to 1MB before the OS even sees what is written, assuming the P2P programs are using stdio which they probably are considering the bugs with files larger than 2GB.
The problem is Kazaa (and probably most other P2P software) explicitly tells it to flush the file after every packet received.
Also it's writing more than just the packet it got each time. The information about what clients the file came from, and the metadata each person put about the file is stored at the end of the file and re-written after every single packet.
They could easily get around the problem by making the file the final size when it is first created and overwriting all the garbage in the middle later.
The problem is Kazaa (and probably most other P2P software) explicitly tells it to flush the file after every packet received.
Also it's writing more than just the packet it got each time. The information about what clients the file came from, and the metadata each person put about the file is stored at the end of the file and re-written after every single packet.
They could easily get around the problem by making the file the final size when it is first created and overwriting all the garbage in the middle later.
It can be held in ram forever, but what happens when something crashes? You lose 30 meg instead of a few kb.
Unfortunately you're a little late with the idea
Design and implementation of a Log-Structured File System
* not speaking for Intel Corp. *
Originally posted by: Sohcan
Unfortunately you're a little late with the idea
Design and implementation of a Log-Structured File System
* not speaking for Intel Corp. *
That was cool
. So was anything ever done with this? This seems like such a patently obvious idea, with huge potential returns. These days there should be tons of applications that could take advantage of this kind of thing. <scratches head>for that other post, even my xp box has decent uptime (averaging two weeks or so) - as long as I have it hooked up to a UPS - the 32MB cost, versus the 5k cost of sudden failure, is negligable.
zephyrprime
Diamond Member
- Feb 18, 2001
- 7,512
- 2
- 81
If you have access to the source code for the programs, you can easily increase the packet size. If you don't about all you can do is set your system to "optimize for system cache" if you're running windows 2k or XP.
How are you gonna set the packet size it RECEIVES?
The real solution since it knows the file size before the download starts is to create the file at its final size initially and fill in the data as it comes in.
Or if you could get ext2fs drivers for win2k..... it intentionally spreads out file start locations to prevent fragmentation rather than always using the lowest # available block.
The real solution since it knows the file size before the download starts is to create the file at its final size initially and fill in the data as it comes in.
Or if you could get ext2fs drivers for win2k..... it intentionally spreads out file start locations to prevent fragmentation rather than always using the lowest # available block.
zephyrprime
Diamond Member
- Feb 18, 2001
- 7,512
- 2
- 81
Oh wait, I just realized that you were talking about p2p apps. Everything the other people have already said is true. P2P apps seem to flush immedately and the OS worries about the insecurity of leaving a lot of stuff in the write cache. So you're basically screwed. I've had winmx fragment a file in 9000 pieces!
torrents is a P2P application. The nice things about the application is the fact that before it ever starts downloading, it creates the files that take up the correct amount of space. It fills the file with the data as it gets it. That way fragmentation isn't a problem.
http://www.superspeed.com/ramdisk.html
http://www.totalidea.de/frameset-tweakxp.htm
Tweak-XP Pro creates a ram drive within windows xp and you
can try it free to try for 30 days
FILE: Ramdisk.sys Sample Driver for Windows 2000
http://support.microsoft.com/default.aspx?scid=KB;en-us;q257405
http://www.totalidea.de/frameset-tweakxp.htm
Tweak-XP Pro creates a ram drive within windows xp and you
can try it free to try for 30 days
FILE: Ramdisk.sys Sample Driver for Windows 2000
http://support.microsoft.com/default.aspx?scid=KB;en-us;q257405
Open source ext2fsd for windows
Commerical ext2fs/ext3fs driver and boot loader for windows
Probably both are based on the linux code for this filesystem which has the smarter allocation algorithm.
TRENDING THREADS
-
Discussion Zen 5 Speculation (EPYC Turin and Strix Point/Granite Ridge - Ryzen 9000)- Started by DisEnchantment
- Replies: 23K
-
Discussion Intel Meteor, Arrow, Lunar & Panther Lakes Discussion Threads
- Started by Tigerick
- Replies: 19K
-
-
-
Intel processors crashing Unreal engine games (and others)- Started by BFG10K
- Replies: 2K
Facebook
Twitter