The BSOD that shouldn't happen

[TechnoPro]

An associate of mine purchased a well-equipped Dell 1 year ago. It has functioned flawlessly except for the occasional BSOD. They began to appear around the 7-8 month mark. There was no software or hardware change at that point to draw any correlations with. They have since reappeared roughly 1 per month.

System specs, pulled from the Dell invoice:

Dimension 8400 Pentium® 4 Processor 540 with HT Technology (3.20GHz, 800 FSB)
Memory 1GB Dual Channel DDR2 SDRAM at 533MHz (2x512M)
Hard Drive 160GB Serial ATA Hard Drive (7200RPM) w/ Native Command Queuing
Video Card 128MB PCI Express? x16 (DVI/VGA/TV-out) ATI Radeon? X300 SE
Monitor 17 inch Ultrasharp? 1704FPT Digital Flat Panel
Floppy Drive and Additional Storage Devices 3.5 in Floppy Drive
Keyboard Dell Quietkey® Keyboard
Mouse Dell 2-button scroll mouse
Network Interface Integrated Gigabit Ethernet
Modem 56K PCI Data/Fax Modem
CD ROM/DVD ROM Dual Drives: 16x DVD-ROM Drive + 48x CD-RW Drive
Sound Card Integrated 5.1 Channel Audio
Speakers Dell A215 Speakers

Attached devices:

Palm Treo 650
Canon SD 450 Camera
Tungsten E Handheld Palm Pilot
HP LaserJet 1320
HP DeskJet 6122
Microsoft Intellimouse Explorer 3403
Epson Scanner 1650

Software wise, it runs XP Pro. The user is super careful about not installing anything questionable on there. Everything he uses is standard, off-the shelf products. My only gripe is AOL, but that is a moot point. Other than that, the system is well protected, well maintenance, and all patched.

Latest system specs (HW and SW) as pulled through Belarc available here.

My associate and I cannot find any discernable pattern as to when the BSODs occur. There is no specific program that is open when they happen, no consistent time of day, etc.

Where should I start investigating deeper?

EDIT -

Genius that I can be, I forgot to post the technical information. Minidumps, bad quality photos of the BSOD, and other information here.

 

[xtknight]

Most likely a bad driver of some sort. If not that, do a memory check (memtest86). Also a small possibility it's the PSU.

What is the error message on the BSODs? Is that consistent? Does it consistently list the same driver *.sys file causing it? Disable automatic reboot and enable minidumps as well.

 

[TechnoPro]

Originally posted by: xtknight
Most likely a bad driver of some sort. If not that, do a memory check (memtest86). Also a small possibility it's the PSU.

What is the error message on the BSODs? Is that consistent? Does it consistently list the same driver *.sys file causing it? Disable automatic reboot and enable minidumps as well.

Please see my edited post. I had neglected to link to the real technical information that I was able to gather.

 

[Matthias99]

In my experience, PAGE_FAULT_IN_NONPAGED_AREA is almost always bad memory or a flaky motherboard/memory controller.

Burn the bootable memtest86+ to a CD (or download the boot floppy version) and let it run on the machine overnight. Any errors indicate bad memory or a misbehaving memory controller (ie, bad motherboard), or rarely a CPU problem.

If that passes, download prime95 and run its torture tests ('Blend' seems to be the most demanding) for at least 12-24 hours. Errors indicate hardware instability, or possibly a corrupted OS/drivers (but usually bad hardware).

 

[dclive]

Don't guess. C'mon guys! Dumps are available.... with those, we don't need to guess - we can know what's going on (with a pretty good amount of certainty.)

You've posted the minidumps, so let's look at those. I looked at the most recent dump from 1-5-06, and that said your problem is Zone Lab's software:

Module[ 19] [C:\WINDOWS\SYSTEM32\VSDATANT.SYS]
Company Name: Zone Labs, LLC
File Description: TrueVector Device Driver
Product Version: (6.1:737.0)
File Version: (6.1:737.0)
File Size (bytes): 372816
File Date: Tue Nov 15 00:50:34 2005
Module TimeDateStamp = 0x43798224 - Tue Nov 15 01:37:24 2005
Module Checksum = 0x0006538c
Module SizeOfImage = 0x00059940
Module Pointer to PDB = [vsdatant.pdb]
Module PDB Signature = 0x43798224
Module PDB Age = 0x1

So remove that and let's see if the BSODs go away.

(By remove that, I mean completely remove all Zone Labs software on the machine, reboot, and enable the built-in Windows firewall, and then check how the machine works.)

 

[TechnoPro]

Originally posted by: dclive
Don't guess. C'mon guys! Dumps are available.... with those, we don't need to guess - we can know what's going on (with a pretty good amount of certainty.)

You've posted the minidumps, so let's look at those. I looked at the most recent dump from 1-5-06, and that said your problem is Zone Lab's software:

Module[ 19] [C:\WINDOWS\SYSTEM32\VSDATANT.SYS]
Company Name: Zone Labs, LLC
File Description: TrueVector Device Driver
Product Version: (6.1:737.0)
File Version: (6.1:737.0)
File Size (bytes): 372816
File Date: Tue Nov 15 00:50:34 2005
Module TimeDateStamp = 0x43798224 - Tue Nov 15 01:37:24 2005
Module Checksum = 0x0006538c
Module SizeOfImage = 0x00059940
Module Pointer to PDB = [vsdatant.pdb]
Module PDB Signature = 0x43798224
Module PDB Age = 0x1

So remove that and let's see if the BSODs go away.

(By remove that, I mean completely remove all Zone Labs software on the machine, reboot, and enable the built-in Windows firewall, and then check how the machine works.)

What tools do you use to analyze the minidumps? I opened this one up in notepad and could not find the string "ZONE" nore "Z O N E". Is there special software that can make this soemwhat more readable?

 

[Nothinman]

Don't guess. C'mon guys! Dumps are available.... with those, we don't need to guess - we can know what's going on (with a pretty good amount of certainty.)

If it is indeed a hardware problem you can't trust the minidumps. Bad memory, dying PSU, too much dust on some contacts, etc could all cause random BSODs that point to different drivers each time or just by dumb luck point to the same driver each time.

 

[VirtualLarry]

Bad caps, perchance?

 

[xtknight]

Originally posted by: TechnoPro
What tools do you use to analyze the minidumps? I opened this one up in notepad and could not find the string "ZONE" nore "Z O N E". Is there special software that can make this soemwhat more readable?

WinDbg tools with kd!analyze command.

Info for modules in D8F0RR61_MPSReports.CAB\D8F0RR61_SYSTEM32_SYS.TXT

 

[xtknight]

There's conflicting errors. Probably memory gone bad.

Probably caused by : vsdatant.sys ( vsdatant+3e6e1 )
Probably caused by : ntoskrnl.exe ( nt+9577b )
Probably caused by : ntoskrnl.exe ( nt+9577b )

Run memtest86 first.

 

[NogginBoink]

Originally posted by: Matthias99
In my experience, PAGE_FAULT_IN_NONPAGED_AREA is almost always bad memory or a flaky motherboard/memory controller.

Your experience and mine are different. My experience is that a STOP 0x50 is almost always a poorly written driver.

My experience includes working technical support for a large software company troubleshotting blue screens and reading dump files.

Dump Mini121405-01.dmp was caused by a system worker thread. It's impossible to tell from the dump what thread submitted the task for the worker thread to do; we'll never really know the cause of this crash.

Dump Mini100605-01.dmp was the same thing.

Dump Mini010506-01.dmp was a STOP 0xC2 and was caused by vsdatant.sys, which I believe dclive correctly identified as a Zone Alarm driver.

The best start to fixing this is to ensure that you're running the latest version of Zone Lab's drivers.

If problems persist, configure your machine to write a Kernel dump instead of a mini dump. (System control panel, advanced, startup and recovery.) A kernel dump has a lot more information in it.

 

[NogginBoink]

Originally posted by: TechnoPro
What tools do you use to analyze the minidumps? I opened this one up in notepad and could not find the string "ZONE" nore "Z O N E". Is there special software that can make this soemwhat more readable?

On Microsoft's site, you can download the "Debugging tools for Windows" packages.

However, unless you're a device driver author, and/or are VERY familiar with assembly language AND the internal workings of Windows, you're not likely to have much success in looking at dump files.

Kernel mode debugging is what sets the ubergeeks apart from the merely freakish geeks. It's part science, part art, but mostly black voodoo magic.

 

[evilsaint]

I've actually seen a lot of Dells (Desktops, notebooks, and even some of their lower-end servers) end up with these inevitable un-curable BSOD's, and recently, one of my colleagues discovered that Dell actually has a great tool for analyzing the drivers on any Dell machine, though ATM, I can't remember what it's called...

Anyways, have your associate enter the service tag of the machine on the Dell website, and then manually search for a 'driver analysis tool' or something like that; it's not listed in their normal tools for the individual machines. I wish I could say for certain that it'll help you come upon the fix for your BSOD's, though on four different Dells, i've seen it work to solve these wonderful little M$ quirks Good luck.

 

[TechnoPro]

A sincere thank you to all that helped me figure this BSOD out, especially dclive. I pulled ZoneAlarm from the affected system and waited roughly one month, which was beyond the upper limit of the bluse screen frequency, and the problem has yet to reappear. To be scientific, I could have put ZA back on, but I would most certainly use the latest version which may have corrected the issue, thus negating the accuracy of the test.