PCI memory on a 64-bit Operating System with 4GB of RAM

[JustaGeek]

...or does it still reside in the ~3 - 4GB area, since all the *32 drivers are written for the 32-bit OS...?

Running 4GB of RAM on Vista 64, would I have all 4GB automatically available to use for the applications, or the PCI memory still "locks up" the PCI memory range ~3 - 4GB?

Someone said that the 64-bit OS would automatically remap the PCI beyond 4GB, so with 4GB of physical RAM installed, OS would not only "see" it, but "use" it as well, instead of "surrendering" it to the PCI, Video Memory etc.

Any thoughts...?

 

[pallejr]

On desktop boards I believe the normal operation is to remap the RAM around the MMIO space.

 

[JustaGeek]

Well, I strongly believe that this area will remain reserved for MMIO "forever", and never be made accessible to the applications.

And in order to avoid conflicts with the "32-bit philosophy", all the new 64-bit drivers and applications should be designed to access the area beyond the 4GB barrier. Only then the 32-bit applications and drivers will be able to "co-exist" without any potential conflicts with the 64-bit applications and drivers.

So today, 3GB of RAM or 4GB of RAM installed on a 64-bit OS makes absolutely no difference. Yes, the 64-bit OS will "see" it all, but it won't use it all for the applications, staying below ~3GB.

With the 6GB or 8GB (up to 128GB) of physical memory installed, the lower part up to 4GB will use the "older" 32-bit technology, and the "newly" created 64-bit applications will use the 4 - 128GB area exclusively, in perfect harmony.

Does it make sense...?

 

[QuixoticOne]

I didn't closely follow the thread about speculations / conclusions,
but I thought I'd add some empirical data to it.

In a 64 bit OS "it just works" to use basically all the memory.

A 32 bit OS can certainly use most if not all of 4GB of memory, and given
that 2x2GB DIMMs are probably more practical to use than 3x1GB, I'd
say just get 4GB and if you end up with a little that's left out (in 32 bits)
due to GPU / PCI memory spaces, it's no big deal, you'll still end up
with 3.xx GBY useful RAM which is good.

Empirically it looks like my BIOS mapped 5GB of memory at
physical addresses over 4GB, and the remaining system DRAM
and GPU / PCI memory at addresses less than that.

Running 64 bit linux, from a usable system capacity a
memory resources list currently shows 1.9GB of 'user memory' in use
out of a total of 7.8GB available, and doubtlessly the other 200MB of
that is probably for system / kernel use.

VISTA x64 takes advantage of the memory on the PC well too,
though most of the memory intensive stuff I do is UNIX. I am
pretty sure the
"BIOS-e820 BIOS-provided physical RAM map" below is set up by
the BIOS and the OS wouldn't have reason to change it regardless of
what OS you're running. There are 2-3 memory hole / remapping type
options in the BIOS that I'm sure affect just how the memory is
mapped by the BIOS.

From an ASUS motherboard with 8GB:
BIOS-provided physical RAM map:
[note 640K memory RAM region below]
BIOS-e820: 0000000000000000 - 000000000009ec00 (usable)
BIOS-e820: 000000000009ec00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
[note 3.0GB contiguous physical usable RAM below]
BIOS-e820: 0000000000100000 - 00000000bff80000 (usable)
BIOS-e820: 00000000bff80000 - 00000000bff8e000 (ACPI data)
BIOS-e820: 00000000bff8e000 - 00000000bffe0000 (ACPI NVS)
BIOS-e820: 00000000bffe0000 - 00000000c0000000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000ffe00000 - 0000000100000000 (reserved)
[note 4GB limit exceeded at addresses of 100000000 and over]
[note 5GB memory mapped below over 4GB address space]
BIOS-e820: 0000000100000000 - 0000000240000000 (usable)

ACPI: RSDP 000FBB80, 0024 (r2 ACPIAM)
ACPI: XSDT BFF80100, 0054 (r1 A_M_I_ OEMXSDT 6000726 MSFT 97)
ACPI: FACP BFF80290, 00F4 (r3 A_M_I_ OEMFACP 6000726 MSFT 97)
ACPI: DSDT BFF805C0, 8C47 (r1 A0761 A0761001 1 INTL 20060113)
ACPI: FACS BFF8E000, 0040
ACPI: APIC BFF80390, 006C (r1 A_M_I_ OEMAPIC 6000726 MSFT 97)
ACPI: MCFG BFF80400, 003C (r1 A_M_I_ OEMMCFG 6000726 MSFT 97)
ACPI: OEMB BFF8E040, 0081 (r1 A_M_I_ AMI_OEM 6000726 MSFT 97)
ACPI: HPET BFF89210, 0038 (r1 A_M_I_ OEMHPET 6000726 MSFT 97)
ACPI: OSFR BFF89250, 00B0 (r1 A_M_I_ OEMOSFR 6000726 MSFT 97)


01:00.0 VGA compatible controller: nVidia Corporation G80 [GeForce 8800 GTX] (rev a2) (prog-if 00 [VGA])
Subsystem: XFX Pine Group Inc. Unknown device 2250
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at f8000000 (32-bit, non-prefetchable) [size=16M]
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at f6000000 (64-bit, non-prefetchable) [size=32M]
I/O ports at ac00
[virtual] Expansion ROM at f9de0000 [disabled] [size=128K]
Capabilities: [60] Power Management version 2
Capabilities: [68] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
Capabilities: [78] Express Endpoint, MSI 00
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nvidia


04:00.0 VGA compatible controller: nVidia Corporation Unknown device 0402 (rev a1) (prog-if 00 [VGA])
Subsystem: eVga.com. Corp. Unknown device c756
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at fd000000 (32-bit, non-prefetchable) [size=16M]
Memory at d0000000 (64-bit, prefetchable) [size=256M]
Memory at fa000000 (64-bit, non-prefetchable) [size=32M]
I/O ports at dc00 [disabled]
[virtual] Expansion ROM at feae0000 [disabled] [size=128K]
Capabilities: [60] Power Management version 2
Capabilities: [68] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
Capabilities: [78] Express Endpoint, MSI 00
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nvidia


00:01.0 PCI bridge: Intel Corporation 82G33/G31/P35 Express PCI Express Root Port (rev 02) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000a000-0000afff
Memory behind bridge: f6000000-f9dfffff
Prefetchable memory behind bridge: 00000000c0000000-00000000cfffffff
Capabilities: [88] Subsystem: ASUSTeK Computer Inc. Unknown device 8295
Capabilities: [80] Power Management version 3
Capabilities: [90] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+
Capabilities: [a0] Express Root Port (Slot+), MSI 00
Kernel driver in use: pcieport-driver
Kernel modules: shpchp

00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 02) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: 0000d000-0000dfff
Memory behind bridge: fa000000-feafffff
Prefetchable memory behind bridge: 00000000d0000000-00000000dfffffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+
Capabilities: [90] Subsystem: ASUSTeK Computer Inc. Unknown device 8277
Capabilities: [a0] Power Management version 2
Kernel driver in use: pcieport-driver
Kernel modules: shpchp

00:1c.4 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 5 (rev 02) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: 0000c000-0000cfff
Memory behind bridge: f9f00000-f9ffffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+
Capabilities: [90] Subsystem: ASUSTeK Computer Inc. Unknown device 8277
Capabilities: [a0] Power Management version 2
Kernel driver in use: pcieport-driver
Kernel modules: shpchp

00:1c.5 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 6 (rev 02) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: 0000b000-0000bfff
Memory behind bridge: f9e00000-f9efffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+
Capabilities: [90] Subsystem: ASUSTeK Computer Inc. Unknown device 8277
Capabilities: [a0] Power Management version 2
Kernel driver in use: pcieport-driver
Kernel modules: shpchp

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01 [Subtractive decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=05, subordinate=05, sec-latency=32
I/O behind bridge: 0000e000-0000efff
Memory behind bridge: feb00000-febfffff
Capabilities: [50] Subsystem: ASUSTeK Computer Inc. Unknown device 8277

 

[QuixoticOne]

Not really; keep in mind that the APPLICATIONS in modern OSes
are using VIRTUAL memory, so EVERY APPLICATION has potentially
its OWN 4GB of VIRTUAL memory address space. And every 4KB
of that space is INDEPENDENTLY virtualized to any physical
address. So it's quite possible to VM map a PHYSICAL memory
region of 2GB size at physical addresses of 5GB-7GB to a VIRTUAL
address of 0GB-2GB as far as the application accesses / sees it.

So 32 bit applications running on a 64 bit OS aren't really at all
limited from using ALL of the memory the machine has.

The only potential problem is related to whether the combination
of CPU IOMMU feature giving better virtualized DMA control
(or its lack thereof), the system device drivers,
and devices 32 vs 64 bit DMA capabilities somewhat limit your
choices of memory mappings and virtualizations.

A typical application doesn't directly do I/O or need to have DMA
directly into its address space with hardware devices so even if it's
a 32 bit application that's totally virtualized on a 64 bit host it'll
be able to access any/all 4GB memory in the PC whether that
memory is broken up into any number of non-contiguous regions,
and whether any of those regions are physically above 4GB.

Also if I recall correctly even 32 bit CPUs often have significantly
more than 32 bit physical memory access capabilities and some MMU
support for mapping parts of those that > 4GB memory space into
a software addressable 4GB space that's more useful. There
are special distinct address "regions" for IO which are different
(in hardware cycles) than address regions corresponding to memory,
AGP has its own MMU / address mapping (GART) indepent of and on
top of the CPU's MMU, etc. for instance.

I know you *can* configure PCI configuration space to be memory mapped
(MMIO) as opposed to using configuration space IO cycles, though I
gather that's not entirely well / fully implemented at this point, so
it may be that the address space impact isn't as severe as it could be
if it were all mmaped.

Also notice that even my 8800GTX which has 768MB physical memory
on it, there's only a 256MB size memory window it occupies.

So today, 3GB of RAM or 4GB of RAM installed on a 64-bit OS makes absolutely no difference. Yes, the 64-bit OS will "see" it all, but it won't use it all for the applications, staying below ~3GB.

With the 6GB or 8GB (up to 128GB) of physical memory installed, the lower part up to 4GB will use the "older" 32-bit technology, and the "newly" created 64-bit applications will use the 4 - 128GB area exclusively, in perfect harmony.

Does it make sense...?

 

[Nothinman]

...or does it still reside in the ~3 - 4GB area, since all the *32 drivers are written for the 32-bit OS...?

It'll stay there mostly because most PCI devices can still only access 32-bit addresses whether the driver is 64-bit or not.

Running 4GB of RAM on Vista 64, would I have all 4GB automatically available to use for the applications, or the PCI memory still "locks up" the PCI memory range ~3 - 4GB?

The memory will be available in both cases because processes only ever see virtual addresses, where the data is in physical memory is irrelevant.

Someone said that the 64-bit OS would automatically remap the PCI beyond 4GB, so with 4GB of physical RAM installed, OS would not only "see" it, but "use" it as well, instead of "surrendering" it to the PCI, Video Memory etc.

Someone was wrong, the kernel looks at the memory map provided by the BIOS and just uses that to determine what memory is available.

And in order to avoid conflicts with the "32-bit philosophy", all the new 64-bit drivers and applications should be designed to access the area beyond the 4GB barrier. Only then the 32-bit applications and drivers will be able to "co-exist" without any potential conflicts with the 64-bit applications and drivers.

Applications shouldn't be designed with any memory layout in mind, virtual memory allows the OS to assign memory from any region to any process without the process caring where it comes from.

Drivers have to be designed with the limitations of the hardware in mind and most cheap PCI devices can only handle 32-bit addresses. Hell some can't even handle that for DMA.

So today, 3GB of RAM or 4GB of RAM installed on a 64-bit OS makes absolutely no difference. Yes, the 64-bit OS will "see" it all, but it won't use it all for the applications, staying below ~3GB.

Not true at all. Any process will be able to use all of the memory that the OS sees.

With the 6GB or 8GB (up to 128GB) of physical memory installed, the lower part up to 4GB will use the "older" 32-bit technology, and the "newly" created 64-bit applications will use the 4 - 128GB area exclusively, in perfect harmony.

Does it make sense...?

Nope, because you don't understand how virtual memory works. Processes never see physical addresses so they could be sitting at the 1G mark or the 100G mark in memory and never know the difference.

 

[JustaGeek]

Originally posted by: Nothinman

With the 6GB or 8GB (up to 128GB) of physical memory installed, the lower part up to 4GB will use the "older" 32-bit technology, and the "newly" created 64-bit applications will use the 4 - 128GB area exclusively, in perfect harmony.

Does it make sense...?

Nope, because you don't understand how virtual memory works. Processes never see physical addresses so they could be sitting at the 1G mark or the 100G mark in memory and never know the difference.

So in the Virtual Memory space, the 32-bit applications will be able to reside "side-by-side" with the future 64-bit processes, no matter where the process is allocated, because the 64-bit OS will take care of proper allocation...?

What I mean is that ex. a 32-bit game being allocated the area below the 2GB current application limit, will be able to "co-exist" with a 64-bit application, which will use whatever space is available - the remainder of the available memory below the 2GB barrier, and whatever else is required above...?

Is that a correct assumption...?

EDIT: Actually, for the processes it doesn't matter if they are 32-bit or 64-bit, and the larger amount of Physical Memory will just increase the Virtual Memory space available to them. The 64-bit OS will take care of the proper assignment of the available physical memory, right...?

I am learning, please be patient...

 

[Peter]

In physical address space, the MMIO will always be in 32-bit-accessible space. The main reason is that lots of MMIO hardware doesn't even support getting mapped into >4G space, and that includes core architecture items like interrupt controllers, timers, and PCIE memory mapped configuration space (see above example, HPET, APIC and MCFG). The secondary reason, for legacy BIOS machines but not EFI machines, is that when the memory map gets built and the PCI(E) hardware gets mapped, the firmware doesn't know whether a 32- or a 64-bit OS is going to run.

Note that "device cannot reside in >4G space" is fundamentally different topic to "device cannot access >4G space". The former is PCI target capability, the latter is busmaster DMA, and no, they aren't connected, and they aren't connected to the physical PCI bus width either.

So the firmware maps the MMIO growing down from 4G, reduces the visible RAM below 4G to eliminate any overlap with MMIO, and if the chipset can do it, remaps this reduction amount to >4G space. As in the ASUS example given above - <4G RAM ends at 3G, with devices and PCIE MMCFG space above it. The 'missing' gigabyte of RAM reappears in >4G space.

And finally, given how modern processors are going to reshuffle their internal view of the physical world through a highly capable MMU anyhow, it simply doesn't make any difference whatsoever what the physical map looks like. Whether your RAM is contiguous, has a couple of holes in it, or is sparsely distributed all over the place, your (modern) OS will be able to handle it.

 

[JustaGeek]

Originally posted by: Peter
In physical address space, the MMIO will always be in 32-bit-accessible space. The main reason is that lots of MMIO hardware doesn't even support getting mapped into >4G space, and that includes core architecture items like interrupt controllers, timers, and PCIE memory mapped configuration space (see above example, HPET, APIC and MCFG). The secondary reason, for legacy BIOS machines but not EFI machines, is that when the memory map gets built and the PCI(E) hardware gets mapped, the firmware doesn't know whether a 32- or a 64-bit OS is going to run.

Note that "device cannot reside in >4G space" is fundamentally different topic to "device cannot access >4G space". The former is PCI target capability, the latter is busmaster DMA, and no, they aren't connected, and they aren't connected to the physical PCI bus width either.

So the firmware maps the MMIO growing down from 4G, reduces the visible RAM below 4G to eliminate any overlap with MMIO, and if the chipset can do it, remaps this reduction amount to >4G space. As in the ASUS example given above - <4G RAM ends at 3G, with devices and PCIE MMCFG space above it. The 'missing' gigabyte of RAM reappears in >4G space.

And finally, given how modern processors are going to reshuffle their internal view of the physical world through a highly capable MMU anyhow, it simply doesn't make any difference whatsoever what the physical map looks like. Whether your RAM is contiguous, has a couple of holes in it, or is sparsely distributed all over the place, your (modern) OS will be able to handle it.

It re-appears in >4GB space, because he has 8GB of RAM installed.

If only 4GB of RAM is installed on a 64 bit system, the MMIO still uses the <4GB, so even if the whole 4GB of physical memory is visible by the 64 bit system, the ~3-4GB area is still inaccessible to applications, correct...?

It means, use 3GB of RAM, or if you want more, install not only 4GB, since the gain might be only a few hundred megabytes depending on the hardware installed (just like in a 32-bit OS), but install 6 GB or 8GB, so you can fully use the area >4GB, with some devices remapped from below 4GB by the chipset if necessary, and if possible.

Correct...?

 

[Peter]

Sorry, you're not making much sense there.

The RAM that would end up in 3-4G space (but can't because there's I/O there) makes its appearance relocated above 4G. You're not losing a single bit of RAM, and that's what you so far fail to understand.

Looking at the ASUS example above, it has RAM up to 0C0000000h (3G), and then again from 100000000h to 2400000000h, which happens to be 5G. 3+5=8 in my math book, so all is fine.

If this were a 4G machine, it'd have RAM up to 0C0000000h and then from 100000000h to 140000000h (1G), and that's how you get all of the four gigs you purchased - just not in one piece. 32-bit operating systems can't reach to 100000000h and above, and that's why it can't get to the remapped "missing" gigabyte.

What you also fail to understand is that the chipset doesn't "remap devices" - it doesn't even map them. Relocating anything after the RAM map has been set up doesn't win you anything anyhow. My previous post has explained why.

 

[Nothinman]

EDIT: Actually, for the processes it doesn't matter if they are 32-bit or 64-bit, and the larger amount of Physical Memory will just increase the Virtual Memory space available to them. The 64-bit OS will take care of the proper assignment of the available physical memory, right...?

The amounts of virtual memory and physical memory are unrelated. A 32-bit process always have 4G of VM even if you have 512M of memory or 16G of memory. Right now AMD64 chips don't implement the full 64-bit addressing but instead they only did 48-bits so instead of each process getting 16EB they get 256TB until AMD decides to expand it. But the idea is still the same, each process has it's own personal virtual address space that's the same size and consists of the same range of virtual addresses (0 to 256TB) as every other process on the system.

 

[JustaGeek]

Originally posted by: Peter
Sorry, you're not making much sense there.

The RAM that would end up in 3-4G space (but can't because there's I/O there) makes its appearance relocated above 4G. You're not losing a single bit of RAM, and that's what you so far fail to understand.

Looking at the ASUS example above, it has RAM up to 0C0000000h (3G), and then again from 100000000h to 2400000000h, which happens to be 5G. 3+5=8 in my math book, so all is fine.

If this were a 4G machine, it'd have RAM up to 0C0000000h and then from 100000000h to 140000000h (1G), and that's how you get all of the four gigs you purchased - just not in one piece. 32-bit operating systems can't reach to 100000000h and above, and that's why it can't get to the remapped "missing" gigabyte.

What you also fail to understand is that the chipset doesn't "remap devices" - it doesn't even map them. Relocating anything after the RAM map has been set up doesn't win you anything anyhow. My previous post has explained why.

If I could understand it, I wouldn't obviously be asking these questions here.

And I appreciate your assistance.

And you said that chipset can remap it here:

"So the firmware maps the MMIO growing down from 4G, reduces the visible RAM below 4G to eliminate any overlap with MMIO, and if the chipset can do it, remaps this reduction amount to >4G space"

The way I understand it (and it might be incorrect) is that if the specific space is occupied by the devices, how can it be made accessible to applications...?

Unless, of course, the 64-bit system takes the 4GB of physical RAM installed, and it "adds" it to the MMIO memory, for a theoretical total of ~4.75GB. In this case, it doesn't matter that the overlap exists in "my" mind - the 64-bit OS still uses the full 4GB.

Does that make more sense...?

 

[Nothinman]

The way I understand it (and it might be incorrect) is that if the specific space is occupied by the devices, how can it be made accessible to applications...?

By remapping the addresses by which the OS uses it above the 4G mark. That's the sole purpose of the memory remap, memory hole, whatever your BIOS calls it function.

Unless, of course, the 64-bit system takes the 4GB of physical RAM installed, and it "adds" it to the MMIO memory, for a theoretical total of ~4.75GB. In this case, it doesn't matter that the overlap exists in "my" mind - the 64-bit OS still uses the full 4GB.

If by "the 64-bit system" you mean the chipset an the BIOS, then yes, but the fact that the hardware is 64-bit is beside the point. A 32-bit system could do the same thing.

 

[pallejr]

http://www.intel.com/design/ch...ts/datashts/316966.htm

If you are technical, try one of Intels specs, it might give you a clue on how it actually works, when the chipset "reclaims" the lost memory. There is also a bit of graphics in there

 

[JustaGeek]

Yes, I am "technical", but in my world 2 + 2 = 4, and everything stays where it is constructed.

Thank you all for the great explanation, and thanks for the link, pallejr.

Let me study the Intel datasheet now...

 

[Nothinman]

You also might want to get an OS design book like Inside Windows or Understanding the Linux kernel and read the chapter(s) on memory management. Once you understand how physical and virtual memory work it makes a lot more sense.

 

[JustaGeek]

Thank you, Nothinman. I will...

 

[Peter]

JustaGeek, reading your output again, I think what blocks your understanding is: You need to figure out the difference between "memory" and "address space".


In the beginning, there was address space. Four gigabytes of it in a 32-bit machine, much more in a 64-bit machine.

The address space was all empty. Then BIOS spoketh: "Let there be RAM, and lots of it." And thus the RAM was detected and enabled, henceforth paving the address space from the near end. And it was good.

And BIOS spoke further: "Let there be devices." And the devices appeared on the far end and grew inward, and it was good.

Until the devices grew over the RAM. That was Not Good.

So BIOS spoke: "Ah well then", pulled the grown-over RAM from under the devices, and neatly reused it on new ground beyond the point where the RAM starts growing down.


In other words: The CPU can address a certain range, 4G in 32-bit mode, 64G in Intel's and one Terabyte in AMD's 64-bit mode. Everything gets mapped into this address range - RAM and all other devices.

BIOS maps the detected amount of RAM starting at address 0, and then does the devices, growing downward from the 4G threshold (for reasons already established earlier, devices need to reside in 32-bit space).

When the address spaced consumed by devices does not grow down far enough to overlap into the space the RAM got mapped into, all is fine and we're done.

When there is overlap, BIOS then revisits the RAM mapping. The RAM that got overlapped gets pulled from where it was mapped, and instead mapped into available address space beyond the 4G barrier. In 32-bit mode, the CPU can't get there, and hence this bit of RAM is "lost"; in 64-bit mode it can, nothing lost.


Let's take the ASUS example above through all those steps:

Available space is 64G, addresses from 0.0000.0000 to F.FFFF.FFFF (periods added to be more readable).

8G RAM are detected, and mapped into address space from 0.0000.0000 to 1.FFFF.FFFF.

Devices are detected, and mapped into address space from 0.FFFF.FFFF to 0.C000.0000 (growing downward, remember?)

So there is overlapped RAM from 0.C000.0000 to 0.FFFF.FFFF. BIOS reprograms the RAM controller to show this piece of RAM at the end of the regularly mapped RAM instead, from 2.0000.0000 to 2.3FFF.FFFF.

 

[Nothinman]

In 32-bit mode, the CPU can't get there, and hence this bit of RAM is "lost"; in 64-bit mode it can, nothing lost.

As long as PAE is supported the CPU can get there just fine, this is how Linux, *BSD, Win2K3 Server Enterprise, etc all get access to >4G of memory on IA32 machines.

 

[Peter]

PAE is Physical Address Extensions, which hints what's going on - you are not physically in 32-bit mode, but in fact in 36-bit mode.

In this mode, the CPU still does only 32-bit math, which is why this mode isn't too popular with developers. But let's not go there - what we need to know for this topic is that PAE mode lives in 36-bit address space.

 

[Nothinman]

Yes but since it's what 32-bit OSes use to access >4G of memory it's relevant. Most developers never even have to consider PAE since the OS does 99% of the translation for them. The few cases that do take PAE into consideration already have and were using something like AWE to work around address space limtations regardless of whether PAE was in use or not.

 

[Peter]

It sure is relevant. However, since it is just another method of generating physical addresses from inside the CPU, it doesn't make a difference to what I've been explaining about the physical address map.

 

[Nothinman]

True PAE doesn't affect the physical address map, but neither does whether the OS is 64-bit or 32-bit since the BIOS sets it up before the OS is loaded and it's up to the OS to either be 64-bit or support PAE to get to any addresses remapped above the 4G mark.

 

[Peter]

Exactly that's why it's irrelevant to the topic of physical address map discussed here ... you actually agree with me, you just haven't noticed yet

 

[Nothinman]

Yea, I'm not sure what I was getting at yesterday. Probably just the fact that most people consider a CPU in PAE mode still in 32-bit mode because it still runs a 32-bit OS which might further confuse the OP once he gets to that point. =)