In any 32-bit operating system, you are limited to 4096MB [4GB] of RAM simply because the size of a 32-bit value will not allow any more. On a 32-bit system, each process is given 4 GB of virtual memory to play with, which is separated into 2 GB of user space that the application can actually use at a time.
The 4 GB limit for Windows, at least, is also a factor of licensing—the home versions of 32-bit Windows, while technically being able to support PAE[Physical Address Extension], have a hard limit of 4 GB for licensing and driver compatibility reasons.
Not only does 32-bit have a hard limit for the amount of memory it can address, there's also another problem: your devices, like your video card and motherboard BIOS take up room in that same 4 GB space, which means the underlying operating system gets access to even less of your RAM.
While 32 bits of information can only access 4 GB of RAM, a 64-bit machine can access 17.2 BILLION gigabits of system memory, banishing any limits far into the future. This also means that your video cards and other devices will not be stealing usable memory space from the operating system.Windows 64-bit Home editions are still limited to 16 GB of RAM for licensing reasons, but the Professional and Ultimate versions can use up to 192 GB of RAM, so keep that in mind when building that killer system.
On Windows, the 64-bit versions also come with a technology to prevent hijacking the kernel, support for hardware-enabled data execution protection, and mandatory digitally signed 64-bit device drivers.
Do 32-bit Applications Work on 64-Bit?
The vast majority of your 32-bit applications will continue to work just fine on 64-bit Windows, which includes a compatibility layer called WoW64, which actually switches the processor back and forth between 32-bit and 64-bit modes depending on which thread needs to execute—making 32-bit software run smoothly even in the 64-bit environment.
There are some exceptions to that rule, however: 32-bit device drivers and low-level system applications like Antivirus, shell extensions that plug into Windows, and some media applications simply won't work without a 64-bit equivalent.
Which Should I Use?
If you are ordering a new PC with 4 GB or more of RAM, you should probably be running a 64-bit version of Windows so you can use all of the available memory, especially if you want a rig with a large video card