[NOTE: This question and answer aren't relevant to After Effects CS5 and later. After Effects CS5 is a 64-bit application and can therefore see and use all of the RAM in a computer.]
Each instance of the After Effects application can use 2-4GB of RAM for rendering (either for previews or final output). If you have multiple processor cores and have enabled Render Multiple Frames Simultaneously multiprocessing, each core can use this amount of RAM for rendering.
From the "Memory usage and storage" page of After Effects Help:
"The operating system imposes certain limits on the amount of memory that an application can use. After Effects on the Mac OS X operating system can use up to 3.5 GB of RAM, although only about 3 GB is actually available to the foreground application, because Mac OS X uses approximately 500 MB to load the user interface libraries. After Effects on 32-bit Windows operating systems can use up to 3 GB of RAM; however, to use more than 2 GB in After Effects, you must configure Windows XP or Windows Vista appropriately.... After Effects on 64-bit Windows operating systems can use up to 4 GB of RAM with no special configuration.
Note: These numbers are for each After Effects process. The background processes used to render multiple frames simultaneously can each use the amount of RAM mentioned above."
Of course, you can only use more than 4GB with a 64-bit operating system. See "CS4 Production Premium on 64-bit operating systems".
There's also the separate question of how much RAM can be used for the RAM cache:
In the Memory & Multiprocessing preferences, After Effects reports the size of the RAM cache---the amount of RAM that can be filled with frames for RAM preview. With Render Multiple Frames Simultaneously turned off, this value is reported as Total After Effects Memory Usage. With Render Multiple Frames Simultaneously turned on, this value is reported as Foreground Memory Usage. This value will not be the full amount of RAM allocated to the foreground application. Rather, it is just the part that can be filled with frames for RAM preview. For example, on a Mac, this value might be 1.79GB. 4GB is the theoretical limit for an After Effects process (because it's 32-bit). The OS takes a bite, getting you down to ~3GB. The RAM cache is only allowed to use 60% of that, which gets you down to 1.79GB. The limitation of the RAM cache to 60% of the memory allocated to the foreground process is to avoid fragmentation problems and therefore out-of-memory errors. The terminology can be confusing in part because in After Effects CS3 we had a Maximum Memory Usage item and a Maximum RAM Cache Size item, and in After Effects CS4 the thing called Foreground Memory Usage maps more closely to the latter, not the former.