When viewing large files, people typically tend to use a pager program such as more or preferably less. Neither of these buffers the entire file into memory, so it's quick to open a large file and scroll around. If you're using more, you should try less. It lets you scroll backwards in the file as well as forwards.
Out of the box, Vim doesn't do so great for really big files. Features like syntax highlighting and undo history can really slow things down on huge buffers; however, there is a plugin that can ease the pain. The LargeFile plugin is available here, and it really does seem to speed up opening and moving around in files larger than a few megs. A word of caution though, if you're dealing with really huge files, a pager program is still the way to go due to the sheer amount of memory required to manipulate those files.