Once the clips are fully cached there should be no "building preview" delay. With very high-def or large files this can take some time. Let the green line complete.
Adding effects may cause re-caching, but that delay should be minimal.
With 2/4K files, consider proxies - a common practice. Smaller versions of the files are used for editing, then the originals are substituted for export.
See the Proxy Editing sticky at the top of this forum. Also. if your PC runs Windows 64-bit, install the VP 64-bit release, which better utilizes memory.