Put the following code in your .onInit function:
System::Call 'kernel32::CreateMutexA(i 0, i 0, t "myMutex") i .r1 ?e' Pop $R0 StrCmp $R0 0 +3 MessageBox MB_OK|MB_ICONEXCLAMATION "The installer is already running." Abort
'myMutex' should be replaced by a unique value.