Using InstallScript to Implement Custom Dialogs

InstallShield 2012 Spring

Project: This information applies to the following project types:

InstallScript MSI

The next step in creating a custom dialog is to write an InstallScript function that processes the end user's interaction with the dialog controls.

To use InstallScript to implement a custom dialog in an InstallScript project:

1. Begin by creating a new InstallScript source file to contain the custom dialog code. Name the file CustomDialog.rul.

Tip: To be able to use the dialog script in other projects, copy the .rul file to another location. In your other projects, open the InstallScript view, right-click the Files item in the InstallScript explorer, and click Insert Script Files.

2. Inside CustomDialog.rul, place the prototype and implementation for your custom dialog function. Add the following code to CustomDialog.rul.

prototype NUMBER CustomDialog( );

function NUMBER CustomDialog( )



3. When you compile your script, indicate that the main script Setup.rul should include the code for your CustomDialog function by inserting the following line near the top of Setup.rul:

#include "CustomDialog.rul"



4. In the CustomDialog.rul script, define the constants that store the numeric IDs of the controls that you added to the dialog. If you copied the Back, Next, and Cancel buttons from a standard dialog, you can add the following lines near the top of CustomDialog.rul:

// control identifiers

#define BUTTON_NEXT    1

#define BUTTON_BACK   12

In general, the numeric ID of a control on a dialog is the number listed in the control's Control Identifier property, displayed in the property list when you select the control in the Dialog Editor.

Important: To enable end users to cancel the installation by clicking the close button in the upper-right corner of the InstallScript dialog, the dialog must have a button control with a value of 2 for the Control Identifier property. You can place this button anywhere on the dialog, and if necessary, you can make this button invisible. This is necessary because the InstallScript engine passes the WM_CLOSE message to the script only if it comes from a valid control in the dialog. If the dialog does not have a button with this identifier, the installation does not pass through the message that is generated when the dialog's close button is clicked.

You need to define additional constants for every other control (for example, check box, edit field, or list box) with which the end user can interact.

5. Your CustomDialog function loads the custom dialog into memory using the EzDefineDialog function:


  "CustomDialog", // nickname for dialog

  ISUSER,         // DLL containing the dialog's resources

  "CustomDialog", // name of dialog in Dialogs view

  0);             // numeric resource ID for dialog; not used here

To learn which arguments you use with EzDefineDialog, see EzDefineDialog.

Tip: The resource ID of a dialog is the dialog's name, as it appears in the Dialog Editor. If you need to specify a numeric resource ID, you can add a numeric ID to the ISResourceId column in the Dialog table for the custom dialog. You can access the Dialog table in the Direct Editor.

6. Create a message loop in your script for the custom dialog. The message loop repeatedly calls the function WaitOnDialog, which returns the numeric control ID for the control with which user interacts with. A typical message loop appears as follows.

// repeatedly call WaitOnDialog until the user exits the dialog

// with the Next, Back, or Cancel button

while (!bDone)


// wait for the user to interact with a control,

// then return its ID

nCtrl = WaitOnDialog("CustomDialog");


// use a switch statement to handle the different controls

switch (nCtrl)


    case CONTROL1:

        // do something when user clicks CONTROL1


    case CONTROL2:

        // do something when user clicks CONTROL2


    // case statements for other controls





For example, CustomDialog currently contains Next, Back, and Cancel buttons. Its message loop might appear as follows:

while (!bDone)


nControl = WaitOnDialog("CustomDialog");


switch (nControl)


    case DLG_INIT:

        // Initialize the back, next, and cancel button enable/disable

        // states for this dialog and replace %P, %VS, %VI with


        // IFX_INSTALLED_DISPLAY_VERSION, respectively, on control IDs 700-724 and 202.

        hwndDlg = CmdGetHwndDlg("CustomDialog");

        SdGeneralInit("CustomDialog", hwndDlg, 0, "");


    case BUTTON_BACK:

        // user clicked Back

        nReturn = BUTTON_BACK;

        bDone = TRUE;


    case BUTTON_NEXT:

        // user clicked Next

        nReturn = BUTTON_NEXT;

        bDone = TRUE;




        // check standard control handling

        if (SdIsStdButton(nControl) && SdDoStdButton(nControl)) then

            bDone = TRUE;






7. When the end user exits the dialog by clicking Back or Next, you should remove the dialog from the screen and from memory using EndDialog and ReleaseDialog:



To use the dialog in the main script, add a call to the CustomDialog function in, for example, the OnFirstUIBefore event handler of Setup.rul.


    szTitle = "";

    szMsg   = "";

    nResult = SdWelcome(szTitle, szMsg);



    nResult = CustomDialog( );

    if (nResult = BUTTON_BACK) goto Dlg_SdWelcome;


// etc.

If the end user clicks Back or Next, the script displays the previous or following dialog. If the user clicks Cancel, the standard confirmation dialog (handled by the OnCanceling event handler) is displayed.

Note: For information on implementing dialog control functionality, see Using InstallScript to Process Dialog Controls.

Special Messages

In addition to returning control identifiers, the WaitOnDialog function returns some special messages.

Immediately before the dialog is displayed on the screen, WaitOnDialog returns the message constant DLG_INIT. In the DLG_INIT case statement, you can set the default states of check boxes and radio buttons, populate and set the current selection in a list box or combo box control, or set the initial text of an edit field.
If an error occurs, WaitOnDialog returns the constant DLG_ERR.