Thanks for your reply. Can you please let me know what should be passed as parameter for that section. While I passes the current Page and try, messages are shown on the top but still modal window is closed.
Note: I have launched modal window from user created Page
If possible you can create the custom modal template with the reference of the OOTB modal template and customize the close screen button to not to close the window if there are any errors in the validation.
You can use the when condition in the close screen button with the function @HasMessages(PageName). If it is true then don't close the window.