If you wrap a specific portion of your automation in a Try...Catch and want to catch a specific exception type, you can click the "+" icon at the bottom of the Catch block to add a new Exception to catch. Clicking the "..." icon on this new line will allow you to select the specific exception you want to catch. Exceptions will be logged unless you have disabled logging or have otherwise disabled logging for exceptions (unlikely as that requires deliberate action). If you want to write a specific message to the log, use the DiagnosticsLog component from the Toolbox on the Advanced tab. It has a method called "Log" that accepts a string that could be anything you want to write to the log. My suggestion would be to preface each log message with a specific set of characters ("***LOG - " for example) so that you can easily locate them.

When you enable the FilePublisher from either the Pega Robotics Runtime tray icon (or in Studio under Tools-->Options-->Pega Robotics-->Runtime Diagnostics) or in the RuntimeConfig.xml (which by default is located in %AppData%\OpenSpan), the file RuntimeLog.txt will be written to %AppData%\OpenSpan. There are various log levels, with verbose containing the most information, although exceptions will be logged at the lowest log level.

I am not sure what you are looking for with custom exceptions. Please provide more information and specific examples of what you are asking for.