It's not possible to undo the branch merge, but while branch merging there is an option to keep the code in branch eventhough it is merged. If this option was selected, then we can manually delete the rules from the merged ruleset. 

If this option is not selected then need to find the rules that was merged from this branch to the actual ruleset. We can find the list of rules from the merged ruleset by writing an report with the conditions like commitdatetimestsmp and operator I'd. Then do the delete. If delete is not possible, try to withdraw those rules in the next ruleset version.

@dazzlynchua You can make use of delete ruleset wizard and delete the merged branch from application. 

Non versioned rules can be deleted by pulling the report as Siva mentioned and delete manually.

Hope this helps.