In general a custom error is generated by a custom rule from your development rulesets. Where a product error is coming from an Out-Of-The-Box rule coming from Pega rulesets.

Now having said that, sometimes an Out-Of-The-Box rule will be failing because a custom rule calling it doesn't have expected elements or parameters.

So it isn't always easy to identify if the problem is from Product or from Customization.