I would like to share my feedback reg this tool. Please go through the below list
It is not really clear from available documents on the web when this tool should be run (before or after the upgrade?). I managed to find out that it should be run on upgraded test env. However, knowing now how this tool works, I think it doesn't matter when we run it. We are able to run it before upgrading the env and check what is the health of our app- this should be the recommended way I believe but unfortunately, it is not so easy to do this now because.... let's jump to the next point.
It is not possible to get the latest version of the tool from the website. The only thing which we can get from the marketplace is the list of critical rules which the tool checks. Because of this, I was not able on 8.4 to install the product rule with a list of rules. It would be good to have the possibility to get the latest version of the product as from I noticed there were some improvements in the tool done
The tool doesn't take into account the major ruleset version of rules during the checking process. For example: if there is OOTB Final rule overridden in the implementation ruleset TEST: 01-01-01 but our app which we want to check contains rulest Test: 02-01 where this rule is not available then despite this, the rule from RS: TEST: 01-01-01 will be returned as the one which needs to be checked.
If an application which we want to check contains rulesets with the path version included then the upgrade checker won't work, for example, App: Test contains ruleset OrgData: 02-02-05 - in such cases tool won't return anything
The tool doesn't check references to rules which are not used anymore in the latest version of the product.
Hope the above list helps others and will improve the tool in the future.