Question
Pegasystems Inc.
AU
Last activity: 4 Oct 2018 13:54 EDT
Any differences between committed and uncommitted hotfixes?
This seems like a dumb question, but I'll ask anyway... Since we got to a point where Hotfixes could be rolled back, I'm wondering what the value is in actually committing a hotfix?
Does an uncommitted hotfix behave 100% the same as a committed hotfix? Does this include code changes? I'm presuming it does, and therefore, what is the value for me to go from a state where I can back it out (uncommitted) to one where I can't back it out (committed)?
Anyone who's had a bad experience with Hotfixes in the past will attest that one of the factors is the lack of any regression test plan provided with them. There's never any supporting material that suggests what I should check to see hasn't been impacted by the hotfix. As a result, at what point am I ever satisfied that the hotfix has not introduced a regression that I simply haven't stumbled across yet? Answer: never.
So why would I commit hotfixes if leaving them uncommitted gives me the same functional outcome whilst leaving the door open to back out a hotfix subsequently found to be destructive.
***Edited by Moderator Marissa to update categories***
**Moderation Team has archived post**
This post has been archived for educational purposes. Contents and links will no longer be updated. If you have the same/similar question, please write a new post.
Accepted Solution
Pegasystems Inc.
US
Braam,
Yes, Pega 7 doesn't have anywhere near as large a dependency web as the PRPC 6 series (and older). The move to more frequent, fully tested releases allows us to avoid that. As does the focus on minimal impact when migrating to new releases. The idea is that you should be moving to the latest Pega 7 releases and not digging in for years on an older one. The infrastructure was not in place to do that in the 6 and earlier series. Additionally, we've been pushing back on making HFixes if there are viable alternatives (or the issue is already fixed in an existing future release). That said, for ANY release, an HFix is only ever unit tested, which means we test to confirm it fixes the issue reported. As you point out, the scenarios you never thought to test are not being looked for by the developers either. It's why I would generally discourage taking HFixes if you aren't experiencing the problem it addresses (unless it's a dependency, in which case you absolutely need it).
Some of the hundreds of hotfixes are due to one HFix breaking something, but many are just patches to keep up with the changing environment. PRPC 6.3 was released, several browser versions ago. mobile was a cutting edge feature that wasn't fully baked into the product the way it is today. We did a lot of work to support all of that. And of course that work, done as HFixes, can introduce issues.
Braam,
Yes, Pega 7 doesn't have anywhere near as large a dependency web as the PRPC 6 series (and older). The move to more frequent, fully tested releases allows us to avoid that. As does the focus on minimal impact when migrating to new releases. The idea is that you should be moving to the latest Pega 7 releases and not digging in for years on an older one. The infrastructure was not in place to do that in the 6 and earlier series. Additionally, we've been pushing back on making HFixes if there are viable alternatives (or the issue is already fixed in an existing future release). That said, for ANY release, an HFix is only ever unit tested, which means we test to confirm it fixes the issue reported. As you point out, the scenarios you never thought to test are not being looked for by the developers either. It's why I would generally discourage taking HFixes if you aren't experiencing the problem it addresses (unless it's a dependency, in which case you absolutely need it).
Some of the hundreds of hotfixes are due to one HFix breaking something, but many are just patches to keep up with the changing environment. PRPC 6.3 was released, several browser versions ago. mobile was a cutting edge feature that wasn't fully baked into the product the way it is today. We did a lot of work to support all of that. And of course that work, done as HFixes, can introduce issues.
I think it's entirely reasonable to install an HFix and wait a week or two before committing it. That helps you be sure that there aren't edge cases that aren't addressed or other interactions.
Chunzhi,
It can certainly feel that way sometimes. Hopefully, if you're sick enough to need the medicine, it helps. And if it doesn't, that we are able to work with you until you are healthy again. Like anything, there is a cost-benefit/risk-reward analysis you need to do before taking an HFix. Particularly if there are a large number of dependencies. And if you can find a way not to need an HFix and the potential cost/risks associated, that's even better.
The challenge is that Pega is such large and flexible software with a very smart and creative user base, so when we come up with features, almost immediately someone starts using them in a way we never anticipated. Then we either need to adjust the software so their implementation can work or tell them not to do that and require them to go back and re-implement. In support, I can say we do both.
Thanks,
Mike
The value of committing a hotfix is that if you install a problematic hotfix later you can rollback that hotfix by itself. Once you've done testing and believe an installed hotfix is working fine, you should commit it, they should never remain in an uncommitted state.
As far as I know, they behave exactly the same whether committed or not and do include code changes, but again, the value of this feature is to easily pull out new hotfixes if they introduce problems or do not resolve the issue you thought they might. If you do not commit your hotfixes you cannot rollback a new set that were installed, you will rollback all uncommitted hotfixes.
If it's determined later that a hotfix is causing an issue in your environment, Global Customer Support can work with you to address it.
Pegasystems Inc.
AU
Thanks Nick.
Getting GCS and/or DBA's involved to take the operating platform backwards seems immeasurably harder than doing a "Rollback All" and re-installing the non-destructive hotfixes a second time. Obviously it depends on the number of hotfixes, but that sounds like half an hour's work. It would be done by the time the DBA returned my call.
I keep seeing the language "... should never remain in an uncommitted state". I've only seen it as a statement and it has never been supported with a justification. If I'm honest, it sounds like a cover story for an odd architectural decision to only allow rollback en masse. In other words:
"commit a bunch of stuff now so that you can roll back a future one individually"
... rather than ...
"roll back anything individually when you find a couple of weeks later it breaks something else and the DB backup you took 14 days ago isn't much use"
Can anyone genuinely justify the benefit of committing a hotfix?
Pegasystems Inc.
US
Braam,
Depending on the version, it is possible to have literally hundreds of hotfixes installed on a system. Some of them may have other special instructions. While that number is a small percentage, if you roll them all back and reinstall, you will need to be aware of those special instructions or that half an hour task can become significantly more complicated and ultimately may require a DBA and/or GCS to help you untangle the mess. So for a system with a handful of straightforward hot fixes that you still have the DL items to reinstall from, not committing them is probably not a big deal.
Committing HFixes that you have tested and are confident address your issue, means if/when you do need to roll back, you minimize risk by not needing to worry about them. While I agree it would be cool to pick and choose which specific hotifxes to roll back. The reality is that it's not that simple. Do you pull out the dependent HFixes as well? Which ones? Again, a single HFix on say 6.3sp1 can pull in a couple hundred dependencies. You may have had some installed already that you definitely don't want to see go away. We can argue about the architectural decision, and there is no doubt room for enhancement, but definitely from a support perspective, you're better off committing an hot fix once you are confident that it meets your needs, so that is unequivocally, what I would recommend.
Thanks,
Mike
Pegasystems Inc.
AU
Thanks Mike,
I'm surprised to hear of scenarios where there are 'hundreds of hotfixes'. I'd hope that's not the case on Pega 7 implementations when we've consciously gone to a model where the ML's were supposed to address that. We've been going at our current customer for 10 months and these are the first hotfixes we've needed to install.
I suppose this is less about hotfix commit and rollback, and more about being "confident that it meets your needs". I struggle to reach that level of confidence as I've been burnt before on hotfixes that solve the problem I needed to solve (which I can test for), but introduce other problems - sometimes seemingly unrelated - which I don't know to test for, and don't materialise until after I've made an earlier commit decision. The existence of the rollback capability seems to provide me with "an out", if used appropriately.
If we're taking hotfixes infrequently, and keeping them uncommitted for ... a week? ... perhaps if we haven't seen any adverse side-effects after that, then I could be more at peace with committing. It still highlights for me that we're not good at performing and/or communicating an impact assessment of hotfixes in order to reach a higher level of confidence sooner. Good to know GCS are there to help, but it's a material project impact to recover from a bad hotfix. Where I come from we don't have contingency in our schedules for it.
Pegasystems Inc.
JP
In my opinion, taking a HFix really like taking any drug. It might cure current pain but almost assured introducing new pain and you need more drugs later.
Accepted Solution
Pegasystems Inc.
US
Braam,
Yes, Pega 7 doesn't have anywhere near as large a dependency web as the PRPC 6 series (and older). The move to more frequent, fully tested releases allows us to avoid that. As does the focus on minimal impact when migrating to new releases. The idea is that you should be moving to the latest Pega 7 releases and not digging in for years on an older one. The infrastructure was not in place to do that in the 6 and earlier series. Additionally, we've been pushing back on making HFixes if there are viable alternatives (or the issue is already fixed in an existing future release). That said, for ANY release, an HFix is only ever unit tested, which means we test to confirm it fixes the issue reported. As you point out, the scenarios you never thought to test are not being looked for by the developers either. It's why I would generally discourage taking HFixes if you aren't experiencing the problem it addresses (unless it's a dependency, in which case you absolutely need it).
Some of the hundreds of hotfixes are due to one HFix breaking something, but many are just patches to keep up with the changing environment. PRPC 6.3 was released, several browser versions ago. mobile was a cutting edge feature that wasn't fully baked into the product the way it is today. We did a lot of work to support all of that. And of course that work, done as HFixes, can introduce issues.
Braam,
Yes, Pega 7 doesn't have anywhere near as large a dependency web as the PRPC 6 series (and older). The move to more frequent, fully tested releases allows us to avoid that. As does the focus on minimal impact when migrating to new releases. The idea is that you should be moving to the latest Pega 7 releases and not digging in for years on an older one. The infrastructure was not in place to do that in the 6 and earlier series. Additionally, we've been pushing back on making HFixes if there are viable alternatives (or the issue is already fixed in an existing future release). That said, for ANY release, an HFix is only ever unit tested, which means we test to confirm it fixes the issue reported. As you point out, the scenarios you never thought to test are not being looked for by the developers either. It's why I would generally discourage taking HFixes if you aren't experiencing the problem it addresses (unless it's a dependency, in which case you absolutely need it).
Some of the hundreds of hotfixes are due to one HFix breaking something, but many are just patches to keep up with the changing environment. PRPC 6.3 was released, several browser versions ago. mobile was a cutting edge feature that wasn't fully baked into the product the way it is today. We did a lot of work to support all of that. And of course that work, done as HFixes, can introduce issues.
I think it's entirely reasonable to install an HFix and wait a week or two before committing it. That helps you be sure that there aren't edge cases that aren't addressed or other interactions.
Chunzhi,
It can certainly feel that way sometimes. Hopefully, if you're sick enough to need the medicine, it helps. And if it doesn't, that we are able to work with you until you are healthy again. Like anything, there is a cost-benefit/risk-reward analysis you need to do before taking an HFix. Particularly if there are a large number of dependencies. And if you can find a way not to need an HFix and the potential cost/risks associated, that's even better.
The challenge is that Pega is such large and flexible software with a very smart and creative user base, so when we come up with features, almost immediately someone starts using them in a way we never anticipated. Then we either need to adjust the software so their implementation can work or tell them not to do that and require them to go back and re-implement. In support, I can say we do both.
Thanks,
Mike
Pegasystems Inc.
IN
Hi NIck/All, Good morning!
If (in case) a Hot fix [DL Item] is instructed to clear temp, cache and restart server post installation via Hot fix manager....
- how can we commit the installed uncommitted hfix
- as we cleared the catalog/manifest file from server...
in this scenario - though we commit the uncommitted Hfix after validation - it still shows up in the uncommitted list
Please share your thoughts/comments and correct me otherwise, Thank you!
psahukaru
Hi Phani,
As far as I know, a catalog or manifest file is not required for you to commit hotfixes. If you've tried to commit a hotfix and it still shows up in the uncommitted list, there is an issue that GCS probably needs to assist with. I've never dealt with this kind of issue myself, but I think I've heard of scenarios where it happens. As this isn't directly related to the main topic of this post, I would suggest that you open an SR for that to be investigated.
Adqura
IT
Hi All,
In all honesty, I have the feeling that this conversation doesn't address the main question, which can probably be rephrased as: "Can I leave my hotfixes uncommitted? If not, why?"
Thanks
Pegasystems Inc.
US
Read the reply from Nick Loving on February 18th. Having them committed/uncommitted only comes into play if you actually want to roll something back. If you install a new hot fix and then after testing with it want to roll it back then you will need to roll back all uncommitted hot fixes. Best practice should be to install a hot fix uncommitted, test with it for a while, and then either commit or rollback. Certainly before you install a new hot fix any previously installed hot fixes should be committed or rolled back.
Cognizant
US
Hi Mike,
We have a situation where a new hotfix need to be installed but there are few uncommitted hotfixes which need to be committed before installing the new one. Question we have in similar to the main question of this thread is, what is the exact difference between committed and uncommitted hotfixes from the technical view than the advantages/intentions ? Will PEGA store a backup of older version of rules etc which are related to hotfix and when we rollback it will restore respective rules. Similarly if we commit then the back up would be removed and hence we cannot do rollback ?
If backup concept as mentioned above is not true, can you please give more information on how exactly the commit/uncommit/rollback works in the back end?
Thanks,
Nishanth