Hello dears,

We've added new column to PR_DATA_IH_FACT table and property to Data-Decision-IH-Fact class and other respective changes (saved Data-Decision-IH-Fact Database Table instance in development environment) have been completed. DB add column operation have been completed on upper environmets first and product rule includes property and related data transform rules populating the FACTEXTENSIONID property value and database table instance has been deployed on upper environments.

We have obtained logs below one of the upper environment. Unusual thing here is that we have not faced with the same issue on another upper environment after the same package has been imported. 

Caused by: com.pega.pegarules.pub.database.BadTableMappingException: Cannot save instance of class Data-Decision-IH-Fact; it belongs to fully exposed table PEGA_DATA_PROD_SAT.PR_DATA_IH_FACT in database PegaDATA, but the instance has properties which do not correspond to columns: FACTEXTENSIONID 
	at com.pega.pegarules.data.internal.access.PageDatabaseMapperImpl.fillInDataStoreSaveStatementWithoutStream(PageDatabaseMapperImpl.java:1704) ~[prprivate-data.jar:?] 
	at com.pega.pegarules.data.internal.access.PageDatabaseMapperImpl.fillInSaveStatementWithoutStream(PageDatabaseMapperImpl.java:1518) ~[prprivate-data.jar:?] 
	at com.pega.pegarules.data.internal.access.BatchProcessor.fillInStmt(BatchProcessor.java:311) ~[prprivate-data.jar:?] 
	at com.pega.pegarules.data.internal.access.BatchProcessor.save(BatchProcessor.java:159) ~[prprivate-data.jar:?] 
	at com.pega.pegarules.data.internal.access.DatabaseImpl.saveDeferred(DatabaseImpl.java:2823) ~[prprivate-data.jar:?] 
	at com.pega.pegarules.data.internal.access.DatabaseImpl.saveDeferred(DatabaseImpl.java:2814) ~[prprivate-data.jar:?] 
	at com.pega.pegarules.data.internal.access.DatabaseImpl.saveDeferred(DatabaseImpl.java:2807) ~[prprivate-data.jar:?] 
	at com.pegarules.generated.pzStoreFactRecords_080101__6610569284587962636.pzStoreFactRecords08_01_01(pzStoreFactRecords_080101__6610569284587962636.java:135) ~[?:?] 
	at com.pegarules.generated.pzStoreFactRecords_080101__6610569284587962636.invoke(pzStoreFactRecords_080101__6610569284587962636.java:77) ~[?:?] 
	at com.pega.pegarules.generation.internal.library.LibraryRuntime.resolveAndinvokeFunctionViaReflection(LibraryRuntime.java:238) ~[prprivate-generation.jar:?] 
	at com.pega.pegarules.generation.internal.library.LibraryRuntime.invokeLibraryRuntime(LibraryRuntime.java:125) ~[prprivate-generation.jar:?] 
	at com.pega.pegarules.session.internal.mgmt.Executable.invokeLibraryRuntime(Executable.java:9829) ~[prprivate-session.jar:?] 
	at com.pega.pegarules.priv.generator.LibrarySupport.resolveAndInvokeFunctionViaReflection(LibrarySupport.java:260) ~[prpublic.jar:?] 
	at com.pegarules.generated.pega_decisionengine_interactionhistory.pzStoreFactRecords(pega_decisionengine_interactionhistory.java:613) ~[?:?] 
	at com.pegarules.generated.pzStoreInInteractionHistory_080201__3072507141625005151.pzStoreInInteractionHistory08_02_01(pzStoreInInteractionHistory_080201__3072507141625005151.java:208) ~[?:?] 
	at com.pegarules.generated.pzStoreInInteractionHistory_080201__3072507141625005151.invoke(pzStoreInInteractionHistory_080201__3072507141625005151.java:77) ~[?:?] 
	at com.pega.pegarules.generation.internal.library.LibraryRuntime.resolveAndinvokeFunctionViaReflection(LibraryRuntime.java:238) ~[prprivate-generation.jar:?] 
	at com.pega.pegarules.generation.internal.library.LibraryRuntime.invokeLibraryRuntime(LibraryRuntime.java:125) ~[prprivate-generation.jar:?] 
	at com.pega.pegarules.session.internal.mgmt.Executable.invokeLibraryRuntime(Executable.java:9829) ~[prprivate-session.jar:?] 
	at com.pega.pegarules.priv.generator.LibrarySupport.resolveAndInvokeFunctionViaReflectionWithException(LibrarySupport.java:277) ~[prpublic.jar:?] 
	at com.pegarules.generated.dataset.ra_action_pxinteractionhistory_624f284cfd1b2c283c9d1ee687491a65$2$1.handleAndResetBatch(ra_action_pxinteractionhistory_624f284cfd1b2c283c9d1ee687491a65.java:306) ~[?:?] 
	at com.pegarules.generated.dataset.ra_action_pxinteractionhistory_624f284cfd1b2c283c9d1ee687491a65$2$1.storeInIH(ra_action_pxinteractionhistory_624f284cfd1b2c283c9d1ee687491a65.java:288) ~[?:?] 
	at com.pegarules.generated.dataset.ra_action_pxinteractionhistory_624f284cfd1b2c283c9d1ee687491a65$2$1.emit(ra_action_pxinteractionhistory_624f284cfd1b2c283c9d1ee687491a65.java:235) ~[?:?] 
	at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.stream.DataObservableImpl$3.emit(DataObservableImpl.java:176) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.dataset.ih.stream.IHSavingInstance.saveToIH(IHSavingInstance.java:139) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.dataset.ih.stream.IHSavingInstance.saveFilteredRecords(IHSavingInstance.java:110) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.dataset.ih.stream.IHSavingInstance.save(IHSavingInstance.java:63) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.dataset.ih.stream.IHCombinedSaveOperation.lambda$save$0(IHCombinedSaveOperation.java:52) ~[d-node.jar:?] 

What did we try to solve this issue?

1. rule/clear api has been executed for all the respective nodes in the cluster.

2. Revalidate and save operation has been executed for the type of Rule-Obj-Property in respective ruleset version includes FACTEXTENSIONID property rule.

3. One of Pega cache cleaner Agent's activity has been manually executed.

4. All nodes in the cluster has been restarted along with deleting the cache files in pega_temp folder.

Unfortunately, after taking all the actions above, same issue was still there.

What else can we do? What could be causing this issue?

Regards.

Mert.