Question
Siemens
IN
Last activity: 3 Dec 2020 23:45 EST
Topic Detection Model in NLP not training via Feedback - FScore decreasing
We are using NLP Topic detection (Max Entropy Model) for categorization of emails. We have also included feedback loop to correct the right category in case the category detected was wrong, We are using pxCaptureTAFeedback for feedback and pxUpdateModels to update model. We are removing the salutations and signature to eliminate the noise. Also we are just passing the key sentence which defines the correct category instead of sending whole mail body. Even after taking utmost care to eliminate the noise for training the model, still the F-Score is decreasing instead of increasing. It went from 0.45 to 0.01 very fast. Now the category itself is not getting detected.
-What could be the root case of this issue?
-Are there any additional steps which i can follow to improve the FScore.
- What is the process if i want to import the model from higher env to lower env, delete the unwanted feedback from model in lower env, and then upload this cleaned up model back to higer env?
- Will the above step improve the fscore?
Bits In Glass
GB
Hi Virendra,
This is expected.
In General, When the Models are initially trained and has unique subset of Training Data with the Test Data, the Fscore would be 1.
When you try to train the same model again, as the data still remained the same, the Fscore is reduced. So you should let the model save the resultants instead of manually training the model.
If you want to get the F-score back again, You can delete the feedback and re-train the model.
Siemens
IN
Hello Sesha,
Without training the model, how will the system detect category for a new sentences or email body over the years? I thought the basic purpose of the model is to learn/train by feedback from the correction and reach to a level where most of the possible combinations are trained and will eventually reach 80-90% accuracy. Which is also mentioned in the article - https://community.pega.com/knowledgebase/articles/decision-management-overview/providing-feedback-text-analytics-models#extraction
Otherwise we could also use basic decisioning rules to "hardcode" the mapping of keywords -> category.
Basic issue is that as we are training the model with key sentences without any noise we are not getting the desired result.
Pegasystems Inc.
IN
Hi Virendra,
What could be the root case of this issue?
Santosh: There could be multiple reasons for this to happen. The best way to find out your issue is to download the current training data with which the model is being trained and analyze it. The model accuracy is very much proportional to the quality of the data. Few things you can focus while you analyze the training data :
1. See if any junk is creeping in (HTML, unwanted tags etc)
2. Find out if there are any overlap with the categories in the training data
3. What is the number of training records at each topic level ( they should be balanced)
-Are there any additional steps which i can follow to improve the FScore.
Santosh: Once you have your findings from your above analysis we should be able to suggest something for your usecase.
- What is the process if i want to import the model from higher env to lower env, delete the unwanted feedback from model in lower env, and then upload this cleaned up model back to higer env?
Hi Virendra,
What could be the root case of this issue?
Santosh: There could be multiple reasons for this to happen. The best way to find out your issue is to download the current training data with which the model is being trained and analyze it. The model accuracy is very much proportional to the quality of the data. Few things you can focus while you analyze the training data :
1. See if any junk is creeping in (HTML, unwanted tags etc)
2. Find out if there are any overlap with the categories in the training data
3. What is the number of training records at each topic level ( they should be balanced)
-Are there any additional steps which i can follow to improve the FScore.
Santosh: Once you have your findings from your above analysis we should be able to suggest something for your usecase.
- What is the process if i want to import the model from higher env to lower env, delete the unwanted feedback from model in lower env, and then upload this cleaned up model back to higer env?
Santosh: If your model is not performing well, we recommend you review the traininig data, curate it, rebuild the model and then promote it to higher env once you are happy with it. Although the feature to automatically update the model is available, we recommend your involvement in model building and reviewing as when the training data is automatically captured, there is high possibility of junk creeping in or there could be a huge imbalance in the training set with one topic vs others.
- Will the above step improve the fscore?
Santosh: Yes, once you complete the above exercise your model should perform better.