How do I compare two models' performance in 7.4?
I'd like to take the same new data set and pass it to two versions of my text analysis model, trained on the same data, where the only difference is one model takes in an extra meta-data parameter (subject and body instead of just subject). Then compare the precision, recall and overall accuracy of both - similar to what I see if I were to just train & test a topic model.
Is there a tutorial how to do this today? I was taking the Decisioning Consultant courses but the Testing your simulation section didn't really cover this (plus doesn't cover 7.4, only 7.2).
Accepted Solution
Cisco Systems Inc
US
I just ended up making the data different outside of pega created 2 models and manually compared the results. Unsure if there's a simple option.
Pegasystems Inc.
NL
Maybe there is, if you're talking about Adaptive Models: while the standard reporting from the Analytics Center (aka Prediction Studio in future releases) focusses on a single prediction (single rule), there is a section in the Report Browser that has a couple of out-of-the-box reports that can give a view across predictions.
From the CDH Portal you would launch the Report Browser from the top-right menu
From other applications that include the Analytics Center portal you would go to Reports on the left-hand side
Maybe there is, if you're talking about Adaptive Models: while the standard reporting from the Analytics Center (aka Prediction Studio in future releases) focusses on a single prediction (single rule), there is a section in the Report Browser that has a couple of out-of-the-box reports that can give a view across predictions.
From the CDH Portal you would launch the Report Browser from the top-right menu
From other applications that include the Analytics Center portal you would go to Reports on the left-hand side
Other applications might also give access to the standard Report Browser in other ways.
From there, choose the Adaptive Decisioning category
Then there's various reports to choose from. Their first screens often look similar, but there is difference in their drill-down capabilities.
For example, the "Adaptive Models Overview" report shows all Adaptive Models in the system. You can use the standard filtering options to filter.
Hope this helps
Otto
Pegasystems Inc.
NL
For text models the "Model Report" might be useful. It's available from the Analytics Center also
This generates a PDF with info about all the models currently selected in the Analytics Center, containing the details you may be looking for
Hope this helps more...