Here at Pega, we had a similar problem with displaying decision rules on the standard developer portal decision rule form, where if there were more than 500 lines in the decision rule, the grid was too slow.
The solution was to not even attempt to show the lines in a grid if there were more than 500 lines, but to instead let the user press the "export to excel" button on the rule form, which displays them more efficiently. The user can also modify the rows in excel and then when they exit excel, the decision rule gets updated with the modified rows.
You could design yours similarly, and even do a Pega trace of the decision rule form excel button processing to see how we do it.
By the way, we also include a dynamic system setting to change the threshold from 500 to another value, since in some environments the grid may behave better or worse depending on the hardware and memory. /Eric