Hi Marc,

this is could be a hypothetical situation (was a design approach question asked by a colleague)

50 million reside in the external system and we want to show it in pega , if we want to show it on UI is pagination feature configurable here .. as per my understanding so far .. though pagination shows data in multiple pages to user , in reality loads all the records in the clipboard .. which may not be a good idea.

and your last question i am not sure if it makes sense , so my design approaches are processing at the back end 

OPTION #1 Run a std agent /queue scheduler in the background and create a report using separate reporting database and using separate Node

                 and use either REST call or mapping external DB into pega classes and

                 use obj-browse in each call get the total count of records currently available in the table and count of                         current received records , ordered by row id

                 and in subsequent calls pass the query with rowid '>' last received rowid, while sum the count of records received so far in and minus from total records available in the external system ( we cannot put any flag of processed records in external system ) until there are no more records to be processed 

OPTION #2 Run BIX extraction tool (but do we need to first load data into pega Database , so that we can use BIX?)

OPTION #3 use external BI tools

would be happy to be corrected in any of the approaches and open for new thoughts.

-Thanks

Govardhan