What are the considerations for loading Data Pages containing large data sets?

Downstream systems that risk making large data sets available as responses to REST APIs (or similar) would ideally provide parameters that can limit this impact. For example, parameters allowing from/to/limit so as to get a more narrow set of results. 

If your application need is a very small subset of the data returned from another System of Record, your Response Data Transform can take responsibility for ignoring irrelevant records and mapping a limited number of fields. However the Data Transform can't make a start on transforming the response until the full response is received, which is the service provider's responsibility.

For any Connector expecting JSON as its response data, utilize the JSON Data Transform as a component of your Response Data Transform to map data directly from the raw JSON response to your Data Page's object class. This is now part of how App Studio creates a REST Data Source for a Data Type, and is significantly more performant than the incumbent approach used by Dev Studio's REST Connector wizard (even as late as Pega 8.4) where the JSON response is first mapped in its entirety to the clipboard before the Response Data Transform is invoked.

Discussion on this topic was sought from the LSA Data Excellence (Pega 8.4) webinar conducted in July 2020. The webinar and its full set of discussions that arose from it are available at LSA Data Excellence: Webinar, Questions & Answers.