Calling REST service in every 1 minute as unable to use KAFKA
Hi,
We have a use case to keep the Pega spine (Pega Cloud) updated frequently but we can't use Kafka or similar streaming mechanism because of the security restriction. So, we are planning to expose a REST service and receive the data from the upstream system in every 1 minute.
The payload in the service invocation per minute can go up to 200 KB. In a day, the total request will be closed to 3000 (max).
Please let me know if you see there is any performance issue in the application in getting the rest service and loading data in every minute? If you have experienced similar situation please let us know.
Appreciate your help in advance.
JPMorgan Chase
IN
Is the data going to change frequently?
How about if the system you are going to interact and integrate with is down? does it hamper your workflow?
200 KB is it going to shoot up to bigger sizes in future, because this will have impact on network data transfer speed.
API performance is questionable when there is network latency in the system.
Number of rest connections will have cost impact from your cloud vendor side as well. Network is chargeable if utilization is beyond allowed limits.
Work out your solution in all these aspects.
Capgemini
US
Thanks @SriHarsha Anika for your reply.
Data will not vary. If the integrated system is down and unable to send data to pega application then decision will be impacted. The other points you mentioned are on the table and being discussed.
JPMorgan Chase
IN
@NeelDas If the data is not varying then making a call every 1 min is not a good choice, load the data in a node level data page to have high performance. load it once, refresh it daily once.
Capgemini
US
@SriHarsha Anika The structure of data won't change but it will be a different record in every minute.
The streaming data will be passed to Pega in minutes; as KAFKA can't be used so finding the alternatives. There are a few other options but leading thought is - with REST service. The challenge in the REST is the performance when the number can go up to 2 M per day and 45-50 TPS can be hit to the servers.
Neel (Pega LSA)