Question
Pegasystems Inc.
JP
Last activity: 25 Dec 2019 2:40 EST
Stream Service prior and post 8.1
Hi,
We are using Pega Platform 8.3.1 and have faced an issue with Search Index, and just figured that it is because "Stream Service" was down for some reason (we don't know why this happened). Per https://community.pega.com/support/support-articles/search-index-not-working-new-items-after-upgrade-pega-81, we have executed Stream Service and Search started to work (see below screenshot). It seems that this Stream Service must be running ("NORMAL") for Search to work.
Hi,
We are using Pega Platform 8.3.1 and have faced an issue with Search Index, and just figured that it is because "Stream Service" was down for some reason (we don't know why this happened). Per https://community.pega.com/support/support-articles/search-index-not-working-new-items-after-upgrade-pega-81, we have executed Stream Service and Search started to work (see below screenshot). It seems that this Stream Service must be running ("NORMAL") for Search to work.
Now, we also have another Pega Platform environment which is version 7.4. When I looked at the same configuration in Stream Service, a node was not even existent (I never did it because I thought this menu is for Decisioning - we are only using Pega Platform). However, Search seems to be working fine.
Question:
"Stream Service has to be running in order for Search to work from version 8.1, but in prior version, it is not required." Is this statement correct? If yes, why is that so?
Accepted Solution
Pegasystems Inc.
IN
Hi Tsuck,
Appreciate your observation, It is completely right. Let me give you a brief overview.
From pega 8.1 onwards, Job scheduler & QueueProcessor have been introduced. Pega platform 8.1 includes three default queue processor rules.
- pyFTSIncrementalIndexer
- pyProcessNotification
- pzStandardProcessor
pyFTSIncrementalIndexer, which replaces agent FTSIncrementalIndexer and performs incremental (search) indexing in the background. This Queue Processor rule posts Rule, Data and Work objects into the Search subsystem, as soon as we create or change them. This helps to keep search data current and reflects the content of the database closely. Therefore you couldn't search in 8.x when the stream node is down while it works in 7.4 without stream node.
If you need more details then please watch the video mentioned in the below link. It covers all the key concepts & demonstrates search with & without stream node in 8.x version.
I hope you got the answer. If you feel so, please mark this post as correct/helpful.
Thanks,
Abhinav
Pegasystems Inc.
IN
Hi,
From Pega 8.1 and later releases, Search runs through the Queue Processor (that is, Kafka and Real-time Decisioning Services). Therefore, you need to enable the Stream service on the Services landing page.
Thanks,
Kranthi
Pegasystems Inc.
JP
Hi Kranthi,
Thanks for your response.
#1. This Stream Service menu is in "Decisioning", and I see it as very confusing because people who are using only Pega Platform would not see this menu carefully. Search is a very basic Pega Platform functionality and I would suggest that to be outside of Decisioning menu. Would you agree?
#2. Could you also give an explanation about what technology Search runs on in prior version (as opposed to Kafka and Real-time Decisioning Services)?
Thanks,
Pegasystems Inc.
US
The Stream service was originally used by Decisioning and not by the base platform, so moving it out from that location would confuse people who were already using it regularly, so I think either way some people would have ended up being confused by its location. Changing it now may just cause more confusion.
Prior to using Kafka, search was utilizing agents and the database. Queue Processors utilizing Kafka are able to perform their processing more quickly.
Pegasystems Inc.
JP
Thanks Nick for your comment, it is very helpful. I understand this way - Search functionality used to utilize agents and database in 7x, and it now utilizes Queue Processor (that is, Kafka and Real-time Decisioning Services) in 8x.
However, this article (https://community.pega.com/knowledgebase/articles/decision-management-overview/stream-service-overview) says Stream Service using Kafka was introduced in 7.4. Isn't this contradictory? As I mentioned in my original post, Search is working fine without any node added in the Stream Service with 7.4.
Thanks,
Pegasystems Inc.
JP
Hi,
Here is my guess to my own question - In 7.4, Stream Service was introduced but Search was not utilizing it at that point (still utilizing Agent). So in 7.4 even if you do not run Stream Service, Search still works fine. Beginning with 8x, Search started to use Stream Service.
If my hypothesis above is correct, it explains what I am observing. Can someone let me know if this is fair or not?
Thanks,
Accepted Solution
Pegasystems Inc.
IN
Hi Tsuck,
Appreciate your observation, It is completely right. Let me give you a brief overview.
From pega 8.1 onwards, Job scheduler & QueueProcessor have been introduced. Pega platform 8.1 includes three default queue processor rules.
- pyFTSIncrementalIndexer
- pyProcessNotification
- pzStandardProcessor
pyFTSIncrementalIndexer, which replaces agent FTSIncrementalIndexer and performs incremental (search) indexing in the background. This Queue Processor rule posts Rule, Data and Work objects into the Search subsystem, as soon as we create or change them. This helps to keep search data current and reflects the content of the database closely. Therefore you couldn't search in 8.x when the stream node is down while it works in 7.4 without stream node.
If you need more details then please watch the video mentioned in the below link. It covers all the key concepts & demonstrates search with & without stream node in 8.x version.
I hope you got the answer. If you feel so, please mark this post as correct/helpful.
Thanks,
Abhinav