Lucene index file size, what's the maximum?
Hi,
We currently are having a growing lucene index file size on our PEGA 7.1.8 system and are wondering what the maximum size can be.
What is still performing?
10GB, 20GB, 100GB?
And, on top of that, how much time would a complete re-index take after clearing the complete directory?
Does anyone know?
Kind Regards,
Arjan Maus
Message was edited by: Lochan to add Category
Pegasystems
IN
If you are using 7.1.8, it should be an Elastic Search index (which uses Lucene underneath).
The size of the index grows the number of cases, rules or data instances grows in your database.
Could you please let us know what the current index size is?
Also, the time taken to index is dependent on the number of cases, rules and data and the size of each instance. You can roughly approximate the size of an instance by doing a length on the BLOB column.
Note that Elastic Search indices (Lucene indices) grow and shrink as indexing updates the index with entries in it. This results in the index actually consuming up to 3 times the size of the index if it was freshly built.
Pegasystems
IN
Your initial post mentioned Pega 7.1.8 and thus my reply.
The reply post above by me is generally true even for Lucene indices.
Are you seeing any concerns with the growing index size? For some customers we have seen much higher index sizes in Pega 6.
Achmea
NL
Rajiv,
Excuse me, I was a little bit confused apparently.
We had some issues within Pega 6.2, but my Question was indeed meant for PEGA 7.1.8.
We are in the process of combining a PEGA 6.1 & PEGA 6.2 environment into an updated PEGA 7.1.8.
Thus, the index file is growing and, as we also deploy new applications, I am a bit concerned about the size of the index files.
Especially if we need to rebuild the complete indexes again from scratch.
It recently took us, on the 6.2 environment over 12 hours and ended up with an index of 6GB.
So yes, I see concerns with a growing index file, but only if, for whatever reason, we need to rebuild if from scratch.
Pegasystems
IN
We are in the process of combining a PEGA 6.1 & PEGA 6.2 environment into an updated PEGA 7.1.8.
Does the mean that all applications will be hosted on the same Pega cluster (a Pega cluster is all application server nodes pointing to the same PegaRULES DB schema)?
If that is not the case, then each cluster should have its own set of index nodes and that should help.
So yes, I see concerns with a growing index file, but only if, for whatever reason, we need to rebuild if from scratch.
I can see the concern here. A general understanding of your growth rate would help understand this better. Also, if you have any archive & purge policy, then that would be useful to know.
Pegasystems
IN
That is big. Note that with 7.1.8, your index sizes should shrink because of some changes we introduced from 6.3 onwards and also because of the upgraded version of the Lucene library in the product.