Question
Scotiabank
CA
Last activity: 4 Oct 2018 13:54 EDT
extract to Hadoop
We need to send data to Hadoop cluster so that data can be used for analytical reporting at any time. Should we be using BIX or can we use the Hadoop connectors?
**Moderation Team has archived post**
This post has been archived for educational purposes. Contents and links will no longer be updated. If you have the same/similar question, please write a new post.
Accepted Solution
Pegasystems Inc.
NL
Hi,
Since 7.1.8 you can move data from any DataSet to HDFS or HBase through Dataflow.
there are 2 datasets that can provide fast data move
in addition to pdn tutorial, you can see eng docs here
Pegasystems Inc.
US
most likely Hadoop connectors, do not think BIX has that capability. See this link for our current bigdata features: https://docs-previous.pega.com/decision-management-reference-materials/introduction-big-data-capabilities-pega-7-platform
Pegasystems
IN
I think it depends on what you want to extract and how you want to extract. Connectors are meant to call external services which could involve Hadoop as well. BIX is meant to do the extraction part of your ETL process. If the volumes are large, BIX is the right choice. That said, there is no direct support for Hadoop in BIX. You can extract to XML, CSV or DB (RDBMS) and then load it up into Hadoop.
Accepted Solution
Pegasystems Inc.
NL
Hi,
Since 7.1.8 you can move data from any DataSet to HDFS or HBase through Dataflow.
there are 2 datasets that can provide fast data move
in addition to pdn tutorial, you can see eng docs here
Pegasystems Inc.
NL
Hi all again,
i forgot to check the mesh spaces access rights.
here you can see pdn reference docs for 2 dataset and Hadoop System record
https://community.pega.com/sites/default/files/help_v719/procomhelpmain.htm
https://community.pega.com/sites/default/files/help_v719/procomhelpmain.htm
And indeed please follow the instructions from PDN tutorial https://pdn.pega.com/tutorial-defining-hadoop-records/tutorial-defining-hadoop-records
This doc is explaining all steps you will need to do
- defining hadoop record
- uploading thirdparty-jars
- defining DataSets for HDFS
- reading and writing with by datasets
Pegasystems Inc.
IN
Hi,
Thank you for posting your query in the PSC. This looks like an inactive post and hence, we suggest you create a new post for your query. Click on the Write a Post button that’s available on the top right pane of this page. Once created, please reply back here with the URL of the new post.
You may also refer this discussion link as a reference in the new thread.