Question
Tata Consultancy Services
IN
Last activity: 16 Oct 2018 12:03 EDT
Not able to process the posts from twitter using Dataflow which contains Twitter Dataset and text analyzer. Facing the issue that Data flow getting failed with status “Completed with failures” on Decisioning: Data Flows landing page.
I have created Data flow that is calling twitter data set and next text analyzer.
But I am facing below issues:
- Twitter data set with Stream option selected and If I run the data flow from the action button of data flow rule, it is creating the test model on “Decisioning: Data Flows” landing page and processing the comments posted on twitter but after some time this test model is getting deleted.
- Twitter data set with Stream/Playback and If I created data flow model from “Decisioning: Data Flows” landing page or from Activity using Dataflow-execute Method, it is creating model but not processing any records from twitter. And Data flow model getting failed with status “Completed with failures” on Decisioning: Data Flows landing page after some time.
- Twitter data set with Playback Option and If I run the data flow from the action button of data flow rule, getting failed with message “since it exceeds the max number of failed record, which is currently set to 0”
Note I have changed max number of failed record count to 1000 using dynamic system setting, but still getting the same issue.
Please suggest me if anyone knows that what could be the reasoned that I am not able to process the records from twitter and reason for failing the model with status as “Completed with Failures”.
Please find the attached screen shots for the issue.
**Moderation Team has archived post**
I have created Data flow that is calling twitter data set and next text analyzer.
But I am facing below issues:
- Twitter data set with Stream option selected and If I run the data flow from the action button of data flow rule, it is creating the test model on “Decisioning: Data Flows” landing page and processing the comments posted on twitter but after some time this test model is getting deleted.
- Twitter data set with Stream/Playback and If I created data flow model from “Decisioning: Data Flows” landing page or from Activity using Dataflow-execute Method, it is creating model but not processing any records from twitter. And Data flow model getting failed with status “Completed with failures” on Decisioning: Data Flows landing page after some time.
- Twitter data set with Playback Option and If I run the data flow from the action button of data flow rule, getting failed with message “since it exceeds the max number of failed record, which is currently set to 0”
Note I have changed max number of failed record count to 1000 using dynamic system setting, but still getting the same issue.
Please suggest me if anyone knows that what could be the reasoned that I am not able to process the records from twitter and reason for failing the model with status as “Completed with Failures”.
Please find the attached screen shots for the issue.
**Moderation Team has archived post**
This post has been archived for educational purposes. Contents and links will no longer be updated. If you have the same/similar question, please write a new post.
Pegasystems Inc.
IN
Hi Srinivas,
Please provide the stack trace of the error. You can get it by following below steps:
- Upon clicking the error, DF Error report opens up in a new window.
Hi Srinivas,
Please provide the stack trace of the error. You can get it by following below steps:
- Upon clicking the error, DF Error report opens up in a new window.
- Click the “View Details” for the failed record. If the trace is not readable, right click and select “View Page Source.
Tata Consultancy Services
IN
Hi Pooja,
Thanks for the response.
From track trace i noticed that record is failing with error code HTTP/1.1 420 .
Reason for Http/1.1 420 error.
|
420 |
Rate Limited |
The client has connected too frequently. For example, an endpoint returns this status if:
|
How can we resolve this issue,is there any option from pega or twitter . Please let me know ASAP.
Below are the dataset options selected when i am getting the issue
1) twitter Dataset-on Stream option selection.--data flow model processing the records from twitter sucessfully. But it is failing after around 8 mins 8 secs .
2) twitter Dataset-on Playback option selection.--data flow model is getting failed within no time when we create the model. given 1 day for search tweets field.
Stack Trace Details Below:
Hi Pooja,
Thanks for the response.
From track trace i noticed that record is failing with error code HTTP/1.1 420 .
Reason for Http/1.1 420 error.
|
420 |
Rate Limited |
The client has connected too frequently. For example, an endpoint returns this status if:
|
How can we resolve this issue,is there any option from pega or twitter . Please let me know ASAP.
Below are the dataset options selected when i am getting the issue
1) twitter Dataset-on Stream option selection.--data flow model processing the records from twitter sucessfully. But it is failing after around 8 mins 8 secs .
2) twitter Dataset-on Playback option selection.--data flow model is getting failed within no time when we create the model. given 1 day for search tweets field.
Stack Trace Details Below:
com.pega.dsm.dnode.api.dataflow.StageException: Exception in stage: MyDataSetForTwitter at com.pega.dsm.dnode.api.dataflow.StageException.create(StageException.java:34) at com.pega.dsm.dnode.api.dataflow.DataFlowStage$StageOutputSubscriber.onError(DataFlowStage.java:482) at com.pega.dsm.dnode.api.dataflow.DataFlowStage$StageInputSubscriber.onError(DataFlowStage.java:351) at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.onError(DataObservableImpl.java:302) at com.pega.nlp.dataset.impl.ConnectorBrowseAllRecordsOperation$1.emit(ConnectorBrowseAllRecordsOperation.java:120) at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:335) at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:53) at com.pega.dsm.dnode.api.dataflow.DataFlow$2.run(DataFlow.java:376) at com.pega.dsm.dnode.api.dataflow.DataFlow$2.run(DataFlow.java:370) at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:52) at com.pega.dsm.dnode.impl.dataflow.DataFlowThreadContext$1.run(DataFlowThreadContext.java:151) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:44) at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:41) at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:52) at com.pega.dsm.dnode.impl.prpc.PrpcThreadFactory$PrpcThread.run(PrpcThreadFactory.java:109) Caused by: java.lang.Throwable: Social data flow recieved error for dataset id: Data-Social-Twitter_MyDataSetForTwitter for dataset type: Twitter with error type: HTTP_ERROR on social object: Streaming error due to reason: HTTP/1.1 420 Enhance Your Calm ... 17 more
Tata Consultancy Services
IN
In previous post stack trace details are from below senario
1) twitter Dataset-on Stream option selection.--data flow model processing the records from twitter sucessfully. But it is failing after around 8 mins 8 secs .
Stack trace details from another senario:
2) twitter Dataset-on Playback option selection.--data flow model is getting failed within no time when we create the model. given 1 day for search tweets field.
In previous post stack trace details are from below senario
1) twitter Dataset-on Stream option selection.--data flow model processing the records from twitter sucessfully. But it is failing after around 8 mins 8 secs .
Stack trace details from another senario:
2) twitter Dataset-on Playback option selection.--data flow model is getting failed within no time when we create the model. given 1 day for search tweets field.
com.pega.dsm.dnode.api.dataflow.StageException: Exception in stage: MyDataSetForTwitter at com.pega.dsm.dnode.api.dataflow.StageException.create(StageException.java:34) at com.pega.dsm.dnode.api.dataflow.DataFlowStage$StageOutputSubscriber.onError(DataFlowStage.java:482) at com.pega.dsm.dnode.api.dataflow.DataFlowStage$StageInputSubscriber.onError(DataFlowStage.java:351) at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.onError(DataObservableImpl.java:302) at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:341) at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:53) at com.pega.dsm.dnode.api.dataflow.DataFlow$2.run(DataFlow.java:376) at com.pega.dsm.dnode.api.dataflow.DataFlow$2.run(DataFlow.java:370) at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:52) at com.pega.dsm.dnode.impl.dataflow.DataFlowThreadContext$1.run(DataFlowThreadContext.java:151) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:44) at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:41) at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:52) at com.pega.dsm.dnode.impl.prpc.PrpcThreadFactory$PrpcThread.run(PrpcThreadFactory.java:109) Caused by: java.lang.UnsupportedOperationException at java.util.AbstractList.add(AbstractList.java:148) at java.util.AbstractList.add(AbstractList.java:108) at com.pega.nlp.connector.twitter.client.TwitterSearchService.clearForNextRun(TwitterSearchService.java:176) at com.pega.nlp.connector.poll.AbstractPollingConnecter.stop(AbstractPollingConnecter.java:385) at com.pega.nlp.connector.twitter.client.TwitterConnecterWrapper.stop(TwitterConnecterWrapper.java:257) at com.pega.nlp.connector.ConnectorWrapper.stop(ConnectorWrapper.java:181) at com.pega.nlp.dataset.impl.ConnectorBrowseAllRecordsOperation$1.emit(ConnectorBrowseAllRecordsOperation.java:135) at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:335) ... 15 more
Pegasystems Inc.
IN
Hi Srinivas,
Please send me the complete stack trace. By performing right click and view page source you can get the complete stack trace.
Tata Consultancy Services
IN
Hi,
I have gone through the view source option but still getting the same stack trace details. Please find the attachment for the same.
Pegasystems Inc.
IN
The issue seems to because of the associated data flow node. Please go to Designer Studio->Decisioning-> Infrastructure->Services-> Data Flow. In it check if a node with Status 'Normal' is present. Also press the 'Edit Settings' and check the no. of threads.
Infosys
GB
Hi , I am getting same error. I have number of threads set to 5 (default) and after 5 retry it gives error in Data Flow run screen.
Hi , I am getting same error. I have number of threads set to 5 (default) and after 5 retry it gives error in Data Flow run screen.
2018-03-27 10:39:03,746 [mple:Administrators]] [ STANDARD] [
] [ ] (ient.HBCTwitterStreamingClient) ERROR - Social data
flow recieved error for dataset id: Data-Social-Twitter_TwitterDS for dataset t
ype: Twitter with error type: HTTP_ERROR on social object: Streaming error due t
o reason: HTTP/1.1 420 Enhance Your Calm Attempt no - 1 to restart...
2018-03-27 10:39:39,560 [mple:Administrators]] [ STANDARD] [
] [ ] (ient.HBCTwitterStreamingClient) ERROR - Social data
flow recieved error for dataset id: Data-Social-Twitter_TwitterDS for dataset t
ype: Twitter with error type: HTTP_ERROR on social object: Streaming error due t
o reason: HTTP/1.1 420 Enhance Your Calm Attempt no - 2 to restart...
2018-03-27 10:40:45,367 [mple:Administrators]] [ STANDARD] [
] [ ] (ient.HBCTwitterStreamingClient) ERROR - Social data
flow recieved error for dataset id: Data-Social-Twitter_TwitterDS for dataset t
ype: Twitter with error type: HTTP_ERROR on social object: Streaming error due t
o reason: HTTP/1.1 420 Enhance Your Calm Attempt no - 3 to restart...
2018-03-27 10:42:21,131 [mple:Administrators]] [ STANDARD] [
] [ ] (ient.HBCTwitterStreamingClient) ERROR - Social data
flow recieved error for dataset id: Data-Social-Twitter_TwitterDS for dataset t
ype: Twitter with error type: HTTP_ERROR on social object: Streaming error due t
o reason: HTTP/1.1 420 Enhance Your Calm Attempt no - 4 to restart...
2018-03-27 10:44:26,997 [mple:Administrators]] [ STANDARD] [
] [ ] (ient.HBCTwitterStreamingClient) ERROR - Social data
flow recieved error for dataset id: Data-Social-Twitter_TwitterDS for dataset t
ype: Twitter with error type: HTTP_ERROR on social object: Streaming error due t
o reason: HTTP/1.1 420 Enhance Your Calm Attempt no - 5 to restart...
2018-03-27 10:47:02,827 [mple:Administrators]] [ STANDARD] [
] [ ] (ient.HBCTwitterStreamingClient) ERROR - Social data
flow recieved error for dataset id: Data-Social-Twitter_TwitterDS for dataset t
ype: Twitter with error type: HTTP_ERROR on social object: Streaming error due t
o reason: HTTP/1.1 420 Enhance Your Calm
Evonsys Inc
US
Any solution to this problem? i am receiving the same error in the log..
HTTP/1.1 420 Enhance Your Calm Attempt no - 1 to restart..
Pegasystems Inc.
IN
Hi,
Thank you for posting your query in the PSC. This looks like an inactive post and hence, we suggest you create a new post for your query. Click on the Write a Post button that’s available on the top right pane of this page. Once created, please reply back here with the URL of the new post.
You may also refer this discussion link as a reference in the new thread.