Question
ATOS
IN
Last activity: 17 Jul 2018 11:31 EDT
Devops Pipeline not moving
I am all set and my pipeline ,but it is not moving ahead in the next stage. I have created 2 JFrog repository which I have configured in the pipeline. Now after everything is set up whenever I am starting my build its not moving to the next stage and its remaining in development stage only, basically its getting stuck in "Generate Build" step. Below are the logs which I am getting. Please help on this
Logs:
2017-12-22 06:31:52,136 [Jayesh Release Manager Devops] [DevOps01.01.01]
[DevOps_010101_1] INFO Logs of CICD build. BuildID: PEGA-PIPELINE-CD DE-3,
Branch list: Empty
2017-12-22 06:31:55,162 [Jayesh Release Manager Devops] [DevOps01.01.01]
[DevOps_010101_1] INFO Build NEW, Build ID:DE-3
2017-12-22 06:31:56,800 [Jayesh Release Manager Devops]
[DevOps01.01.01] [DevOps_010101_1] INFO Build is moved to new stage:Development
2017-12-22 06:31:58,672
[Jayesh Release Manager Devops] [DevOps01.01.01] [DevOps_010101_1] INFO New task started.
Task name:Publish, in stage:Development
Pegasystems Inc.
IN
Seems like Dev system is not able to send information back to Orchestrator system. There could be multiple reasons leading to this issue
a) Check if you have specified correct authentication profile for Dev system while configuring your pipeline.
Make sure credentials specified in this auth profile work on the Dev system. If this is not correct then specify correct credentials. Abort current build and launch a new one.
b) Check if authentication profile ReleaseManager is created on your Dev system. Check if credentials specified in it point to your release manager operator on orchestrator system. Dev system sends process completion information to Orchestrator system using this operator.
Please let us know if the issue doesn't get resolved from above steps.
p.s. check in build level report (from gear icon in build's row on the landing page) on the status of other tasks.
if it shows multiple tasks in the report then a manual refresh of landing page will show the updated state of build.
Thanks,
Mudit Sharma
ATOS
IN
HI Mudit,
My authentication profile seems to be correct. I am using authentication profile on RM system which contains credentials through which SOR/Dev can be accessed. But I am having one more doubt that if I am using Dev,SOR and RM on same system, then how can I use same application as dev and SOR.
And whether I should include "PEGADevOpsFoundation" layer into my application stack of dev.
Also can I have you email id so that I can send you screenshots of my configuration details.
Pegasystems Inc.
IN
Hi Jayesh,
Just wanted to notify you that you can post you config details right here after masking all internal/proprietary information so that all of the community can help or benefit. If you are unable to do so or if this needs someone to look into your system for you, please create an SR with Pega Support.
Regards,
Pegasystems Inc.
IN
hi Jayesh,
I believe in your setup RM=SOR=DEV system. If so then add "PEGADevOpsFoundation" application to your pipeline's application i.e. app for which pipeline is created.
Secondly, did you verify that "ReleaseManager" auth profile has been correctly created on all systems?
The issue you are facing seems to be due to broken communication between RM and DEV systems. Even though in your case both are same but still REST calls are made to complete the process. Therefore, verifying all auth profiles is our first line of investigation.
If "ReleaseManager" auth profile is also correct, then can you please go to Pega logs (in our general way via DS>Systems->Operations->Logs) and check if any logs are generated after you start the build.
Thanks,
Mudit Sharma
ATOS
IN
HI Mudit,
Thanks for the reply.
I checked the "ReleaseManager" profile which has the credentials of RM administrator (Admin.Devops_RM) which seems to be correct. Also I am using RM=SOR=DEV set up, so only 1 "ReleaseManager" auth profile can be created. I am not getting any logs after generating the build via (DS>Systems->Operations->Logs) .
On RM system I have created auth profile which has credentials of administrator for the app which I am building the pipeline. Is there any other auth profile whose configuration I should check ? I am not sure what I am doing wrong.
Do I need to associate auth profiles with ruleset. But that will restrict other app for the "ReleaseManager" auth profile I guess. I have got one error in tracer for which I have added the attachment.
ATOS
IN
Hi Mudit,
Can you please suggest something if you have some other set of checks which we need to do, to identify where the issue really exists here.
Thanks & Regards,
Jayesh
Pegasystems Inc.
IN
Hi Jayesh,
I have given few clarifications and posted couple of questions please address them.
1) One "ReleaseManager" auth profile is enough in your setup, no need to associate auth profiles to any ruleset.
2) Can you confirm if your pipeline application on Dev and SOR is built-on "PegaDevOpsFoundation:01.01.01"?
On RM system I have created auth profile which has credentials of administrator for the app which I am building the pipeline.
3) Can you confirm, if this user logging into pipeline application and designer studio portal?
I am not getting any logs after generating the build via (DS>Systems->Operations->Logs)
4) In your case Dev=SOR=RM, so you should definitely get some logs after triggering of build. Can you re-check and send us the logs? If you have lost those logs then you can re-try the scenario by aborting this build and start new build.
Thanks
Kishor
ATOS
IN
Hi Kishor,
First of all , I am having one SOR and one RM. Same application I cant have as Dev and SOR on same system. SO I don't have dev technically.
I have SOR an RM which are on same system along with dev and prod repository. SOR is build on "PegaDevOpsFoundation:01.01.01". Am I doing it wrongly? Can I have SOR application with different application name?
Yes this user login into designer studio portal. But it doesn't have access to RM application.
I will try again and share you the logs if I will get any. Please check my replies for the questions you have asked and correct me if anything looks wrong.
Thanks,
Jayesh
Pegasystems Inc.
IN
Can I have SOR application with different application name?
I did't understand this part. For example, if you are creating pipeline for an application "MyApp1" then in the MyApp1 depends on application stack it should have PegaDevOpsFoundation application on SOR system.
Yes this user login into designer studio portal. But it doesn't have access to RM application.
Yes. This user should login to pipeline applcation not RM application which is correct.
ATOS
IN
Hi Kishor,
Suppose I am building the pipeline for dev application "MyApp1", then on my server I have SOR which is nothing but ""MyApp1" built on PegaDevOpsFoundation and separate RM application. On this server I don't have dev application "MyApp1", its only has SOR and RM.
Hope this answers your query. Please let me know if you have any further doubt.
Thanks,
Jayesh
Pegasystems Inc.
IN
Hi Jayesh,
Yes it answers my question. Please share the prpc logs on start of new build.
Note: SOR is not an application, it is yet another PRPC server which acts as central repository of your applications.
Updated: 10 Jan 2018 11:30 EST
ATOS
IN
Hi Vasak,
Please find the logs attached which I am getting on start of new build.
Please help me to resolve this issue.
Thanks,
Jayesh
Updated: 10 Jan 2018 11:29 EST
Pegasystems Inc.
IN
hi Jayesh,
This issue is due to http port not open. your log file has below error
2018-01-10 08:31:18,989 [.PegaWorkManager : 2] [ STANDARD] [ ] [ DevOps:01.01.01] ( connect.rest.RESTConnector) ERROR Rule-Connect-REST.Pega-Int-Pipeline.pzPostTaskStatus Jayesh - HTTP connect execution failure: org.apache.http.conn.HttpHostConnectException: Connect to XX.XX.XX.XX:80 [/XX.XX.XX.XX] failed: Connection refused: connect java.lang.RuntimeException: org.apache.http.conn.HttpHostConnectException: Connect to XX.XX.XX.XX:80 [/XX.XX.XX.XX] failed: Connection refused: connect .
Response of publish is logged as
hi Jayesh,
This issue is due to http port not open. your log file has below error
2018-01-10 08:31:18,989 [.PegaWorkManager : 2] [ STANDARD] [ ] [ DevOps:01.01.01] ( connect.rest.RESTConnector) ERROR Rule-Connect-REST.Pega-Int-Pipeline.pzPostTaskStatus Jayesh - HTTP connect execution failure: org.apache.http.conn.HttpHostConnectException: Connect to XX.XX.XX.XX:80 [/XX.XX.XX.XX] failed: Connection refused: connect java.lang.RuntimeException: org.apache.http.conn.HttpHostConnectException: Connect to XX.XX.XX.XX:80 [/XX.XX.XX.XX] failed: Connection refused: connect .
Response of publish is logged as
2018-01-10 08:31:15,977 [.PegaWorkManager : 2] [ STANDARD] [ ] [ DevOps:01.01.01] (apper.Pega_Int_Pipeline.Action) INFO Jayesh - Task execution completed for task : publish
Posting status to release manager system. Status object:
{
"pxObjClass":"Pega-Int-Pipeline"
,"pyApplicationName":"DevOps"
,"pyApplicationVersion":"01.01.01"
,"pyBuildArtifactName":"DevOps_010101_7.zip"
,"pyCallBackURL":"http://XX.XX.XX.XX:80/prweb/PRRestService/cicd/v1/task/publish/status?FlowName=pzPublishRAP&FlowActionName=pzPauseTask"
,"pyDoPackage":"true"
,"pyID":"PEGA-PIPELINE-CD DE-13"
,"pyProductName":"DevOps"
,"pyProductVersion":"20171222.062732"
,"pyRepositoryName":"Devops_1"
,"pyStatusMessage":"Publish failed.Please verify the path details.Forbidden"
,"pyStatusValue":"FAILED"
}
Resolution:
1. Can you check if dev repository configured for pipeline is of type S3/JFrog. It would work with these 2 types only.
2. Open you dev repository instance and do "Test connection". Check if this connection is successful
Thanks,
Mudit Sharma
ATOS
IN
Hi Mudit,
My repository is of type JFrog and I am getting "Connection Successful: Authenticated" on test connection.
This means the port should be open. Why this forbidden error is coming, if I am getting connection successful?
Should I open a product support ticket now?
Thanks,
Jayesh
ATOS
IN
Hi Vasak/Mudit,
PFA the logs for my current build. In generate build my task publish completed successfully and we can see the build into our dev JFrog repository. But after that it didn't moved to deployment stage at all.
Can you please suggest what can be the issue here.
Thanks,
Jayesh
Pegasystems Inc.
IN
hi Jayesh,
I'm still seeing lot of exceptions as
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to <<>ip> failed: Connection refused: connect at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:?] at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:338) ~[httpclient-4.5.2.jar:?] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:?]
Did you build says Development stage as completed and has moved to QA stage now? Can you attach screenshot of your build's status from visualization landing page ?
Thanks,
Mudit Sharma
ATOS
IN
Hi Mudit,
It stays in Development stage itself but in Jfrog repository the build is published, so it should move to QA stage.
Also in the log you can see that it says publish completed successfully. So why its not moving to QA stage.
Even we are not getting any response message from rest.
Thanks,
Jayesh
Pegasystems Inc.
IN
hi Jayesh,
After the publish task complete a rest call is sent from Dev Server to Orchestrator server. This rest call is responsible for moving the build further.
In your case DEV=RM system.
Looking at your logs it seems like your server is not able to make rest call to itself. We too faced similar issues and realized that port to the server needs to be opened to make call to itself. Please get it touch with your IT team to enable this for you.
Thanks,
Mudit Sharma
ATOS
IN
HI Mudit,
Thanks for your reply!
Actually the port is open, but the issue here is it is taking "public url" for connecting to RM and therefore its not able to connect. If we give "localhost" or private url then it can connect without any issue. I have updated the DSS -"RMURL" also with private ip but still its taking the public url only. Is there any any other setting through which we can set the RM url which the rest call should pick.
Thanks,
Jayesh
Pegasystems Inc.
IN
hi Jayesh,
We pick server's url from pxRequestor.pxReqContextURI . We cannot pick from DSS RMURL as it created only on Dev system(and Dev may not be equal to RM).
What is the reason that it is not able to connect using public URL ?
Thanks,
Mudit Sharma
ATOS
IN
Hi Mudit,
Thanks for your reply!
We have connected through VPN using internal URL and in the clipboard I can see that the value for pxRequestor.pxReqContextURI is private URL. But in the logs still the error is coming for public URL that it is not able to connect. I am not sure how in it is taking the public URL if pxRequestor.pxReqContextURI value is private URL.
Please note our server is configured to work with private ip only if the rest calls are happening from within the server.
Can we have sharing session if possible?
Thanks,
Jayesh
Pegasystems Inc.
IN
hi Jayesh,
Please raise an SR for same. It would be great if you can put SR number here. We can then take it up at priority and have the required sharing session.
Thanks,
Mudit Sharma
ATOS
IN
Hi Mudit,
Our CD part is working fine after the session we had last time. But now i am facing issue with CI part.
I have got one more 7.3.1 instance later and made that as my dev server. I have created a branch and created few rules into it. I have locked my branch and tried to publish my branch into the PEGA type repository through which i am connecting to SOR server. On pulish i am getting message that "Branch has been published successfully ,but pipeline was not able to initiate on the orchestration server".
Also in SOR system that branch is not present. How to check whether the branch got published and what can be the possible issue for this.
Also i think i don't have to add "PEGADevOpsFoundation" into my dev application stack. Am I correct?
Thanks,
Jayesh
Pegasystems Inc.
US
Hi Jayesh,
Glad you've got CD working. I'm hoping I can help you out with the CI process. It looks like your log is filtered by operator ID; it might be helpful to share the full log. The most relevant piece of information I see is this line:
Invalid JSON request. Reason : JSON text must start with '{' or '[' at 19 [character 8 line 4]
This line, along with the screenshot you've shared suggests that invoking the "cases" service on the RM system is failing for one of a few reasons.
Hi Jayesh,
Glad you've got CD working. I'm hoping I can help you out with the CI process. It looks like your log is filtered by operator ID; it might be helpful to share the full log. The most relevant piece of information I see is this line:
Invalid JSON request. Reason : JSON text must start with '{' or '[' at 19 [character 8 line 4]
This line, along with the screenshot you've shared suggests that invoking the "cases" service on the RM system is failing for one of a few reasons.
- Make sure the operator defined in ReleaseManager authentication profile has a correct use name and password.
- Make sure the operator defined in ReleaseManager authentication profile has the PegaRULES:PegaAPI role, or some other role which inherits from it.
- On the SOR system, make sure that your RMURL dynamic system setting has the correct hostname and format. It should end in "prweb/PRRestService"
- I see that the dev system in you're screenshot is not SSL enabled. If the RM system you are connecting to is not SSL enabled you may encounter this error. If this is just for trying out the DevOps features you can uncheck "Require TLS/SSL for REST services in this package" on the "api" service package, otherwise I'd recommend into securing the environments with SSL certificates.
If none of these resolve your problem, and you want to explore this a bit on your own you can try tracing the the service named "artifacts" in the PegaRepository service package on the SOR system. Look at the value of param.responseBody after the Connect-REST call. The response may give a better indication as to what is going wrong.
Also i think i don't have to add "PEGADevOpsFoundation" into my dev application stack. Am I correct?
The answer to this is, Yes and no. The application record should ideally be the same on both the Dev and SoR systems, but it is only required to have PegaDevOpsFoundation on the SoR system. If you are seeing the error which is shown in your screenshot, then you have this configured correctly.
If none of these suggestions are helpful I'd be happy to go through some more debug steps with you.
-Ryan
ATOS
IN
HI Ryan/Mudit,
I tried these steps but it didn't resolved my issue for CI. PFA, logs and tracer screenshot.
Can we have a sharing session , i guess that will be helpful.
Thanks,
Jayesh
Pegasystems Inc.
IN
hi Jayesh,
Branch's name should not have "_" in it's name. Please create a new branch with no special characters in the name and try again.
Thanks,
Mudit Sharma
ATOS
IN
Hi Mudit,
I have created new branch without any special character and tried again, but it is not resolving the issue.
I am still getting same message. PFA
Can we have a session like last time? It will help a lot.
Thanks,
Jayesh
ATOS
IN
Hi Mudit,
Now i am getting the below error message. Do i need to delete the restore point created for this branch in SOR.
How can we clean the stale version of the branch manually?
"Branch was successfully published. The CICD process failed to clean up a stale version of this branch."
Thanks,
Hi Mudit,
Now i am getting the below error message. Do i need to delete the restore point created for this branch in SOR.
How can we clean the stale version of the branch manually?
"Branch was successfully published. The CICD process failed to clean up a stale version of this branch."
Thanks,
Jayesh
Pegasystems Inc.
US
Hi Jayesh,
From your follow up comment it sounds like you got everything working. If you weren't sure, and also for anyone else who may encounter this error, this can be caused if the branch has checked out rules on the SoR system. The checkouts prevent the stale version of the branch from being deleted.
-Ryan
Pegasystems Inc.
US
Hi @JayeshV8678!
So glad to hear that it's working for you now. Could you share your resolution and mark that as the Correct Answer? That will help others who may have the same question later.
Thanks!
Ernst & Young LLP
IN
Hi All,
Now I am able to connect between DEV(8080) and orchestration(8082) server . Artifacts are getting created and its storing in Dev Repository.
Now I am trying to move Zip file to staging environment (8081) so here I have imported "PegaDevOpsFoundation" package and I changed
DMAppAdmin operator id password (same as DEV and Orchestration Authentication profile)
But that staging environment deployment is not happening .Please let me know if anything I have to add in Staging environment.
Thanks !
Pegasystems Inc.
US
Since you updated your post with this information and this is an older thread, I'm going to close it to future comments.
Please continue the discussion on your thread.
Thanks!