Limitations of Pega Scenario testing
Types of scenario tests:
Scenario test cases can be created either for a case type or a portal. It is not possible to record a test case from a certain point of a case type flow. One need to record a test case from the starting of the case type flow to whatever step/stage required. For other scenarios, portal level test cases can be used.
Persona based testing:
Scenario testing framework runs within the platform and can be run only in requestor context i.e. a login to Pega system is needed. Due to this limitation, a single test case can’t offer recording scenarios involving multiple personas or multiple login/logout actions. This is in the roadmap and will be addressed in future.
Data driven (Parametrization) approach:
Data is tightly coupled with test cases. It is not possible to run the same test with different input/output datasets. Each test is associated with a single input/output data set. Multiple test cases need to be created to test a scenario with different input/output combinations. This is in the roadmap and will be addressed in future.
Scheduling of tests:
There is no out of the box support to schedule the scenario test cases or the test suites created for an application stack. You may have to leverage open source tools like Jenkins to schedule tests to run in the pipeline on a daily basis.
Running of tests from CI pipeline
To run scenario tests from the pipeline, customers should have accounts with third-party platforms like Cross Browser testing or an in-house selenium grid which can take care of launching the Pega instance and logging the user into the portal. You may refer to following articles to understand how to integrate the tests with CI tools.
https://collaborate.pega.com/discussion/browserstack-integration-scenario-testing-framework
https://collaborate.pega.com/discussion/crossbrowsertesting-integration-scenario-testing-framework
https://collaborate.pega.com/discussion/selenium-grid-integration-scenario-testing-framework
Keeping tests inside tests
A scenario test cannot be included in another scenario test, which is generally the case with other test frameworks that allow atomic level test cases to be included in the main test case
Execution of scenario tests
Scenario tests are portal dependent. i.e., once recorded on a portal, they cannot be manually run from a different portal. They have to run in the portal on which they are recorded. If you are executing the tests from developer studio or through a pipeline, it is possible to execute the test cases from built-on applications across different portals from a top layer application, provided correct application quality settings are configured to include all the required applications.
Support to various UI elements & scenarios for recording:
Though majority of the out of the box UI controls are supported by this framework, there are few controls and known scenarios which are not supported. Please refer following information to understand what is supported and what is not.
Controls supported
- pxButton
- pxTextInput
- pxMenuItem
- Label
- pxRichTextEditor
- pxLink
- pxIcon
- pxDisplayText
- pxTextArea
- pzModalButton
- pxGridEditModalSubmit
- pzOverlayButton
- pxDateTime
- Paragraph
- pxEmail
- pxPhone
- pxPassword
- pxNumber
- pxURL
- pxCurrency
- Decimal
- pxInteger
- pxSlider
- pxRadioButtons
- pzMultiFilePath
- FieldError
- CustomErrorList
- pxDropDown
- pxAutoComplete
- pxCheckbox
- pxAnyPicker
- pxMultiSelect
- pxAttachContent
Controls not supported
- Signature capture
- Hidden Text
Layouts supported
|
Structural Component |
Exceptions |
|
Tab |
Nil |
|
Embedded Section |
Nil |
|
Columns |
Nil |
|
Dynamic Layout |
Title text |
|
Repeated Dynamic Layout |
Reordering, show details |
|
Layout Group |
Touch and swipe |
|
Dynamic Layout Group |
Adding new tab, Header text, Touch and swipe |
|
Table |
Filtering, sorting, reordering, header text |
|
Hierarchical Table |
Filtering, sorting, reordering, resizing, node opening, header text, editing on “click row to edit” mode |
|
Navigational Tree |
Filtering, sorting, reordering, resizing, node opening, header text |
Layouts not supported
- Dynamic Container
- Ajax Container
Some known scenarios that are not supported
- Popup windows are NOT supported to be recorded or asserted. Currently the recording scope stays with elements of the main window.
- Click is needed from mouse or touch pad to record a DOM element and make assertions. Hovering over will NOT be recorded by scenario testing framework as they is no click involved.
- Scenario testing framework does NOT support saving or parsing files (like pdf, xlsx) as part of the test case. Hence, file operations like assertions on processed file content, assertions with file contents cannot be done.
- Date values entered in date fields can NOT be referred from system to use the present date. It will select the same value used while recording over and over.
- Modal dialog boxes which is usually used for confirmation to proceed or cancellation flow are NOT supported to be recorded or asserted.
- Whole tab refresh while recording a test case might cause abrupt FAILURE during test case execution.
- Dropdown control that gets populated with dynamic values/options can NOT be recorded as value to be selected has to be included in test case at the time of recording
@vadlp Hi,
Thanks for this information. Can you please let us know the assertion features in details(comprising the role of drop down values and how will it impact the test step) like how can we leverage this. any realtime/hypothetical example will be suffice. Also consider below points.
a) is it pop window or modal window which is not supported
b) in Pega we will have case id generated for each instance whose format could be prefix-Incremental value (i.e. S-1,S-2.....) in our application the format is S-W-1,S-W-2...or S-N-1,S-N-2... how can we configure this so that while execution it should pick dynamic case id which will always have incremental value.
c) we have an option of selecting multiple test suites in Dev Studio for execution, how will it execute once launched from portal or we have to run one test suite at times
d) There is no search button or filter criteria on portal against recorded test cases, imagine if I want to change certain fields/add step, rerecord in a test case on top of which I have created 1000 test cases, I have to click on load more,more... to reach there. this is cumbersome
e) Can user A and B at same time can run the test suite S-1 from their individual login(Portal is same). what will happen in that case?
Thanks for adding time delay settings on individual step
Regards,
Prajapati
For dropdowns, Scenario test simply captures the entered value and during run time same value is applied. if you want to verify list of values present in dropdown, better to write pegaunits for source of the dropdown. For suppose, if your dropdown source is Data page, better to write pegaunit for data page and assert the list of values returned in data page.
Coming to the assertions feature, As of now, when you record a step for drop down for entering values. scenario test simply takes the DOM properties of dropdown and keep them in assertions. like for a dropdown it simply captures value, tooltip, required, hasErrors etc.. And these assertions will be executed before execution of the step and see if there are any assertion failures and go for actions to perform on dropdown. As of now, we have only one comparator only i.e "Is Eauals" we are planning to extend these comparators in future. Below is the screen shot to explain assertions captured for dropdown.
a) Its Popup window. its like another browser window comes as pop on click of link.
For example, shown in below image. when you click "Advanced" from attachments in case type. popup window comes
For dropdowns, Scenario test simply captures the entered value and during run time same value is applied. if you want to verify list of values present in dropdown, better to write pegaunits for source of the dropdown. For suppose, if your dropdown source is Data page, better to write pegaunit for data page and assert the list of values returned in data page.
Coming to the assertions feature, As of now, when you record a step for drop down for entering values. scenario test simply takes the DOM properties of dropdown and keep them in assertions. like for a dropdown it simply captures value, tooltip, required, hasErrors etc.. And these assertions will be executed before execution of the step and see if there are any assertion failures and go for actions to perform on dropdown. As of now, we have only one comparator only i.e "Is Eauals" we are planning to extend these comparators in future. Below is the screen shot to explain assertions captured for dropdown.
a) Its Popup window. its like another browser window comes as pop on click of link.
For example, shown in below image. when you click "Advanced" from attachments in case type. popup window comes
b) It will be addressed in release two of scenario testing. from release two onwards, you can setup actions for scenario test cases. ex: Creating a work object or execution of activity before the scenario test execution and you can refer those dynamic values as part of test case steps by accessing dynamic as step data. the release two of scenario test would be some where around early month of May.
c) I did not get your question, Can you please elaborate your use case. but as per my understanding. you can execute test suite from Dev portal and rest service. But it can not be executed from any other portal. But from Dev portal if you select two test suites, it will try to execute is sequence manner, one suite after the other suite.
d) We shall try to address it this in release 2. Alternatively you can open test case from landing page in Dev portal( Configure-> Application->Quality-> Automation testing-> Scenario testing and filter the test case from landing page and open the test case and edit it.
e) Yes, User A and B can execute same test case from different logins, basically when user A and B logins with different logins they are different requestors the will not have any impact of other user execution. In general, its like two different users login and doing their actions.
We are currently on Pega 8.4.2 version and can you please let me know when can we get the below feature in Pega's scenario Testing OR let me know if its already available in higher versions.
1. Ability to test with multiple persona's for a case lifecycle.
2. Dynamically pass the case data before the instantiation of the test case.
3. As mentioned above anything on the "Data driven (Parametrization) approach:"..
Kindly respond as we are evaluating on leveraging the pega test scenario Automation and in case if this features are already available then we can even go for Pega 8.5 Version.
1)Ability to test with multiple persona's for a case lifecycle.
Ans) Currently multiple personas are not supported. Multiple personas testing is supported in future releases.
2) Dynamically pass the case data before the instantiation of the test case.
Ans) This is supported from 8.5 and above platform versions , for this you can download the 'PegaTestAutomationKit' ( https://community.pega.com/marketplace/components/pega-test-automation-kit ) from market place and add this component to your application. From 8.5 platform version on words scenario test is separated from platform and releasing the features as part of component 'PegaTestAutomationKit'
Note: Minimum platform version to support 'PegaTestAutomationKit' is pega 8.5.
3) As mentioned above anything on the "Data driven (Parametrization) approach:"..
Ans) As of now, we support setup to scenario testing, i.e you can call activity or DT or create work or data instances before running scenario test. But data driven scenario test is not supported as of now. We will consider this in future releases.
Thanks for your response.
For #2 point, if Scenario Test Automation is a component from Pega 8.5 version, is there a plan pega releases new features in that component frequently? Also, if we are on pega 8.5 version can we able to use latest releases in it component or will that tag again to the latest pega version only ?
Is there any documentation on the New features which are available in Pega 8.5 corresponding Test case automation kit compared to Pega 8.4.4 inbuilt test case automation.
This component is support all the versions from 8.5 and above PRPC platform versions.
I hope you have gone through this link ( https://community.pega.com/marketplace/components/pega-test-automation-kit ). This link has release notes and related documentation links.
Thanks for all the responses. One more question please.. We have a Page list which is displayed through repeating grid or Navigation tree, Currently we are noticing all the pages in that page list is having same "data-test-id" and with that during the scenario test recording its not recording the list fields.
Also in Pega 8.4.4, it says Auto-complete & multi select controls are not supported. We have few auto-complete controls and in the box when we paste the complete text(Instead of selecting it from the auto complete results) in that box its working fine.
Need to understand whether we use that way for the auto complete fields.
Thanks. Currently we are in Pega 8.4.4 version as we just upgraded and its not feasible to upgrade again.
We have a dynamic grid, and currently all the rows for a checkbox field is having the same Test ID. We are not even looking for Assertion of any fields in the grid, as at least the recording of checkbox selection in the grid will help us to record the test case.
Any inputs on how to generate the unique test id for each row of a field in dynamic grid..
@RajeshVenkata is pxIconAddItem supported in scenario testing? I have a grid which is having this control to open a model dialog and while creating scenario test it is not capturing this control and my test is failing. I am using 8.6.3.
Current framework does not support 'pxIconAddItem'.
This kind of controls are recorded by using custom steps in pega scenario testing. which is part of pega test automation kit 1.6.
Once PTAK 1.6 released you can add custom steps based on xpath, where you can find xpath of this control on UI and try to add a step based on xpath, After adding custom step you can run test case with this control.
To download 1.6 version, as soon as its available, keep an eye on below, 1.6 would be release soon. linkhttps://community.pega.com/marketplace/components/pega-test-automation-kit
are there any limitations running the scenario tests from user portal? I would like to provide the ability to the business users to run the scenario tests from their logged in portal(example: user portal).
I have tried to build a new landing page with the same screen as dev studio landing page. The scenario tests are not getting executed. Please see the - INC-250589 for all details.
are there any limitations running the scenario tests from user portal? I would like to provide the ability to the business users to run the scenario tests from their logged in portal(example: user portal).
I have tried to build a new landing page with the same screen as dev studio landing page. The scenario tests are not getting executed. Please see the - INC-250589 for all details.
are there any limitations running the scenario tests from user portal? I would like to provide the ability to the business users to run the scenario tests from their logged in portal(example: user portal).
I have tried to build a new landing page with the same screen as dev studio landing page. The scenario tests are not getting executed. Please see the - INC-250589 for all details.
are there any limitations running the scenario tests from user portal? I would like to provide the ability to the business users to run the scenario tests from their logged in portal(example: user portal).
I have tried to build a new landing page with the same screen as dev studio landing page. The scenario tests are not getting executed. Please see the - INC-250589 for all details.
@vadlp I noticed there seems to be a limitation when it comes to using Layout Group(Tab). While recording the scenario test, it does not pick up clicking on a tab. The orange box will appear around the tab, signifying that it knows it's an element, but you cannot mark for assertion, etc. This causes our tests to fails as we have embedded sections that on deferred loads. If anyone has ran into this or has any suggestions, I'd love to hear them.
There is a limitation to switch the tab in dynamic layout in scenario testing. to overcome it you can use the custom steps which can accept xpaths for click actions. in your case you need to click on tab and switch to tab context. to know about custom steps you can follow link. https://docs.pega.com/bundle/platform-88/page/platform/devops/adding-custom-steps-in-scenario-tests.html Ex custom steps for tab switching :
//h3[@class='layout-group-item-title'][text()='Customer Information'] - for tab switching , you may have to modify your xpath as per your browser html rendering. @dawam6
Hi @vadlp,
We are currently assessing Pega scenario test framework (PegaTestAutomationKit v 1.6) for the internal facing Pega application which is leveraging UIKit for the user interface. While recording the test case we have observed that the scenario test framework doesn't recognise the interaction with pyAssignmentsLabel control on the screen. I am wondering if there is an OOTB way to workaround it so we can record this by manually editing the test case?
Should we expect the support for this control in the future release of PegaTestAutomationKit?
Thank you,
Piyush