diff --git a/src/main/xar-resources/data/integration-testing/integration-testing.xml b/src/main/xar-resources/data/integration-testing/integration-testing.xml index 78eb5b1f..9ecfa5b4 100644 --- a/src/main/xar-resources/data/integration-testing/integration-testing.xml +++ b/src/main/xar-resources/data/integration-testing/integration-testing.xml @@ -10,7 +10,7 @@ - This article discusses intergration testing of eXist-db applications. It also covers recommendations for the configuration of automated test environments, and explains the minimum testing requirements for apps that are published under the + This article discusses integration testing of eXist-db applications. It also covers recommendations for the configuration of automated test environments, and explains the minimum testing requirements for apps that are published under the eXist-db namespace. It assumes that you are familiar with the XQSuite framework @@ -20,118 +20,118 @@ Introduction - Creating an automated mininal testsuite is possible with relatively little effort. It pays to take the need for testing into account when you start developing your application. This enables others to extend your program with new features, by - knowning that these don't break existing functions. It also allows test-only contributions, helping you to gradually advance your test coverage. The following section will walk you trough the three main aspects of such a minimal test setup. + Creating an automated minimal test suite is possible with relatively little effort. It pays to take the need for testing into account when you start developing your application. This enables others to extend your program with new features by + knowing that these don't break existing functions. It also allows test-only contributions, helping you to gradually improve your test coverage. The following section will walk you trough the three main aspects of such a minimal test setup. Building on a clean system Before you start designing tests, you should start to automate your build process. This ensures that things don't only work on your system, and it can catch some common errors. The examples in this article will use - travis-ci - as it is the most popular continual integration (ci) provider in the eXist-db organization on Github, other popuar choices include - appveyor, - jenkins, - circl-ci, … . - The services typically require a small configuration file, so you can build your code on a clean virtual machine, without the risk of local files interfering. For travis the required name of such a file is - .travis.yml + Travis CI + as it is the most popular continuous integration (CI) service used by the eXist-db organization on GitHub. Other popular choices include + AppVeyor, + Jenkins, and + CircleCI. + CI services typically require a small configuration file, instructing the service how to run your code on a clean virtual machine, without the risk of local files interfering. For Travis the required name of such a file is + .travis.yml, and in its simplest form it would look like this: - For the correct way to create such a configuration file for other ci providers please consult their documentation. In all cases, since eXist-db is written in Java, your app should be build on a system that comes with the minimal Java version + For the correct way to create such a configuration file for other CI services please consult their documentation. In all cases, since eXist-db is written in Java, your app should be built on a system that comes with the minimal Java version required by eXist-db. - Most providers will automatically detect your build tool and run the required command even if you don't specify it. In the above example, our app to be tested uses + Most CI services will automatically detect your build tool and run the required command even if you don't specify it. In the above example, our app to be tested uses ant - as a build tool, change + as a build tool; change ant to suite your needs, e.g. maven clean package, npm install, etc. - If you have multiple build targets for production and development, you should make sure that each build target is actually run by the ci service. + If you have multiple build targets for production and development, you should make sure that each build target is actually run by the CI service. You can extend this basic template according to your needs, e.g.: apps written in Java might want to run the build step on multiple Java versions (by adding - - openjdk11); or to test building on different operation systems. You should consult your ci services documentation for the list of available configuration options. + - openjdk11); or to test building on different operation systems. You should consult your CI service's documentation for the list of available configuration options. Add a running eXist-db instance and install your app - The next step takes the result of your automated build process and installs it in a running eXist-db instance (external tools might want to talk to a running eXist-db instance in some other way). We are going to use exist's - docker images - for this, since it is supported by all ci providers, and it tends to be the quickest way of getting an instance up and running. Let's extend the file created in the previous section. + The next step takes the result of your automated build process and installs it in a running eXist-db instance (external tools might want to talk to a running eXist-db instance in some other way). We are going to use eXist-db's + Docker images + for this, since it is supported by all CI services, and it tends to be the quickest way of getting an instance up and running. Let's extend the file created in the previous section. The :release and :latest - tags are specifically designed for use in ci environments. You can also specify exact version to use (e.g. - :4.4.0) to ensure backwards compatibility. These two tags will ensure that your code is tested against both the most current stable release, and upcoming changes ahead of time. + tags are specifically designed for use in CI environments. You can also specify exact version to use (e.g., + :4.4.0) to ensure backwards compatibility. These two tags will ensure that your code is tested against both the most current stable release and upcoming changes ahead of time. - To actually install the app, we copied it into exist's + To actually install the app, we copied it into eXist-db's autodeploy folder, which will make sure that any dependencies that you declared for your app will also be installed. If you require more complex installation steps, you can find more examples and links in the docker-existdb readme. So far, we have simply automated the basic steps of building and installing your app. This already catches some basic and particularly severe errors, but it is not a very realistic test of what users actually experience when they install your - application. Before we can refine the way we simulate usage patterns, we first need to add the means of running actual tests within our ci environment. + application. Before we can refine the way we simulate usage patterns, we first need to add the means to run actual tests within our CI environment. - Integrating unit tests into ci - Integration testing and unit testing go hand in hand, as one without the other does not work well. If you are writing an application, you already should have unit tests for the functional components of your code. By running your unit tests - inside your ci server, these become immediately visible to potential contributors, and you have the advantage of immediate feedback on every code change. + Integrating unit tests into CI + Integration testing and unit testing go hand in hand, since one without the other does not work well. If you are writing an application, you already should have unit tests for the functional components of your code. By running your unit tests + inside your CI server, these become immediately visible to potential contributors, and you have the advantage of immediate feedback on every code change. - Tests that are invisible to other contributers because they are hidden away, and have only ever been run on the original author's system, are of very limited use. + Tests that are invisible to other contributors because they are hidden away and have only ever been run on the original author's system are of very limited use. As with the previous options there are different test runners to do this work for you, such as - junit + JUnit for Java, Mocha for JavaScript, XQSuite for XQuery. To run your tests, we are going to leverage the support for running unit tests of our build system (e.g.: npm test, - mvm test, …): + mvm test, etc.): - Just as with the building example, many providers will execute this command automatically. But even in a simple case it helps others understand your code, and to find your tests, if you make the test command explicit. + Just as with the building example, many CI services will execute this command automatically. But even in a simple case it helps others understand your code, and to find your tests, if you make the test command explicit. - If you use more then one test runner, you can simply add additional test commands to the script parameter. Since exist with our app is already running in the background, it is also possible to run your + If you use more then one test runner, you can simply add additional test commands to the script parameter. In the case of our app, eXist-db is already running in the background, so it is also possible to run your XQSuite unit tests. You can see how this is configured, for apps using our the yeoman templates. - How to write good unit tests is beyond the scope of this article. Whenever you are struggling with your integration test you should, however, ask yourself if what you are trying to achieve might not be better served by creating unit tests. - Whichever solution works best for you, you should not rely on either unit or integration tests alone, and both should be integrated into your ci pipeline. + How to write good unit tests is beyond the scope of this article. Whenever you are struggling with your integration test, however, you should ask yourself if what you are trying to achieve might not be better served by creating unit tests. + Whichever solution works best for you, you should not rely on either unit or integration tests alone, and both should be integrated into your CI pipeline. Testing your app in a controlled context - Unlike unit tests which excel at testing individual functions, and are quick to write and perform, interation tests focus on the complex interaction between your code and that of the larger environment. Just search for + Unlike unit tests which excel at testing individual functions and are quick to write and perform, interaction tests focus on the complex interaction between your code and that of the larger environment. Just search for 2 unit tests 0 integration tests to see what we mean. - For eXist-db applications, integration tests will typically involve a browser, as we are trying to mimick the way a user interacts with our application from their system. + For eXist-db applications, integration tests will typically involve a browser, as we are trying to mimic the way a user interacts with our application from their system. - While our examples focus on browser testing, you can see shell based integration tests at the - testsuite - for building the docker images we used earlier. + While our examples focus on browser testing, you can see shell-based integration tests at the + test suite + for building the Docker images we used earlier. Common tools for browser testing include Cypress, Selenium, and - webdriver. As before the choice is up to you, whichever you choose it should be clearely documented so your contributors know how to adjust test cases for new features and how to + WebDriver. As before the choice is up to you, whichever you choose it should be clearly documented so your contributors know how to adjust test cases for new features and how to maintain your tests. We focus on Cypress, as it does not require any additional steps for configuring a browser first. If you need to perform cross-browser testing you can take a look at services such as - sauce-labs + Sauce Labs - You can simply execute the Cypress test command inside your ci test script after the unit test command we added earlier. + You can simply execute the Cypress test command inside your CI test script after the unit test command we added earlier. - With Cypress you write your tests in the same fashion as you would with Mocha unit tests; however, you now address the rendered document inside a browser instead of individual js functions. + With Cypress you write your tests in the same fashion as you would with Mocha unit tests; however, you now address the rendered document inside a browser instead of individual JavaScript functions. The above example opens a page (the dashboard) in the browser, logs in, and closes the window it just opened. You can do many more things, but these examples are meant to provide a good starting point for creating your first integration test. If there are any console errors or problems with rendering content Cypress will create an error message and your tests will fail. To check the syntax of these commands, and to see many more examples please visit the Cypress documentation. - Now that we have built our app in a clean system, executed its unit tests, and opened the start page of our freshly installed app in clean eXist-db instance, we have achieved a basic smoke test. We switched it on and there was no smoke. Proper - testing can now commence. Obviously you might want to visit multiple pages, or compare screenshots to avoid visual regressions, or compare images in multiple browsers, or … . - All of which are excellent ideas, and with these basics in place it hopefully no longer seem so daunting a task. Depending on your own specific requirments we would encourage your to browse other eXist-db repositories in addition to the - documentation of your ci and testsuite vendors. Chances are someone already has created a solid test, similar to your needs. + Now that we have built our app in a clean system, executed its unit tests, and opened the start page of our freshly installed app in clean eXist-db instance, we have achieved a basic smoke test. We switched it on, and there was no smoke. Proper + testing can now commence. Obviously you might want to visit multiple pages, or compare screenshots to avoid visual regressions, or compare images in multiple browsers, etc. + All of which are excellent ideas, and with these basics in place it hopefully no longer seem so daunting a task. Depending on your own specific requirements, we would encourage your to browse other eXist-db repositories in addition to the + documentation of your CI services and test suite frameworks. Chances are someone already has created a solid test similar to your needs. diff --git a/src/main/xar-resources/data/integration-testing/listings/travis-1.txt b/src/main/xar-resources/data/integration-testing/listings/travis-1.txt index 5657c258..8da3b0aa 100644 --- a/src/main/xar-resources/data/integration-testing/listings/travis-1.txt +++ b/src/main/xar-resources/data/integration-testing/listings/travis-1.txt @@ -1,7 +1,7 @@ -# Tell travis that we want java +# Tell Travis that we want Java. language: java -# This should be the minimal java version required by eXist-db +# This should be the minimal Java version required by eXist-db. jdk: - oraclejdk8 diff --git a/src/main/xar-resources/data/integration-testing/listings/travis-2.txt b/src/main/xar-resources/data/integration-testing/listings/travis-2.txt index 0e3e0f97..94d0fa54 100644 --- a/src/main/xar-resources/data/integration-testing/listings/travis-2.txt +++ b/src/main/xar-resources/data/integration-testing/listings/travis-2.txt @@ -4,11 +4,11 @@ jdk: - oraclejdk8 - oraclejdk11 -# Tell travis that we are using docker +# Tell Travis that we are using Docker. services: - - docker + - docker -# Always test against current and upcoming releases +# Always test against current and upcoming releases. env: - img=existdb/existdb:latest - img=existdb/existdb:release @@ -16,7 +16,7 @@ env: install: - ant develop -# take the .xar created above and install and deploy in a clean exist instance +# take the .xar created above and install and deploy in a clean eXist-db instance. before_script: - docker cp ./build/*-dev.xar exist-ci:exist/autodeploy - docker start exist-ci diff --git a/src/main/xar-resources/data/testing/testing.xml b/src/main/xar-resources/data/testing/testing.xml index 625d607a..a71f2677 100644 --- a/src/main/xar-resources/data/testing/testing.xml +++ b/src/main/xar-resources/data/testing/testing.xml @@ -17,8 +17,8 @@ Different kinds of tests play an essential role in maintaining a high quality code base both for eXist-db itself, and for applications that interact with eXist-db. The test framework of eXist-db and its apps is under continuous development, with frequent changes to the software and services in use. Covering them all is not feasible in this article. We recommend that you check the official - Github repositories - and search for systems that are similar to yours for further insipiration. + GitHub repositories + and search for systems that are similar to yours for further inspiration. While the terminology varies between different sources dealing with software testing, the following categories are widely acknowledged to apply irrespective of terminological differences. @@ -26,7 +26,7 @@ Validation: - XML comes with different means of validating and thus testing your data strucutures. When implementing a testsuite, you should first consider how your code can leverage these native features via strong typing and schema-based validation. + XML comes with different means of validating and thus testing your data structures. When implementing a test suite, you should first consider how your code can leverage these native features via strong typing and schema-based validation. @@ -40,19 +40,19 @@ Integration testing: - Here we look at your programm as a whole, similar to how a user would interact with it, by simulating user input and running you program not in isolation but in concert with different applications. + Here we look at your program as a whole, similar to how a user would interact with it, by simulating user input and running your program not in isolation but in concert with different applications. (WIP) Performance and Stress testing: - Performance testing ensures that your code keeps running under heavy load, or with very little resources. Stress tests go one step further by purposfully crashing your application to ensure that it recovers gracefully. Practically + Performance testing ensures that your code keeps running under heavy load, or with very little resources. Stress tests go one step further by purposefully crashing your application to ensure that it recovers gracefully. Practically speaking, the difference between the two is fluid. - When running these different tests in concert, we usually speak of end-to-end testing (e2e). There will never be a one-size-fits-all solution to take care of all your testing needs. Writing good testsuites takes time and planning. You can find - out more about available options and their use cases in the articles linked above. The remainder of this article will discuss some general considerations for how to design your testsuite, without referencing a specifc implementation. The listings use - pseudo-code to focus on relevant lines. Pseudo-code is not intended to be executable, so it won't run if you copy-paste it into e.g. eXide. For working code, please consult the articles links throughout. + When running these different tests in concert, we usually speak of end-to-end testing (e2e). There will never be a one-size-fits-all solution to take care of all your testing needs. Writing good test suites takes time and planning. You can find + out more about available options and their use cases in the articles linked above. The remainder of this article will discuss some general considerations for how to design your test suite, without referencing a specific implementation. The listings use + pseudo-code to focus on relevant lines. Pseudo-code is not intended to be executable, so it won't run if you copy-paste it into, e.g., eXide. For working code, please consult the articles links throughout. @@ -62,7 +62,7 @@ configure, with a higher chance of providing you with the information that you need when a test fails. Selection Example - To illustrate this, let's take a theoretical example: imagine your code includes an online form where users submit a date that needs to be stored in the database. For the purpuse of this example it doesn't matter if this is text-input form, a + To illustrate this, let's take a theoretical example: imagine your code includes an online form where users submit a date that needs to be stored in the database. For the purpose of this example it doesn't matter if this is text-input form, a calendar selector, or other form. Provided that you are using eXist-db, we assume that the date will be stored in some kind of XML file. A basic form example @@ -94,15 +94,15 @@ Testing Corner Cases For in-depth examples of unit testing in XQuery please see the examples in the - xQsuite article + XQSuite article Integration testing is most effective when it can rely on existing unit tests and focus on how a user typically interacts with the UI components. Does the browser display the input form correctly at different resolutions, can the form be selected with a mouse, or on - touch screens, … ? + touch screens, etc.? - browser testing + Browser testing @@ -113,17 +113,17 @@ Lastly, performance and stress tests - would simply assume all of the above to be in place. They would check for problems that might occur, when large numbers of users select the same or invalide dates, or access the same form simultaneously. - This quick example should give you an idea on how to think about testing your code. Each type of test has its own reason for being, and requires your attention. A good performance test cannot substitue unit or integration tests. If you feel - that your tests are not working well, because they frequently fail to show problems in your code, and when they do they don't make identifying the source of the problem easier, chances are that you are trying to test something with one type of test, + would simply assume all of the above to be in place. They would check for problems that might occur, e.g., when large numbers of users select the same or invalid dates, or access the same form simultaneously. + This quick example should give you an idea on how to think about testing your code. Each type of test has its own reason for being and requires your attention. A good performance test cannot substitute unit or integration tests. If you feel + that your tests are not working well, because they frequently fail to show problems in your code, and when they do they don't make identifying the source of the problem easier, chances are that you are trying to test something with one type of test that would be better suited for another type. Test Coverage - Ideally, we would always achieve full test coverage for our code. So that every meaningful unit of code has a corresponding set of tests. While other programming languages offer automated means of analysing your code to indentify areas that - aren't tested, such tools are unfortunately not very common for XQuery. How much testing is necessary for you to ship with confidence, rests on each developer. For apps published under the eXist-db namespace, we have outlined a set of minimum - requirementes (more is obviously desirable) which are featured in the section on + Ideally, we would always achieve full test coverage for our code. So that every meaningful unit of code has a corresponding set of tests. While other programming languages offer automated means of analysing your code to identify areas that + aren't tested, such tools are unfortunately not very common for XQuery. The task of determining how much testing is necessary for you to ship with confidence rests on each developer. For apps published under the eXist-db namespace, we have outlined a set of minimum + requirements (more is obviously desirable) which are featured in the section on integration testing.