This document describes the baseline performance for Reporting for the Confluence application. It serves as a reference of the application performance at its current state with the setup outlined in this document. However, please note that this is not a guide for application performance tuning.

Approach

We use a similar setup to the DCAPT (Data Center Application Performance Test) framework used by Atlassian to design the test to set baseline performance for Reporting for the Confluence application.  We appreciate your sharing any baseline-related experience you have or may have in the future - please go ahead and Email us.

Test Environment Setup

We are running our test in an AWS environment using AWS Quick Starts for deploying Confluence Data Center. AWS environments allow us to scale up and easily do modifications in the components as needed quickly. The specifications are as follows:

Below is the specification where Confluence is running as a server. It can scale up between 1, 2, and 4 nodes during the test.

Below is the specification for the test runner that simulates the number of users accessing the Confluence server virtually. The specification below can be adjusted depending on how many concurrent users we plan to run.

Technology

Our team is using Taurus to configure and run automated performance tests. While it is an open-source tool, Taurus also provides a variety of testing tools from JMeter, Gatling, Selenium, and many more. It then provides the result in the most effective report.

In this test, we use Selenium executor to run functional tests locally with Selenium WebDriver using Chrome browser. It uses JUnit test runners to assert whether the page is rendered correctly.

Methodology

We ran three separate phases based on the number of nodes; 1 node, 2 nodes, and 4 nodes. In each phase, we executed different scenarios sequentially by modifying the number of concurrent users. In this case, we had two variables that affect the time to render which are the number of nodes and concurrent users.

Our main objective was to benchmark Reporting performance in different configurations based on the number of concurrent users and the number of nodes. Especially, how long it takes for each Reporting scenario to render and how the number of nodes can affect the performance.

For the purposes of this test, we doubled the number of concurrent users on each test run with the assumption of 5 concurrent users running on a single node as a baseline. 

Descriptions:

• Concurrent user - number of targets concurrent virtual users.

• Ramp-up - ramp-up time to reach target concurrency.

• Hold-for - time to hold target concurrency.

• Iterations - limit scenario iterations number.

Test scenario and dataset

We use the enterprise-scale dataset provided by Atlassian for these tests. We deliberately excluded Attachments files in the test data but maintained the links to Attachments as the attachment file size itself does not affect the Reporting query process. 

We selected common scenarios and macro usage that have a significant impact on the performance. Below is the list of Reporting for Confluence scenarios that are included in the test:

• Building a forum in Confluence with Reporting

• List all Comments including unresolved inline Comments

• List all Contents in a Space

• Displaying a list of children for a Page with customized details

• List all Pages in a Space

• Listing all available Spaces

• Check whether a Page exists

Test Results

Test results on single node setup

Open

Test results on 2 nodes instance

Open

Test results on 4 nodes instance

Comparing average response time on different nodes

Conclusion

From the test results, we can see that at 20 users, the increase from 1 to 2 nodes in an instance can reduce the average response time for the page to load. However, increasing nodes from 2 to 4 nodes do not show a significant difference reduction in the time for the page to load.

Please note that the time taken for a page containing Reporting macros to load does not necessarily represent the time taken for Reporting macros to render as there are other factors contributing to the loading time such as the presence of macros from other plugins. The page will only load once all macros, including macros from other plugins, are completely rendered.