When it comes to understanding what an App Factory is, think of the word “multi.” An App Factory is a multidisciplinary development function designed to bring together multiple parts of the business to create a multitude of projects.
Whether it’s centralized or not, the App Factory has some common denominators. One of those main commonalities every App Factory needs to address is quality assurance. A QA team ensures that every project, no matter the business unit or discipline, meets the organization’s compliance, branding, and performance standards.
Curious as to how to best implement QA into your App Factory? Take a page out of our book. At Mendix, we operate our own Mendix App Factory. Here’s how we approach QA with ours.
What is the Mx AppFactory?
Mx AppFactory is an internal unit in Mendix which builds applications for Mendix with Mendix. As a result, we have multidisciplinary development teams working on a multitude of projects for various departments and stakeholders within Mendix and Siemens.
About the Quality Team
The quality team is a cross-functional accelerator which helps drive the quality of the Mx AppFactory team’s deliverables. Started in Dec 2020, the team grew from one member to a team of four quality experts. The team approaches the factory line with a clear statement that “quality is the responsibility of the entire team.” The test engineer acts as a Quality expert to guide and drive quality activities in the team.
Our Mission
- To empower AppFactory teams to deliver the best-in class software and be true ambassadors for quality within Mendix and its customers.
How we drive our mission
We use the below wheel to help us ensure quality along the AppFactory lines.
Quality Assessment
As part of the preparation for a new project, we have to understand the technical and functional aspects of the product being built. Next step is to assess the project using information we gathered and derive required quality activities needed for the project. To assess these projects we defined a simple checklist, which the teams can fill in. This answers will be specific to their project needs and scope.
Key areas of focus in the assessment checklist:
- End users/Personas using the application
- Duration of project
- Team Knowledge on domain and technology
- Technical aspects involved in the project
- Type and quality of data handled in the application
- Non functional aspects of the application (Example: Performance, Security..etc)
- Generic quality related activities common in all projects
The assessment results are documented and used in further stages of the project.
Quality Strategy
As a result of assessment, a high level quality strategy is defined, covering the following aspects.
- The support needed from quality experts in the project
- Personas to be taken into account while testing
- Types of testing needed (APIs/Web/Native)
- Tools to check code quality, security vulnerabilities and other aspects
- Test Automation effort needed
- Data quality checks needed
- Non functional testing (Example: Performance, Security testing..etc) needed
- General testing effort needed
This quality strategy is a living document which evolves during the project life cycle.
Code Quality Tools
Based on the strategy defined, tools are used to monitor and validate code quality. Quality experts facilitate the setup and implementation of these tools.
Below are the tools we primarily use:
- ACR – Automatic code review tool for Mendix projects
- Snyk – Tool for scanning dependencies (3rd party libraries) in Mendix project for known vulnerabilities and possible licensing caveats
- Veracode – Tool for scanning Mendix code for identification of risks and vulnerabilities
Test Engineering
As part of the quality strategy the types and level of testing needed are also defined. With this input, the quality expert helps in setting up and driving various testing activities during the development cycle.
During story definition and development
Some activities performed during this phase are:
- Story refinement to validate testability of the story and add tester notes
- Reviewing unit tests developed by the team
- Reviewing and validating story acceptance criteria (some teams also manage tests in Jira X-Ray for reference)
- Making sure Definition of Done items are closed before moving stories to Done
Pre release validations
Some activities performed during this phase are:
- Validating all the features added for the release are working End to End
- Validating existing features are not impacted (Regression tests)
- Implement non-functional tests (this includes Performance, Penetration testing, etc.) as per the requirement of the project
- Making sure Definition of Done of the release is met.
Post release validations and production monitoring
The team sets up some tests in production environment, so that application can be validated post deployment. Validation tasks include:
- Validating new Features added for the release are working after deployment
- Sample smoke tests to validate that core features of the application are working as expected
Additionally monitoring can be setup to check:
- Application usage
- Application performance
- Application quality
- Application business value
Test Automation
Automation of testing is done based on the project’s need.
- For applications involving REST APIs, tests are automated while developing stories.
- For applications without APIs, core functional tests are automated at UI level. This is recommended only when the UI of the application is tested manually and is stable (no major changes planned).
All automation tests are executed in bitbucket pipelines.
The tools/framework we use for test automation are:
- Postman for API test automation
- Robot framework for web (UI level) test automation
Tracking progress
The progress of the quality activities are tracked and monitored in a dashboard.
Best Practices
Based on the experience gained by the team members from testing Mendix and other applications, best practices are defined and documented.
Some key areas mentioned in the document are
- Team mindset towards quality
- Different testing types
- Branching and testing strategy for Mendix applications
- Test Automation best practices
Continuous Improvement
We as a team believe in continuous improvement by performing retrospectives and deep-dive sessions. We have frequent sync ups with other departments inside Mendix to understand and exchange ideas on way of working and adapt what is best for AppFactory.
Challenges
The above implementation has its challenges as well. Here are just a few:
- The quality team member changing his mindset of being a quality expert instead of traditional test engineer
- Instilling a “quality mindset” in the team as a whole.
- Coming from a non low-code background, the quality experts needed to understand how Mendix applications are developed to provide better support to teams.
- Implementing test automation for teams which have no prior experience in testing or test automation.
Key takeaways
Some key takeaways from our experience so far:
- Start by understanding the team’s process and their current quality activities (every team has one)
- Product quality is a team responsibility
- Choose your tools depending on the teams expertise and way of working
- Have quality activities added in Definition of Done of stories and release processes
- Last but very important one is setting up quality needs patience and perseverance
For more details and questions please reach out to me: [email protected]