Mendix 9 Spotlight: Task Queue
Mendix 9 Spotlight: Task Queue by Tom Knippenberg
With Mendix 9, we introduced the Task Queue to allow you to asynchronously run microflows or Java actions in a distributed manner. This feature enables you to build high-performing stateless apps without thinking about synchronization and task management.
In this post, I’ll tell you more about the Task Queue and how it’s different from the deprecated Process Queue module. I’ve also asked Marcel Groeneweg, a Mendix expert and MVP from ITvisors to share some of his first experiences with the Task Queue with us.
What is the Task Queue?
The Task Queue is a horizontally scalable queuing solution to help you make your apps more reliable and perform better. You can perform long-running tasks in the background to prevent the user from having to wait, divide the load across multiple runtime instances, and optimally allocate resources by using multiple queues.
The Task Queue is typically used in cases where there is a high volume of transactions, tasks take a long time, or where reliability is key. All nodes can pick up queued tasks, effectively adding the option of horizontal scaling to the toolbox. Tasks picked up by a node that fails during execution will be picked up again, so you are less dependent on a single node succeeding. Since you can queue tasks from a scheduled event as well, you can work around a scheduled event’s dependency on a single node and ensure batches of tasks are always executed.
Awesome. How do I start?
Getting started with the Task Queue is simple to do. After adding a new Task Queue in Studio Pro, you can indicate on a Call microflow activity that the microflow should be executed in the background. After enqueueing the task, it will be picked up by one of your runtime instances for processing. Finally, it will be moved to the ProcessedQueueTask table with a success state, based on which you could implement further logic.
Consider a case where an administrator wants to generate PDF and Word documents from a list of bills in the database. The user triggers a microflow by pressing a button, which calls sub microflows in a loop to generate the files. Without the Task Queue, the page will show a loading bar until the action is completed, preventing the user from working with the app in the meantime.
To improve this, we can specify that the sub microflows should be executed through a Task Queue. Since all nodes can now help generate the documents, the user will get results faster, as well as not being blocked while the files are being generated in the background.
Comparison with the Process Queue
There are several differences between the Task Queue and the deprecated Process Queue module:
- The Task Queue is a native part of Mendix and therefore no longer requires you to set up additional entities and associations, lowering the complexity of your domain model.
- If you were thinking that was nice, the Task Queue also allows for horizontal scaling, making it the go-to solution for multi-instance environments.
- Coming down the pike — the Task Queue will support automatically retrying tasks, which is on the roadmap for later this year.
Experiences with the Task Queue
That’s all well in good, but how is Task Queue working in real-world scenarios. I spoke with Marcel Groeneweg, certified Mendix Expert at ITvisors and Mendix MVP, who has already worked with the Task Queue in Mendix 9.
Q: What have you done with the Task Queue?
A: We have replaced some long-running background processes that were previously executed from the Process Queue. These background processes gather data from external systems and process it. We’re also using a queue to download images. Each image is a separate task to prevent one failed download from breaking all others and to keep the transaction size small.
Q: How are you liking it so far?
A: Easy to use, really good to configure this in the model. No more configuration after deployment in configuration pages in the app itself.
Q: How was the migration from the Process Queue module?
A: Really easy. Just configure one or more queues, change the microflow parameters (remove the additional entity related to Process Queue) and replace the Process Queue logic in the calling microflow with a single microflow call activity.
Q: How did you prepare for this migration?
A: Impact could easily be determined by right-clicking the Process Queue module in the model to ‘Find usages’. For each background task, a cross-module association exists from your entity to ProcessQueue.QueuedAction and these need to be cleaned up. What remains are references to the ProcessQueue.Process entity for configuration and project security settings which can also be removed.
Q: Any suggestions for the future?
A: Cluster-wide limits (run only one task at a time) and having automatic retries with exponential retry intervals would be great. Next to that, it would be convenient if an identifier is returned when scheduling a task in a Java action so we can keep track of it. Finally, being able to set a delay or specific time for execution would be really powerful.
In this blog I covered the functionality of the Task Queue: a horizontally scalable solution to execute microflows asynchronously. Go try it out in Mendix 9 and make sure to share your experiences with us.