Data Export for Reporting and Analytics
How can I export my Mendix application data to a report database?
Mendix applications follow a micro services architecture where the application is responsible for storing, securing and validation of the application data. This means that in the Mendix Cloud other software doesn’t have direct access to the database of the application. Reporting and analytics tools need to either fetch the required data through APIs of the app, or the data needs to be exported to a dedicated reporting database, data warehouse or datalake where it can be accessed from reporting tools.
Using a dedicated reporting datastore is usually the best approach as it allows you to report on data from multiple applications. Additionally, it reduces the impact on the application itself, as complex reporting queries are not affecting queries of the application itself.
Two common ways to export mendix application data to a data warehouse or data lake are:
- Every application provides APIs that can be used to pull data from an ETL tool into a data warehouse.
- Use entity event handlers to push data to a central place as soon as the data has changed.
How can an ETL tool be used to pull application data for a data warehouse or data lake?
You can provide APIs where an ETL tool can fetch the data required. One common way to implement this is using Odata APIs which provide an easy way to have flexible data APIs on top of your data models. Using View Entities you can optimize the data retrieval ETL use cases. Every entity can include a change data which can be used to implement recurrent data fetching by an ETL tool to avoid exporting the same large datasets over and over again.
How can entity event handlers be used to push application data to a Data Warehouse or Data Lake?
You can define events handlers on all or selected persistent entities in your domain models. In these event handlers you can provide custom logic, for example, to push the data to a REST endpoint or external database. One example where this approach is used is the Advanced Audit Trail module, which sends all data events to a central full text search database for reporting.