How it works: nginx and error pages

on December 26, 2014

Share:

In the week before Christmas, we released a nice little extension to our hosting app platform: It’s now possible to use customized error pages for an application. This blog post will show how web server configuration for applications is done at Mendix, and how this additional feature is implemented, on top of it.

If you’re only interested in how to use the custom error page feature, and not in how it’s built, read the custom error pages blog post on the corporate tech blog.


Green Monsters!

If you’re using or maintaining a Mendix application that runs in the Mendix hosting environment, you might have seen them occasionally… The green monsters, running around, eating computers and modems. While we often call them “the green monsters”, these animals are actually called TumblBeasts and were created by The Oatmeal as a gift for Tumblr to use in their error pages. Unfortunately, Tumblr stopped using them after a while, and the creator of them told whoever would want to could use the images.

Here’s the most famous page of a set of pages that we’ll discuss in a bit. It’s the page that is shown when a deployed application is stopped:

503-appnode-app


Let’s dive in…

Actually, four different green monsters pages currently exist, containing different text, shown in different occasions. In order to understand why, and what they mean, let’s have a look at a simplified view on the web server configuration we use to serve applications over HTTPs to the world:

graphic

The front facing web server listens on the actual public IP address where your application url points at, and directly handles all browser connections for multiple different applications. The front facing web server knows about every existing application url attached to running applications, and knows where they are running in the internal network behind it. The appnode web server is an nginx instance that is running on the application server where the actual Mendix Runtime process is started as well.


Front facing web server configuration

Let’s follow the arrows and see what kind of problematic situations we could encounter. In the previous paragraph, I mentioned that the front facing web server listens on IP addresses. These addresses are not a secret, they’re defined in DNS for each existing application url. As example, I’ll be using the url https://application.mendixcloud.com:

$ host application.mendixcloud.com
application.mendixcloud.com is an alias for mendixcloud.com.cname.xs4.mendix.net.
mendixcloud.com.cname.xs4.mendix.net has address 82.94.167.224
mendixcloud.com.cname.xs4.mendix.net has IPv6 address 2001:888:2177:1::e0

When trying to resolve the application url name to an IP address, DNS tells us that application.mendixcloud.com refers to mendixcloud.com.cname.xs4.mendix.net., which uses the addresses 82.94.167.224 for IPv4 connections and 2001:888:2177:1::e0, for IPv6 connections.

The actual nginx configuration which makes the front facing web server listen on those addresses looks like this:

server {
    listen 82.94.167.224:443 default_server ssl;
    listen [2001:888:2177:1::e0]:443 default_server ipv6only=on ssl;
    server_name _;
    root /usr/share/mendix-monsters/;
    ssl_certificate /etc/ssl/nginx/mendixcloud.com.crt;
    ssl_certificate_key /etc/ssl/nginx/mendixcloud.com.key;
    error_page 503 @503;
    location / {
        return 503;
    }
    location /mx_error_page/ {
    }
    location @503 {
        try_files /503-ff-catchall.html =503;
    }
}

Besides this default configuration, every configured application has its own server definition in the front facing web server configuration. Here’s the one for our example application:

upstream application {
    server 10.140.48.12;
    server [2001:888:2177:30::c];
    keepalive 8;
}
server {
    listen [2001:888:2177:1::e0]:443;
    listen 82.94.167.224:443;
    server_name application.mendixcloud.com;
    root /usr/share/mendix-monsters/;
    ssl_certificate /etc/ssl/nginx/mendixcloud.com.crt;
    ssl_certificate_key /etc/ssl/nginx/mendixcloud.com.key;
    add_header Strict-Transport-Security "max-age=31536000;";
    error_page 502 @502;
    location / {
        proxy_pass http://application;
        client_max_body_size 1024M;
    }
    location /mx_error_page/ {
    }
    location @502 {
        try_files /502-ff-app.html =502;
    }
}

The most important part of the information in here is the server_name line, in combination with the server lines in the upstream part, which tell the front facing web server where to forward requests for this specific url, in order to reach the running application. All requests for urls that do not have a specific server block like this one, will hit the default server above.


What could possibly go wrong?

But, hold on. I was talking about problematic situations and error pages. When looking a bit more closely at the two configuration snippets, we see all kinds of 503 and 502 numbers and there are even references to html pages.

The page mentioned in the first configuration section is the 503-ff-catchall, an error page shown when a request is made to the front facing web server for an url that is not known to the web server:

503-ff-catchall

To see this page, you have to visit a url that is not present in the web server configuration. A little trick to do so, is by just visiting one of the plain IP addresses, or the alias name mendixcloud.com.cname.xs4.mendix.net, because we don’t put these anywhere in a server_name line in the configuration. Another way is to point a fake url to one of the IP addresses using the local hosts file on your computer, and then trying to visit that url.

The text in the page reads: The application you were looking for is not configured in this location. – You reached a web server which has no configuration for the application you’re try to reach yet, or, does not have it any more. If you’re currently executing an application migration, this issue will likely solve itself in a few minutes. Try refreshing this page after some time. If you think your application should be running and reachable, this may be due to a bug or misconfiguration. Please contact Mendix Support in this case.

The second page mentioned is 502-ff-app. This is an error page that is shown for an existing application in case the corresponding application server is unreachable. This should not happen during normal operation, and means there is possibly a network outage, or unexpected server downtime. The page returns a HTTP status code of 502 Bad Gateway.

502-ff-app

This page tells you: The application you were looking for is currently unavailable. – The application server is currently unreachable. The server may be down, or there may be a network problem. Please try again later. If you are the administrator of this application, and you are not aware of any planned maintenance, please contact Mendix Support.


Intermezzo

The html pages and images used in the error pages that are shown, are put inside a little debian package, called mendix-monsters, to make it very easy to distribute changes to them, and to easily get them installed on all web servers and application servers as a dependency of another package, or using puppet. The try_files directive in the nginx configuration ignores the location / when looking for content, but images or other static content have to be placed inside the mx_error_page directory, which gets excluded from the forced 503 response on a request, as seen above.

$ dpkg -L mendix-monsters
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/mendix-monsters
/usr/share/doc/mendix-monsters/changelog.Debian.gz
/usr/share/doc/mendix-monsters/README.Debian
/usr/share/doc/mendix-monsters/copyright
/usr/share/mendix-monsters
/usr/share/mendix-monsters/503-appnode-app.html
/usr/share/mendix-monsters/502-ff-app.html
/usr/share/mendix-monsters/mx_error_page
/usr/share/mendix-monsters/mx_error_page/server.png
/usr/share/mendix-monsters/mx_error_page/running.png
/usr/share/mendix-monsters/mx_error_page/mendix.png
/usr/share/mendix-monsters/503-appnode-catchall.html
/usr/share/mendix-monsters/503-ff-catchall.html

Application server nginx configuration

When a request matches a known application url, it gets forwarded to the application server itself, where it’s greeted by a second nginx instance.

A running application

Here’s the nginx configuration that is used for a running application. This configuration has another upstream definition, that points to the actual running Mendix Runtime process. All static content gets served directory from the file system by nginx.

upstream application {
    server 127.0.0.1:8000;
    keepalive 8;
}
server {
    listen [::];
    listen 0.0.0.0;
    server_name application.mendixcloud.com;
    root /srv/cloud/slots/10001/deploy/web/;
    add_header X-Frame-Options "SAMEORIGIN";
    error_page 403 @403-custom;
    error_page 404 @404-custom;
    error_page 503 @503-custom;
    location = /debugger/ {
        proxy_pass http://application;
        client_max_body_size 1024M;
    }
    location = /file {
        proxy_pass http://application;
        client_max_body_size 1024M;
    }
    location /mx_error_page/ {
        root /usr/share/mendix-monsters/;
    }
    location /mxclientsystem/ {
        root /srv/cloud/slots/10001/deploy/runtimes/4.8.9/runtime;
    }
    location = /xas/ {
        proxy_pass http://application;
        client_max_body_size 1024M;
    }
    location @403 {
        root /usr/share/mendix-monsters/;
        try_files /403-appnode-app.html =403;
    }
    location @403-custom {
        try_files /error_page/403.html @403;
    }
    location @404 {
        root /usr/share/mendix-monsters/;
        try_files /404-appnode-app.html =404;
    }
    location @404-custom {
        try_files /error_page/404.html @404;
    }
    location @503 {
        root /usr/share/mendix-monsters/;
        try_files /503-appnode-app.html =503;
    }
    location @503-custom {
        try_files /error_page/offline.html @503;
    }
}

Well, the configuration your application is using can be quite different, depending on the choices that were made in the configuration screens in the deployment portal.

Anyway, the part I’d like to focus on are the error_page definitions. In case of a 403 Forbidden or 404 Not Found, first the 404.html or 403.html in the error_page location of the project will be considered. If this file is not present, nginx will see if there’s a default page in our mendix-monsters package. If not, the default nginx messages are shown. The 503 page, offline.html, does actually not make much sense when the application is running, but it certainly does after stopping the application.

By including an error_page folder in the project, containing these html files, it’s possible to show custom error pages for the Forbidden and Not Found responses. When working on an application using the Mendix Business Modeler, the error_page folder can be placed inside the ‘theme’ folder.

A stopped application

For a stopped application, the configuration looks similar, but different. Only the error page definitions remain, and there’s a big catchall that points to the 503 Unavailable page. Still, 404 and 403 custom error page configuration get included, and the error_page folder from the application contents remains accessible. When using an IP filter or client certificate verification, the customized Forbidden page should still show up when a client connection does not meet the requirements to be able to access the application.

server {
    listen [::];
    listen 0.0.0.0;
    server_name application.mendixcloud.com;
    root /srv/cloud/slots/10001/deploy/web/;
    add_header X-Frame-Options "SAMEORIGIN";
    error_page 403 @403-custom;
    error_page 404 @404-custom;
    error_page 503 @503-custom;
    location / {
        return 503;
    }
    location /error_page/ {
    }
    location /mx_error_page/ {
        root /usr/share/mendix-monsters/;
    }
    location @403 {
        root /usr/share/mendix-monsters/;
        try_files /403-appnode-app.html =403;
    }
    location @403-custom {
        try_files /error_page/403.html @403;
    }
    location @404 {
        root /usr/share/mendix-monsters/;
        try_files /404-appnode-app.html =404;
    }
    location @404-custom {
        try_files /error_page/404.html @404;
    }
    location @503 {
        root /usr/share/mendix-monsters/;
        try_files /503-appnode-app.html =503;
    }
    location @503-custom {
        try_files /error_page/offline.html @503;
    }
}

The default page that will be shown when stopping the application is likely the most well known green monsters page. This page again uses content from mx_error_page, provided by the mendix-monsters package, to display the default monsters page:

503-appnode-app

When the error_page folder is present in the project, containing an offline.html page, it will be shown instead of the monsters. Keep in mind that any extra content that is referenced from the page must be present in error_page as well, because it’s the only part of the static content included with the application that is still visible.

What if I clean my environment?

When using the ‘clear environment’ functionality from the deployment portal, all data and the deployed application are removed. The application specific web server configuration is removed as well.

Just like the front facing web server, the application server nginx configuration has a catch all definition that will catch all requests for the application as soon as the part containing the specific server_name is gone:

server {
    listen [::] default_server ipv6only=on;
    listen 0.0.0.0 default_server;
    server_name _;
    root /usr/share/mendix-monsters/;
    error_page 503 @503;
    location / {
        return 503;
    }
    location /mx_error_page/ {
    }
    location @503 {
        try_files /503-appnode-catchall.html =503;
    }
}

This is the fourth and last page that is currently present by default.

503-appnode-catchall

The text is: The application you were looking for is currently unavailable – It looks like this application location is not in use yet. Please try again later. If you are the administrator of this application, try to deploy an application from the Cloud Portal.

So…

Right now four pages are included in the default set of error pages, and it’s possible to replace the offline page, the Forbidden page and the Not Found page with content from your own application.

The configuration shown here, and the design of the default colored error pages are subject to change. This blog post reflects the configuration and monsters pages that were used at the time this blog post was written.

Subscribe to Our Blog

Receive Mendix platform tips, tricks, and other resources straight to your inbox every two weeks.

RSS Feed of the Mendix Blog

About Hans van Kranenburg

Hans is a senior sysops @ mendix. He loves Debian and extending the Intarwebz by building computer networks.