Understanding timeouts
From All n One's bxp software Wixi
Contents
1 Overview
HTTP and HTTPS is a "best effort" delivery format, which means that sometimes the message doesn't get through. Just like the postal service, sometimes the post can get lost in the system. https://en.wikipedia.org/wiki/Best-effort_delivery
The Internet protocol offers a best-effort service of delivering datagrams between hosts. Those may be lost, arbitrarily delayed, corrupted, or duplicated.
This is the nature of the Internet, rather than anything the bxp can control. Instead, bxp attempts to put as many resources in place to reduce the chances of messages failing to delivery. But perfection isn't possible.
If there is a timeout that is recurring frequently it is important to understand the where the timeout is occurring. This article, helps with visuals to understand the troubleshooting of timeouts.
2 Primary elements
3 Troubleshooting sequence
3.1 Basic sequence
- Open Browser
- DNS resolution
- Request to Internal Proxy
- Request to firewall
- Request to Load Balancer
- Request to Web Server
- Request to Database Server
3.2 DNS resolution
- In a browser, you go to your system we will use client_demo, for all these examples.
In the background, the first thing the browser has to do is to do a DNS lookup on the name ww3.allnone.ie
You can repeat this by opening a command window and typing "ping ww3.allnone.ie"
Windows Key and the R key. In this window type cmd and you get the following.
ww3.allnone.ie will always resolve to 82.195.138.166
3.3 Internal Proxy
A proxy is a device which allows security to be applied to your company's internet access. It can also be used in certain instances to speed up access by "caching" material for the entire organisation. Examples of caches would be Squid proxy cache or Wingate.
There are software and hardware versions of this technology. If the cache isn't correctly configured or the internet access into the proxy is not working properly, timeouts can happen.
You can get custom messages from these devices which give you a clue as to where the challenge lies. usually in the source of the page or on the page itself it will say which cache tool is in use.
3.4 bxp Load Balancer
In order to maintain high availability bxp uses a software based load balancing solution to swap between web servers in the event one web server becomes unavailable.
If both web servers both become unavailable for any reason the message will appear as follows
This is an extremely rare occurrence due to the solution being high availability.
There is also a timeout on the load balancer that say any query longer than 30 minutes, kill and report that the operation has failed.
3.5 Web Server
So if all the previous elements are working correctly and the request gets to the Web server, then the web server software is what will need to process the request. If the request is complicated or the web server software, for some reason crashes, then the request will not be fulfilled in time. The timeout period on the web server for requests is 1 hour. i.e. if you can't get the work done in 1 hour to serve back a web page, kill the page and all associated processing and return all allocated memory.
However the load balancer will show it's message and time out at the 30 minute mark, using current configurations. So it is impossible to see an error message for this timeout. Instead you'll get the load balancer timeout message. The web server will continue to execute up to 1 hour, even if the load balancer has timed out.
3.6 Database server
If the request gets through to the Db server, but there is a lot of data to process or the Db server is running very very slowly, then the time out is 1 day for Db requests. Again these can never be seen, as the load balancer will time out long before the Db queries time out.
Instead we use a MySQL reporting tool called the slowlog to figure out slowly running queries and find better ways to implement these queries, their through the use or indexes or the application of new code.
4 Client side timeouts
If a client elects to use the "Force Inactive Logout" engine available in bxp Security_-_Force_inactive_logout this can add another layer of complexity to troubleshooting.
This engine does not interact with bxp servers in any way. A timer is started on the page and the user is required to log some interaction with the page, or else, the log out button is automatically pressed on the users behalf in the browser.
This process does not stop execution on the web or database servers. Instead it just performs a log out function.
The primary issue seen when using the Force Intactive Logout engine is multiple tabs. User opens tab 1. user opens tab 2 and works away in tab 2. Tab 1 after X minutes times out and logs out. User works away in tab 2 and is automatically logged out with no warning. The reason is that the warning did appear but in tab 1 and as it wasn't clicked, has logged the account out. For this reason it is highly recommended not to use multiple tabs when using the Force Inactive Logut engine.
5 Long reports
Long reports often cause the greatest challenge with timeouts. Large volumes of data can sometimes take many minutes if not hours to complete. The All n One bxp development team use 5 minutes as a mark of "long". Anything greater than 5 minutes will come with a "schedule-able" alternate solution. e.g. scheduled reports can be placed directly into the Report Library module Report_Library_-_Scheduled_Reporting
2017-03, update, there is a project being implement to change all operations, which take greater than 2 minutes to complete, to be transferred seamlessly to a scheduling server. Updates on this project will be made available as and when the engine is implemented.
