Configuring a Rock Web Farm Before getting started you’ll want to complete the following pre-steps: Web Farm License Key: Running a web farm requires a unique license key from Spark. This is necessary as the web farm feature has an additional cost to help support the feature without the need to use funding from donations. Reach out to Spark at Infrastructure Configuration: The configuration steps below assume you have all of the needed infrastructure configured and Rock installed and running. It’s outside of the scope of this document to provide step-by-step guidance on configuring the infrastructure, as these steps will be very different depending on your cloud hosting provider. Reach out to a Rock Partner (https://www.rockrms.com/partners) if you need help with this configuration. Below is a list of the high-level components that should be configured: App Gateway configured and working.Database server configured.Web application nodes configured and running.Rock installed on each web application node. Be sure that Rock jobs is only installed to run on a single node of your cluster. This is set in the server’s web.config file. With these steps complete, you’re now ready to configure the web farm. Below are the steps you’ll need to follow. 1. Configure the Message Bus If you haven’t yet configured a message bus in Rock, you’ll need to do that before proceeding. See the Admin Hero Guide for more information on configuring the message bus. 2. Activate the Web Farm Navigate to Admin Tools > System Settings > Web Farm to enable the web farm feature. This is where you will activate the web farm and enter your unique license key. The remaining settings can be left as is. The additional configuration values are discussed below in the section on how nodes elect leaders. 3. Restart All Web Nodes Once the configuration is set, you ’ll need to restart all the nodes in the farm. Once restarted, the web farm is fully configured. You’ll want to ensure that each node is displayed on the Web Farm page. The Secret Life of Web Farm Nodes Behind the scenes, the nodes of a web farm communicate with each other over the message bus to keep each other in the loop on what’s happening inside the cluster. These messages are related to two primary types: Cache Invalidation: When a node changes a value that lives inside of the Rock cache, it must notify the other nodes in the farm to also update their cache.Node Status: As the nodes start-up, they elect a leader of the cluster. One of the roles of the leader is to check-in with each node on a routine basis to ensure it is still up and running. Any discrepancies the leader finds will be reported in the web farm log. Node Start-up / Shutdown As nodes start-up, they report their status to the web farm log and mark their status as active. They also report when they shut down and change their status to inactive. This notifies the leader which nodes are expected to be running. Leader Responsibilities As noted above, one of the responsibilities of the node leader is to check the status of every node marked as being active. This status check is run on a configured polling schedule. On each run, the leader sends every active node a ping. If a node does not respond within a few seconds with a pong response, the leader reports the node as having been marked active but no longer responding. Leader Election You may be wondering how a leader is elected by the cluster. The election process is a sophisticated algorithm that allows the nodes to jointly elect a leader as well as trigger a re-election event, should the leader go AWOL. Let’s take a closer look at how the election process works. As a node starts-up, it randomly selects a polling interval to use. The selection of the random interval is guided by the configuration of the web farm. The configuration settings tell the node the minimum and maximum interval value. Using these values as boundaries, the node randomly selects a value somewhere between. It then checks the database to see if another node already has a similar interval (what constitutes a similar interval is also configurable). If a node is found with a similar interval it tries another random interval until an acceptable value is generated. This interval is then stored in the database for that specific node. The leader of the cluster is the node with the smallest interval, meaning it will be the first to send out a request to other nodes for a status check. As each other node receives a status check request, it also notes that it has been contacted by the leader and therefore should reset its timer to ping other nodes. If the leader goes offline, the node with the next lowest interval will reach its status check interval and assume the leadership role. These changes of leadership are noted in the web farm log. When a leader node comes back online, their smaller interval length will kick back in, and it will regain leadership within a status cycle or two. The current leader of a cluster is noted by a small icon in the upper right of the node card. Leader Node In the above scenario, the leader node is randomly selected. If you prefer, you can specify a particular node as leader by setting the interval value on the node’s WebFarmNode record in the database. Setting the ConfiguredLeadershipPollingIntervalSeconds property disables the random value, which currently must be updated in SQL. Placing a value lower than the minimum will ensure that the node is always the leader when running. If that node goes offline, another node will still step in until it comes back online. Node Names By default, the name of the node will be read from the service's machine name. If you prefer to use a different name, you can provide it in the web.config file with the app setting key of “NodeName”. Node Metrics You’ll notice that basic CPU metrics are provided for each node in the cluster. This provides a quick way to confirm that the load is being roughly balanced across your nodes. If you find that these charts vary greatly across your nodes, it may mean that something is not correctly configured in your application gateway. Current Limitations Rock’s Web Farm features will continue to grow as more time and resources become available to fund resources. Below is a list of known limitations. Page Routes: Changes and additions of page routes currently are not communicated to other nodes in the web farm. These are updated nightly when the AppPool restarts. If you need them sooner, you’ll need to restart each node in the cluster.Job Runner: The current feature set will not prevent two servers from running jobs.