Zoomed Image
USA
800 691 9120
UK
01225 704844

Failover

xAssets Configuration Guide
Deployment and Administration

Failover

This page explains the xAssets server-side failover capability, which maintains a near-real-time standby copy of your xAssets databases on a separate server. If the primary server becomes unavailable, users can be redirected to the failover server with minimal data loss.

How Failover Works

The failover system uses the built-in backup system to continuously ship database and file backups from the primary server to one or more failover servers. The failover server automatically restores these backups, maintaining a read-only standby copy of each database.

Architecture

Primary Server (e.g. BA)              Failover Server (e.g. BB)
┌──────────────────────┐               ┌──────────────────────┐
│ Batch Service        │               │ Batch Service        │
│  └─ Backup (every    │               │  └─ Restore (every   │
│     15 minutes)      │── HTTPS ──>   │     15 minutes)      │
│                      │  (chunked     │                      │
│ SQL Server (R/W)     │   upload)     │ SQL Server (STANDBY) │
│ CustomerData/        │               │ CustomerData/        │
│ CustomerDataPrivate/ │               │ CustomerDataPrivate/ │
└──────────────────────┘               └──────────────────────┘

On each backup cycle (typically every 15 minutes), the primary server:

  1. Creates a database backup (full daily, transaction log every 15 minutes).
  2. Creates file backups of the CustomerData and CustomerDataPrivate folders (full or incremental).
  3. Ships the backup files to each configured failover server via HTTPS.

The failover server, on its own 15-minute cycle:

  1. Detects new backup files.
  2. Restores the database backup (full or incremental log restore).
  3. Extracts the file backups to the appropriate folders.
  4. Patches the database connection settings to point to the local SQL Server.
  5. Restores SQL logins so authentication works on the failover server.

Recovery Objectives

Metric Target
Recovery Point Objective (RPO) ~15 minutes (one backup cycle)
Recovery Time Objective (RTO) ~15-30 minutes (manual failover activation)

Prerequisites

Before setting up failover, ensure:

  • A second Windows Server with IIS, SQL Server, and the xAssets application installed.
  • The failover server can reach the primary server via HTTPS (for the primary to check failover status) and the primary can reach the failover server via HTTPS (to ship backups).
  • SQL Server on the primary uses the Full recovery model (required for transaction log backups).
  • The p_get_userlogins stored procedure exists in the master database on the primary server (used to script SQL logins into the backup for replication to the failover server).

Setting Up the Primary Server

Step 1: Configure Special Options

Set these Special Options in the management database on the primary server:

Special Option Value Description
BackupFailoverServers e.g. bb Comma-separated list of failover target names. Each name is used to look up a credential pack. For multiple targets, separate with commas: bb,cmdb3.
BackupRetainDays 2 Days to retain local backup files (default: 2)

Step 2: Create Credential Packs

For each failover target listed in BackupFailoverServers, create a credential pack. The pack name must be BackupTarget{name} where {name} matches the entry in the Special Option.

For example, for a target named bb:

  1. Create a credential pack named BackupTargetbb.
  2. Set the URL to the failover server's xAssets URL (e.g. https://bb.hosted.xassets.net:44301).
  3. Set the API Token to the API key configured on the failover server.
  4. Set the Secret to the matching API secret.

Step 3: Schedule the Backup

  1. Create an AMSX transformation containing the single command Backup.
  2. Schedule this transformation to run every 15 minutes via Admin > Batch > Schedules.

The backup system handles everything automatically: database backups, file backups, shipping to failover targets, and cleanup of old files.

Setting Up the Failover Server

Step 1: Install xAssets

Install the xAssets application on the failover server as you would for a normal installation. The failover server needs its own IIS site, SQL Server instance, and management database.

Step 2: Configure Special Options

Set these Special Options on the failover server's management database:

Special Option Value Description
BackupRetainDays 1 Shorter retention on failover (saves disk space)

Step 3: Schedule the Restore

  1. Create an AMSX transformation containing the single command Restore.
  2. Schedule this transformation to run every 15 minutes, offset by approximately 5 minutes from the primary's backup schedule (to allow time for the backup to be shipped before the restore runs).

To restore only a specific database instead of all databases: use Restore 'databasename'.

Step 4: Prepare the SQL Data Folder

Ensure the folder c:\sqldata\ exists on the failover server. Restored databases are placed in this location by default.

Step 5: Ensure the Backups Folder Exists

The backups folder (determined by the management database's DataConnection configuration) must exist on the failover server. The IIS application pool identity must have read/write access to this folder.

DNS Configuration

Failover servers use a DNS naming convention that allows the application to automatically determine which database to serve. The convention is:

{databasename}-failover.yourdomain.com → Failover server IP
{databasename}.yourdomain.com          → Primary server IP

For example:

  • jiratest3.hosted.xassets.net points to the primary server.
  • jiratest3-failover.hosted.xassets.net points to the failover server.

The xAssets application automatically strips the -failover suffix from the DNS hostname when matching it to a database name. This means the same database (jiratest3) is served from the failover server when accessed via the -failover URL, without any application configuration changes.

The -failover suffix is stripped in three places in the application code:

  1. DNS-to-database mapping -- determines which database to connect to based on the hostname.
  2. Database list matching -- matches the incoming hostname against available databases.
  3. Logon flow -- resolves authentication token paths correctly on the failover server.

Testing Failover

  1. Verify backup shipping -- on the primary server, check the backup log ({db}_backup.log in the backups folder) to confirm backups are being shipped successfully.
  2. Verify restore -- on the failover server, check the restore log ({db}_restore.log) to confirm restores are completing without errors.
  3. Access the failover URL -- open https://{dbname}-failover.yourdomain.com in a browser and confirm you can log in and see recent data.
  4. Check data freshness -- compare a recently modified record on the primary with the same record on the failover to confirm the standby copy is current (within the 15-minute RPO).

Self-Healing Log Chain Breaks

If the backup chain is broken for any reason (server restart during a backup cycle, network interruption, disk full), the system recovers automatically:

  1. The failover server detects the broken chain when a log backup fails to apply (LSN mismatch).
  2. It creates a flag file requesting a fresh full backup from the primary.
  3. On the primary's next backup cycle, it detects the request and forces a full backup.
  4. The full backup is shipped and restored, re-establishing a valid chain.
  5. Subsequent cycles resume normal incremental log shipping.

This entire recovery process happens without manual intervention, typically within 2-3 backup cycles (30-45 minutes).

Monitoring

Log Files

Both sides maintain activity logs in the backups folder:

File Server Contents
{db}_backup.log Primary Backup creation, file shipping, errors
{db}_restore.log Failover Full/incremental restores, log chain breaks, errors

Marker Files

The failover server tracks restore progress with marker files:

File Purpose
{db}_last_restored.txt Last restored database backup (set and sequence number)
{db}_last_customerdata_restored.txt Last restored CustomerData zip
{db}_last_customerdataprivate_restored.txt Last restored CustomerDataPrivate zip
{db}_needs_full.flag Failover is requesting a fresh full backup from the primary

Tip: If the failover server appears out of date, check the marker files and restore log first. Common causes are network timeouts during shipping (check the primary's backup log) or disk space issues on the failover server.

Activating Failover

When the primary server is unavailable and you need to switch users to the failover server:

  1. Update DNS records to point the primary hostname to the failover server's IP address, or instruct users to access the -failover URL directly.
  2. If the failover database is in STANDBY mode, bring it online: 
    RESTORE DATABASE [databasename] WITH RECOVERY;
    This converts the database from read-only standby to fully read-write.
  3. Stop the restore batch job on the failover server (it is no longer needed once the database is in read-write mode).

Warning: Once you bring a standby database online with RESTORE WITH RECOVERY, you cannot apply further log backups from the primary. This is a one-way operation. To return to failover mode after the primary is recovered, you must re-establish the backup chain from scratch.