Quip Disaster Recovery Policy: Crisis Response, Backups, and Recovery Testing

This article explains how Quip responds to production and security incidents, how customer data is backed up, how the disaster recovery system is structured, and how these procedures are tested and reviewed on an ongoing basis.

Crisis Response

Uptime and recovering from outages is critical to Quip's success. Customers expect the highest availability possible, and Quip's policies and actions as a company are oriented around ensuring the highest possible uptime and availability for customer data.

In the case of a serious production or security issue, Quip follows internal on-call escalation procedures with a target response time of 15 minutes.

For any information security issues, in addition to escalating to the operations person on call, the issue is immediately escalated to the CEO or Head of Engineering.

After any serious security or production issue, the DevOps team conducts a post-mortem of the issue, which the on-call and engineering staff review. Results of the review are shared with the customer in question if the problem relates to customer data.

See the Information Security Policy for more details.

Restoring Customer Data

Quip maintains multiple, off-site backups of all user data. This includes both incremental backups (no more than 5 minutes old) and daily snapshots. For example, if a customer accidentally deletes a large batch of documents, Quip can restore from a recent incremental backup rather than waiting for the next daily snapshot.

See the Backup Policy for more information on backup policies and the availability of customer data.

The entire site can be restored from serving from any backup, whether incremental or snapshot.

Disaster Recovery

To handle serving outages and disasters, Quip has implemented a multi-tier system:

1. All frontends on the system have multiple replicas, so no single frontend can cause the site to be unavailable.
2. All databases are sharded to multiple instances, so no single customer or data issue can cause the site to be unavailable to all customers (isolation).
3. All significant reads on the system are transactionally-consistent failover reads to secondary databases held in another datacenter, meaning a short-term serving issue in a single datacenter does not affect access to data.
4. All databases are Multi-AZ RDS databases, which automatically fail over in the case of regional unavailability or outage.
5. In the case of a failure of RDS across multiple availability zones, rendering the RDS service unavailable, Quip manually fails over to its own secondary replica servers, which are independent instances and can withstand a complete outage of the normal RDS system.
6. In the case of a catastrophic failure of RDS and Quip's own independent secondary instances, Quip restores from a snapshot or incremental backup to a new series of database servers.

Steps 1 through 5 can typically be accomplished in less than 10 minutes. In the case of a failure of all 5 layers, the final step 6 can be accomplished in approximately 1 hour.

Ongoing Recovery Testing

To ensure that recovery systems are maintained and work well, Quip's Operations team tests the failover steps listed above quarterly.

To further test these systems, Quip maintains a separate internal set of servers that run in "disaster recovery" mode. These internal testing servers simulate an outage of the primary database. Quip tests these servers regularly (weekly to monthly) to confirm the site can withstand a primary database outage while still serving all user data.

Review of Procedures

All disaster recovery systems and procedures are reviewed by Quip Operations employees quarterly.

All disaster recovery policies are reviewed and updated by the Head of Engineering each quarter.