All systems, whether manual or automated, will experience problems over time. Our PolicyStat system is no exception. We hope, however, that we can positively differentiate our service by the way we respond to those problems. We will always strive to quickly identify a problem, quickly address that problem, and be transparent with our clients. We've completed our post-mortem of the incident last weekend and, as promised, have a follow-up report below.
On March 1, 2014, the monthly "Inactive Users With Responsibilities" email notification was sent to a much broader audience than intended. Some site administrators received multiple extra and incorrect notifications. The emails went out between 8:01pm and 8:49pm (EST), and all notifications were immediately stopped when the issue was discovered. After a thorough post-mortem and root cause analysis into the issue, our engineering team has determined the following:
- No customer data was at risk or exposed.
- Our existing layered, data safeguards worked correctly.
- The root cause of the problem was a software defect that was installed prior to the notifications being generated. This defect was fixed early morning March 2.
- Additional safeguards and checks against this type of issue have been put into place to prevent recurrence.
We appreciate your patience and understanding while we worked to analyze this issue. Additionally, if you have any questions or concerns regarding this event please don't hesitate to contact me.
Mark A. Boxberger – VP, Client Services
PolicyStat, LLC | 12800 N. Meridian St, Ste 125 | Carmel, IN 46032
e: firstname.lastname@example.org | w: policystat.com | t: 317.644.1296 x1100 | f: 317.723.6269