Transactions are currently intermittently available in the web and mobile applications. Swipes, payments, and support messages are all working. We’ll update here when we’ve resolved the issue. Thanks for your patience! (^ctb 12:40 PM PST)

Resolved! Transactions are available again. If you have any issues please reach out to Customer Relations from the web or mobile app. Thanks! (^ctb 12:52 PM PST)

We’re currently experiencing an issue that is affecting logins to our web and mobile apps, and is also causing our phone support line to be unavailable. You can still reach Customer Relations at support@simple.com. We’ll update as things develop. Thanks for your patience! (^ctb 20:21:50 PST)

Update: Logins to our web and mobile application are now succeeding and the phone support line is available again. We’re still debugging issues with our onboarding, so new customers cannot redeem invites at this time. We’ll update when we’ve resolved that issue. Thanks! (^will 21:05 PST)

Resolved: Onboarding is back on and invites can be redeemed now. If you encounter any issues, please contact Customer Relations at 888-248-0632 or support@simple.com. Thanks! (^ctb 21:41:55 PST)

Tonight, Friday, January 18th, 2013 from 8PM PST until Midnight PST, Photo Check Deposit will be intermittently available while our partner performs routine maintenance on its systems. If you have any questions please contact Customer Relations at 888-248-0632 or by signing in to the web or mobile apps and sending us a message. Thanks! (^ctb 2:40 PM PST)

Resolved: We are seeing several instances of successfully deposited checks using Photo Check Deposit. We’ve turned on the feature for customers again. If you are seeing any further issues, please contact Customer Relations at 888-248-0632 or use our Support Messaging feature on iOS or the web. Thanks! (^tom 6:18 PM PST)

As of approximately 3:27 PM PST today, we began seeing issues with Photo Check Deposit. We immediately contacted our partner and have been working with them to resolve the issue. As of 3:36 PM PST, we have turned off Photo Check Deposit for customers until the issue is resolved. We will update this post as we learn more. (^tom 4:20 PM PST)

On 12/31/2012, shortly after 4:00 PM PST (00:00 UTC), we started receiving alerts from our monitoring system that we were unable to reach our transaction processor (hereafter referred to as “processor”). The first alerts came in at 4:02 PM PST, indicating that servers on their end were unreachable. Customer Relations received a call at 4:05 PM PST about a declined transaction, and immediately notified engineering. We also observed that our ingest service, which consumes data from this partner, had stopped. At 4:09 PM PST we contacted our processor to escalate the issue. They confirmed that they were aware of the problem and taking action. We also notified our partner bank, The Bancorp Bank, though they were not affected.

The root cause of this outage was discovered to be a “leap second” added by an upstream NTP server (time.apple.com). It has since been removed from the pool of NTP servers that our processor uses. A similar leap second issue was reported widely in the news last summer. Leap seconds were introduced into timekeeping in 1972 and some technology companies have techniques for dealing with these changes — Google uses what they call the “leap smear” to spread-out the added second over a longer period of time. If leap seconds are added to a year, it’s usually only done in either June or December.

Around 4:15 PM PST these events were followed by reports that card swipes, ATM withdrawals, and our card activation line were also affected. Customer Relations staff were able to confirm these reports by testing transactions with Square and attempting withdrawals at nearby ATMs. We updated our status blog at 4:22 PM PST and continued to handle a large amount of customer contact via phone, email, support chat, and Twitter.

At 4:22 PM PST our processor began the process of a full shutdown and restart of the systems that handle Visa (credit) and Star (ATM/debit) transactions for Simple. At 4:29 PM PST we decided to pause signups for any new customers, since our ability to order new cards was affected.

Our processor began bringing their systems back online at 5:10 PM PST, followed by a restoration of all customer data. Restoration was completed at 6:15 PM PST, all services were re-enabled, and the first successful Visa transaction came in one minute after. Staff continued testing card swipes and ATM withdrawals and by 6:50 PM PST it appeared that most things were back to normal. At 6:20 PM PST we re-enabled signups.

Staff still observed some intermittent swipe failures with PIN transactions for a small number of customers. At 7:52 PM PST, our processor resolved all transaction issues by performing another restore of customer data, and we confirmed that no more customers were affected. We posted the all-clear on our status blog at 8:09 PM PST.

To all our customers who were affected by the outage, please accept our sincere apologies. We are making changes to our processing architecture that we expect will mitigate this in the future. If you enjoy solving hard problems and want to help build the future of banking, check out our job openings — we’d love to talk.

- Chris Brentano, operations engineer

Resolved! All services are restored and operating normally. Both we and our transaction partner are continuing to monitor the situation for any fallout but we’re confident that everything is back to normal. Again, if you have any questions, please contact Customer Relations at 888-248-0632 or support@simple.com. Thanks everyone for your patience and Happy New Year! (^ctb 20:09 12/31/2012)

Update: The activation line is back up, card swipes (purchases) are working, and sending payments is good, but we’re still seeing intermittent ATM issues. We’ve also re-enabled signups for new customers. We’re still watching everything but things are coming about. We’ll send an all clear once we’ve confirmed everything is back online! (^ctb 19:06, 12/31/2012)

Update: We’ve temporarily disabled new signups while we work with our partner on resolving the underlying issues. If you have any questions please contact Customer Relations at 888-248-0632 or support@simple.com. Thanks! (^ctb 16:39, 12/31/2012)

As of approximately 4:15 PM PST today, Simple card swipes are being declined, ATM access is unavailable, and the card activation line for new customers is unavailable. We’re working with our transaction partner to resolve this issue as soon as possible. We’ll update here as we learn more about the issue. Thanks for your patience! (^ctb 16:22, 12/31/2012)

We’re investigating an issue that’s causing a delay for transactions to show up in the Activity feed. Purchases (swipes) and other Simple features are working normally. We’ll post an update as soon as we know more. (^ctb, 06:40 PST)

Update 08:40 PST: We’re working with our partner to resolve this issue. Transactions continue to be delayed and attempts to call our card activation line may fail. Other services are operating normally. (^will)

Update 10:00 PST: This issue is resolved. Transactions that occurred during the outage will begin appearing in your Activity, while new transactions should show up immediately. Thanks for your patience! (^will)

We’re investigating issues with the phone number new customers call to activate their cards or update their card PIN. All other services are funtioning normally. (^Will, 11:20 PDT)

2012-09-13: We’re testing a fix for this problem. (^Will)
2012-09-14: Our tests are now succeeding and the issues affecting the activation line are resolved. Thanks for your patience! (^Will)

We’re investigating reports of failed card swipes. (^Will, 06:22 PDT)

06:45 PDT: Our tests are now passing and this issue is resolved. Thanks for your patience! (^Will)

We’re investigating delayed and failed payments scheduled by our customers. (^Will, 11:30 PDT 2012-08-27)

12:50 PDT 2012-08-28: Our test payments now succeed and this issue is resolved. Please send a support message if you have questions about your own payments that may have failed. Thanks for your patience! (^Will)