Intermittent disruption to web, mobile

Our monitoring alerted engineers to intermittent failure to load our web or mobile applications. Financial operations are unaffected, so swipes and deposits can be performed as usual. We’re investigating now and will update here as we make progress. Thank you!

0951 PT: We have found and resolved the cause of the intermittent failures to load the web application. We continue to debug elevated latencies between internal APIs but expect no further customer impact.

Ongoing account issues following maintenance

We’ve received reports that a small number of customers may be experiencing the following issues stemming from our recent scheduled downtime. We’ll continue updating below as we make progress. If you need support with something urgent that’s not listed here, please send our customer relations team a support message. Thank you!

Update 2014-8-28:

  • We have removed any transactions in your Activity that were displayed more than once, in error. Only the original transaction is now displayed.

Update 2014-8-25:

  • We have successfully processed the delayed transactions that were authorized prior to the 6th. If any of your transactions were affected, they are now displaying properly on your transaction list with the date the transaction was authorized.
  • We continue to work on payment display issues. We’ve observed that some payments disappear from the ‘Scheduled Activities’ section of the app during processing. They will reappear in the ‘Activities’ section when they are completely processed, which may take several hours.
  • We’re investigating card declines due to balance discrepancies caused by duplicated or delayed transactions. If your card is unexpectedly declined, please send us a message through your account so we can investigate. If your situation is urgent, please give us a call at 1-888-248-0632.

Update 2014-8-22:

  • We have applied a fix for the payment processing issues that we were experiencing, and payments will be sent successfully. Some payments that were previously sent electronically will temporarily be sent as a paper check.
  • We continue to investigate payment display issues.

Update 2014-8-21:

  • We continue to work to address issues with payment processing and display. We recommend making important, time-sensitive payments with your Simple card or via auto-debit with your account and routing number.

Update 2014-08-20:

  • We’re investigating issues with Square Cash deposits. If you’re expecting a credit via Square Cash and it is displaying improperly or it seems to be delayed, please contact our customer relations team by sending a message through your account.

Update 2014-08-19:

  • Our team is working to address issues with payment processing and display. Known issues include the fact that Payments scheduled to be sent more than 30 days in the future may not be displayed. Please monitor your payments, and if you see anything that concerns you, contact our customer relations team by sending a message through your account.
  • We continue to work on processing delayed transactions that were authorized prior to August 6th, and on improving the way duplicated transactions and their reversals are displayed.

Update 2014-08-15:

  • Many of the erroneously duplicated transactions have been reversed, and we’re working to reverse those that remain. In some cases, the duplicated debit may not be displayed, but the reversal credit is. This is a display error that we’re working to resolve.
  • Some transactions that occurred on August 6th were subject to a delay in processing. These charges are being processed now. The pending pre-authorizations for these charges may have fallen off, so the completed charge will have a different date than the original card swipe.

Update 2014-08-14:

  • Check holds: We’ve observed issues with the way check holds are displayed. When a check hold expires, the pending hold will disappear and the funds from the check will become available, but your balance may not appear to increase. Swiping your card will cause your balance to update and display the check funds.
  • Duplicated transactions: New transactions will not be duplicated. We continue to work to remove previously duplicated transactions from accounts.
  • Fluctuating balances and Safe-to-Spend : The situation is stable since the previous update.

Update 2014-08-13:

  • Duplicate transactions: Some customers continue to report duplicate transactions. We’re working to remove these from accounts.
  • Fluctuating balances and Safe-to-Spend: Following the last week’s scheduled downtime, some transactions were subject to a delay in processing. We’re processing these transactions now, and as they appear, balances and Safe-to-Spend will adjust to reflect these transactions.

Update 2014-08-11:

  • Duplicated Transactions: Your balance is not affected by this display error and duplicated debits have been automatically reversed. Our team processed two sets of transaction data for some purchases during last week’s scheduled downtime, causing some transactions to display twice. We’ve identified the source of the duplicate data and applied a fix, so future transactions should come through normally.
  • Holds and Available Balance: Pending transactions that were processed shortly before or after our maintenance window may be taking longer than usual to clear. Remember, Pending transactions are automatically deducted from your Safe-to-Spend. This issue will resolve itself as transactions clear.
  • Payment Goals: Some Payments may have been dissociated with their respective Goals and deducted from your Safe-to-Spend. To troubleshoot, delete and reschedule the payment. Or you can transfer funds from the associated Goal back to your Safe-to-Spend.

Onboarding disabled

We disabled our Onboarding service at the beginning of the scheduled maintenance earlier this week. We plan to keep it disabled while we perform ongoing maintenance to other services. While Onboarding is disabled, new customers will not be able to open Simple accounts. All submissions via simple.com will instead receive a link to our application form as soon as we begin accepting new customers.

We’ll update here as we make progress re-enabling Onboarding.

Ongoing maintenance

We continue to work on issues stemming from our recent scheduled maintenance.

1350 PM PT: Engineers have applied a fix for the interruptions seen earlier today on the web application and mobile APIs.

1150 AM PT: Engineers are continuing their work to fix the intermittent issues we saw earlier. We are monitoring the web application and mobile APIs and they have remained stable. We will provide an update in 1 hour unless we have new information to share.

The Onboarding service (new signups) remains temporarily disabled. In the meantime, emails submitted via simple.com will receive link to our application as soon as we begin accepting new customers.

1030 AM PT: Engineers continue to work on a fix to resolve the intermittent issues with the web application and mobile apis. Our next update will come in 1 hour unless we have new information to provide.

The Onboarding service (new signups) remains temporarily disabled. In the meantime, emails submitted via simple.com will receive link to our application as soon as we begin accepting new customers.

0930 AM PT: Both the web application and mobile APIs have remained stable since the last update. We continue to test our fix.

0900 AM PT: Our engineers have isolated the issue affecting the web application and mobile APIs; we’re working on a fix.

0830 AM PT: Our engineers are investigating intermittent issues with our web application and mobile APIs. We’ll update here as we learn more.

Ongoing maintenance

Our team continues to work around the clock on issues stemming from the recent extended scheduled maintenance (<http://status.simple.com/2014/08/04/payments-upgrades/http://status.simple.com/2014/08/05/scheduled-maintenance/). We will continue updating here as we make progress on these issues.

4:30 PM PT: We have made further progress on the check holds. We’ve also deployed additional resources to sustain load that proved problematic yesterday; we’ve confirmed that these new resources are handling load well today.

12:30 PM PT: Our engineers have fixed the issue that caused some PIN swipes to be declined. We continue to work on items around check holds.

11:30 AM PT: Our engineers are investigating reports that check holds are not being released. If you need access to your check funds immediately, please reach out to our Customer Relations team at 888-248-0632. We are also tracking reports that some card swipes requiring a PIN are being declined. Customers report that resetting the PIN via our mobile or web app allows them to use their cards for PIN purchases.

Scheduled maintenance

11:15 PM PDT 8/6 Scheduled maintenance is complete and all customer cards should be functional now. Detailed timeline below.

10:48 PM PDT We’ve begun upgrading our systems. In order to to complete all scheduled maintenance, we will be taking some of our services down for a few hours.

Please see below for a breakdown of features and functionality during this time.

  • Customers’ Simple Visa® Cards will not function on August 6th from 2:00 – 9:00 am EDT (11:00 pm – 6:00 am PDT).
  • During the above hours, External Accounts and Instant features will also be disabled. Scheduled transfers will be processed, but customers will be unable to create new transfers.
  • Payments will be disabled from 3:00 am EDT on August 5th (12:00 am PDT) until 9:00 am EDT (6:00 am PDT) August 6th. Payments scheduled to be sent on these dates will be processed normally, but customers will be unable to schedule new payments or edit existing payments.

We will continue to provide updates on our progress here.

23:59 AM PDT We continue to upgrade our systems and are currently on target with our maintenance schedule.

00:48 AM PDT Our engineering team’s work on scheduled maintenance is on track.

01:41 AM PDT Our systems upgrades are on schedule and work is proceeding according to plan.

02:47 AM PDT We’re continuing to upgrade our systems as planned and have begun testing some new services.

03:56 AM PDT We are testing our new services throughout the early morning hours as our engineering team continues their work on the upgrades.

05:02 AM PDT Our engineering team is continuing work this morning on the planned upgrades. We will continue to update here throughout the scheduled downtime.

05:57 AM PDT Our updates to card activity have been completed. Card transactions can now be completed successfully.

External accounts, payments, and Instant transfers are still undergoing maintenance and are unavailable at this time.

07:08 AM PDT We continue to update our infrastructure. During this time, some customers may experience card declines. External Accounts, Payments, and Instant remain disabled. Please send a support message if you have any questions. We will share an update once we have the all clear.

10:30 AM PDT Our operations team is investigating intermittent issues with our mobile API and web application. We’ll update here as we make progress.

10:45 AM PDT We’ve nearly completed our scheduled maintenance, but a small number of customers will be unable to use their cards until the process is complete at 10:00 pm PDT tonight, Wednesday, August 6th.

Payments are now available and our engineering team continues their work to bring External Accounts and Instant back online. Our operations team is also investigating intermittent failures to load our mobile and web applications. Follow along here for updates.

11:00 AM PDT We have identified and resolved the cause of the intermittent issues loading our web and mobile applications. Instant is now available and our engineers continue to work to enable External Accounts.

11:30 AM PDT Our engineers continue to work on External Accounts. All deposits have been received and processed as normal but will not be visible in Activity until External Accounts is back online.

1:15 PM PDT We have restored the External Accounts service. We continue to work on issues related to the maintenance, but core functionality is restored for most customers. We’ll continue updating as we make progress on the remaining issues.

7:24 PM PDT We experienced a short frontend service outage but have fully recovered. Engineers are still investigating the root cause but believe this to be unrelated to the scheduled maintenance.

10:20 PM PDT We are seeing successful swipes for a many of the affected customers and are running maintenance on the remaining accounts. We expect the remaining affected accounts to be coming online gradually over the course of the night.

11:15 PM PDT We’ve completed running maintenance on affected accounts. All customers whose cards were seeing declines should be functional now.

Payments upgrades

We’re about to begin a scheduled maintenance window to upgrade our payments service. The Payments feature will be disabled from 0300 EDT (0000 PDT) August 5th until 0900 EDT (0600 PDT) on August 6th. Payments scheduled to be sent on these dates will be processed normally, but customers will be unable to schedule new payments or edit existing payments. In addition, the Onboarding service will be disabled. Other Simple services will continue functioning normally through August 5th; please note that a subsequent maintenance window is scheduled to begin at 0200 EDT August 6th (11PM PDT August 5th).

During this Payments maintenance window, our engineers will update here periodically until service is restored.

00:15 AM PT: We have now disabled the Payments and Onboarding services and are beginning the upgrade process.

00:22 AM PT: Engineers are debugging an unrelated error in the web application that is preventing Activity from loading. We expect to have a fix shortly.

01:00 AM PT: Engineers have deployed a fix to the web application; Activity now loads as expected. We continue to make progress on the Payments upgrades.

02:30 AM PT: Engineers continue to make progress on the Payments upgrade.

10:00 AM PT: We successfully completed the first phase of the Payments upgrade. The service will remain unavailable throughout the day while we work with our partner to update their systems. We’ll continue updating here, though less frequently, until we enter the next phase of this upgrade.

5:30 PM PT: Our partner confirms that their systems are now updated. We’ve begun validating the upgrade and will continue to do so through the evening.

11:00 PM PT: We have begun the final phase of this upgrade. You can follow our progress in a separate post.

This work was completed in the following maintenance linked above.