Payments upgrades

We’re about to begin a scheduled maintenance window to upgrade our payments service. The Payments feature will be disabled from 0300 EDT (0000 PDT) August 5th until 0900 EDT (0600 PDT) on August 6th. Payments scheduled to be sent on these dates will be processed normally, but customers will be unable to schedule new payments or edit existing payments. In addition, the Onboarding service will be disabled. Other Simple services will continue functioning normally through August 5th; please note that a subsequent maintenance window is scheduled to begin at 0200 EDT August 6th (11PM PDT August 5th).

During this Payments maintenance window, our engineers will update here periodically until service is restored.

00:15 AM PT: We have now disabled the Payments and Onboarding services and are beginning the upgrade process.

00:22 AM PT: Engineers are debugging an unrelated error in the web application that is preventing Activity from loading. We expect to have a fix shortly.

01:00 AM PT: Engineers have deployed a fix to the web application; Activity now loads as expected. We continue to make progress on the Payments upgrades.

02:30 AM PT: Engineers continue to make progress on the Payments upgrade.

10:00 AM PT: We successfully completed the first phase of the Payments upgrade. The service will remain unavailable throughout the day while we work with our partner to update their systems. We’ll continue updating here, though less frequently, until we enter the next phase of this upgrade.

5:30 PM PT: Our partner confirms that their systems are now updated. We’ve begun validating the upgrade and will continue to do so through the evening.

11:00 PM PT: We have begun the final phase of this upgrade. You can follow our progress in a separate post.

This work was completed in the following maintenance linked above.

Network Issues

[13:14 PDT] We are investigating network issues in our infrastructure. Card swipes and ATM withdrawals are not affected. We will update as we know more. Thanks! (^jake)

[13:23 PDT] The network issues are resolved and services are operating normally. Thanks for your patience! (^jake)

Web and mobile access failure

Our engineers are investigating intermittent failure to load our web  and mobile applications. Financial operations, including swipes and deposits, are unaffected and continue to operate normally. We’ll continue to update here as we make progress. Thank you! (^will)

3:15 PM: We determined that our authentication service was unable to handle increased traffic. Our engineers increased the size of the cluster supporting this service to handle the increased load. Our monitoring now confirms that access to both the web and mobile applications has been restored. Thank you for your patience! (^will)

Intermittent Connectivity Issues

14:00 UTC, We are seeing intermittent performance issues with connecting to our web app,  mobile apps, and Our engineers are aware of the solution and are working on it.

14:15 UTC, Connectivity should be back to normal. We believe we have fixed the problem and have introduced a small-term fix. We’re working on a long term fix in order to avoid this issue again.


Android service unavailable

2:00 PM PT: Our engineers are debugging an issue that is causing API requests to fail for Android devices. This issue is limited to Android; iOS and the web application are unaffected. All other operations, including swiper and deposits, are working as expected. We’ll update here soon. Thanks for your patience! (^will)

2:30 PM PT: This issue is now resolved. While attempting to resolve it, all access to and both our iOS and Android API endpoints was disrupted; a complete fix is now in place. All services are operating normally. (^will)

Check deposit unavailable

[9:46 PDT] We are currently investigating an issue with check deposits. Until further notice, the mobile applications will publish an error during the check deposit flow and the check will not be uploaded for review. No other features are affected during this time. Please contact our Customer Relations team with any questions. Thank you! (^jake)

[10:04 PDT] The issue is resolved and check deposit has been reactivated. Thank you! (^jake)

Activity Feed Maintenance

[21:25 PDT] We are performing a minor maintenance on our transactions service which means balances and the activity feed will be unavailable for approximately 60 minutes. This maintenance does not affect card swipes, ATM withdrawals, or features such as ACH transfers and payments. At the end of the maintenance, activity will again load and the Safe-to-Spend balance will be normal. Thank you! (^jake)

[22:15 PDT] The maintenance is now complete. The activity feed should now be loading, and Safe-to-Spend should be correct. Please contact our Customer Relations team with any questions. Thanks! (^jake)