• Latest
  • Trending
Incident Post Mortem: October 27, 2021

Incident Post Mortem: October 27, 2021

October 30, 2021
OmniaVerse – Will emergence of this unique young Crypto star bring new hope for the Crypto world.

OmniaVerse – Will emergence of this unique young Crypto star bring new hope for the Crypto world.

June 30, 2022
One More Way to Make Working From Home Secure

One More Way to Make Working From Home Secure

June 30, 2022
Axie Infinity to deploy a large patch today

Axie Infinity to deploy a large patch today

June 30, 2022
Ethereum Rising Gas Fees are Still Concerning But Presents Opportunity For Decentralized Exchanges

Ethereum Rising Gas Fees are Still Concerning But Presents Opportunity For Decentralized Exchanges

June 30, 2022
Crypto Vs. Stocks: The Similarities and Differences | by Cryptohopper | The Capital | Jun, 2022

Crypto Vs. Stocks: The Similarities and Differences | by Cryptohopper | The Capital | Jun, 2022

June 30, 2022
OpenSea data breach causes massive leak of users’ email addresses

OpenSea data breach causes massive leak of users’ email addresses

June 30, 2022
Deutsche Bank Predicts Bitcoin May Reclaim $28,000 This Year

Deutsche Bank Predicts Bitcoin May Reclaim $28,000 This Year

June 30, 2022
Marathon Digital Holdings Provides Update on Its Bitcoin Mining Operations in Montana

Marathon Digital Holdings Provides Update on Its Bitcoin Mining Operations in Montana

June 30, 2022
Grayscale Sues SEC Over Bitcoin ETF Application Rejection

Grayscale Sues SEC Over Bitcoin ETF Application Rejection

June 30, 2022
  • Contact Us
  • Privacy Policy
Thursday, June 30, 2022
Mount Rushmore Crypto
  • Home
  • Top News
  • Crypto
  • Crypto Technical Analysis
  • Store
No Result
View All Result
  • Home
  • Top News
  • Crypto
  • Crypto Technical Analysis
  • Store
No Result
View All Result
Mount Rushmore Crypto
No Result
View All Result
Home Crypto

Incident Post Mortem: October 27, 2021

Automatic by Automatic
October 30, 2021
in Crypto, Top News
0
Incident Post Mortem: October 27, 2021
0
SHARES
5
VIEWS
Share on FacebookShare on Twitter


Summary

Between approximately 6:40 am and 10:42 am PT, and again between 12:20 pm and 2:32 pm PT on Wednesday, October 27th, we experienced intermittent outages on Coinbase.com, Coinbase mobile apps, and Coinbase Pro. During these outages, many users experienced slow loading times and errors while attempting to access Coinbase, or were unable to use features like buying, selling, and trading through our Retail and Pro websites and apps. The Exchange itself was not materially impacted. This post is intended to describe what occurred and the causes, and to discuss how we plan to avoid such problems in the future.

Related articles

OmniaVerse – Will emergence of this unique young Crypto star bring new hope for the Crypto world.

OmniaVerse – Will emergence of this unique young Crypto star bring new hope for the Crypto world.

June 30, 2022
One More Way to Make Working From Home Secure

One More Way to Make Working From Home Secure

June 30, 2022

We’re continuing to learn more about these events, and will continue to update this post with additional details that may be of interest.

The Incident

On the morning of October 27th PT, we experienced a significant increase in traffic. As traffic increased, our engineers were alerted about elevated error rates appearing across a number of services.

The following functionality was affected:

  • Logged-out experience: users that were not logged in experienced errors when visiting coinbase.com or our mobile apps.
  • Coinbase Pro: users were temporarily unable to log in to Coinbase Pro.
  • Transfers: There was a higher rate of cancelled and refunded transfers during this time, as well as delays in processing on-chain money movements. Users may have been unable to see their latest transfer history.

Root Cause Analysis

These issues were caused by two separate but related outages. Both were triggered by system bottlenecks caused by the elevated traffic.

Traffic to Coinbase — 10/27/2021

In the first outage, we observed traffic patterns that were several times greater than previous peaks. This increase in traffic began to overload a datastore responsible for our rewards functionality. As latency increased on this database, related services became saturated and started to deplete resources as well. This resulted in a chain of failures and a more widespread outage.

Query capacity to key database cluster

The second outage was also triggered by a spike in traffic levels. In the early afternoon, engineers were alerted that our payment processing was being similarly overloaded. Unfortunately, an automated maintenance event that was already underway slowed our ability to scale this cluster up to meet with demand, and a set of failures similar to those that occurred during the first outage followed.

Elevated query latency for Payments cluster

In this instance, the servers that power our logged-out experience were also affected. As these servers became overwhelmed, they were unable to serve new traffic and were ultimately marked by our load balancer as unhealthy and removed from its pool, causing coinbase.com to become unavailable to users who were logged out or who were attempting to log in. Other impacted functionality included the ability to buy, sell, and trade in both Coinbase’s retail application as well as Coinbase Pro.

At 2:32pm PT, our services returned to normal operation.

Resolution & Improvements

For the first outage, once the caching changes were deployed, the rewards database was scaled up, and additional replicas became available. Afterwards, our system was able to resume normal operation.

To resolve the second outage, we upgraded the under-capacity payments cluster to a larger instance size and introduced additional read-only replicas.

To prevent similar issues in the future, we are taking several additional actions:

  1. Reorganizing our largest services: we will continue to shard and isolate our largest services to avoid hitting limits like those mentioned previously.
  2. Enhanced load testing: we’re enhancing our load testing framework to be more representative of new traffic patterns that we saw during this event.
  3. Additional scaling: we are further scaling several of our databases that we observed operating close to limits at Wednesday’s elevated traffic levels.

We take the uptime and performance of our infrastructure very seriously, and we’re working hard to support the millions of customers that choose Coinbase to manage their cryptocurrency. If you’re interested in solving scaling challenges like those presented here, come work with us.


Incident Post Mortem: October 27, 2021 was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.



Source link

ShareTweetShareShare
Previous Post

Crypto.com Integrates with Solana, App Becomes 3rd Most Downloaded as Ad Features Matt Damon

Next Post

Starbucks Card Is Now Reloadable with Crypto, Coffee Giant Exploring Tokenizing Through Blockchain

Related Posts

OmniaVerse – Will emergence of this unique young Crypto star bring new hope for the Crypto world.

OmniaVerse – Will emergence of this unique young Crypto star bring new hope for the Crypto world.

by Automatic
June 30, 2022
0

Toronto, June 28, 2022 (GLOBE NEWSWIRE) — Even with the current state of the Crypto Market, a first of its...

One More Way to Make Working From Home Secure

One More Way to Make Working From Home Secure

by Automatic
June 30, 2022
0

In the same way, individuals can prove that, for example, they have paid their taxes, that they live in a...

Axie Infinity to deploy a large patch today

Axie Infinity to deploy a large patch today

by Automatic
June 30, 2022
0

Axie Infinity is set to enter a crucial phase in its development after months of delay. The Axie Infinity team...

Ethereum Rising Gas Fees are Still Concerning But Presents Opportunity For Decentralized Exchanges

Ethereum Rising Gas Fees are Still Concerning But Presents Opportunity For Decentralized Exchanges

by Automatic
June 30, 2022
0

Paying gas fees is a crucial component of exchanging crypto and mining digital assets yet remains a significant cost barrier...

Crypto Vs. Stocks: The Similarities and Differences | by Cryptohopper | The Capital | Jun, 2022

Crypto Vs. Stocks: The Similarities and Differences | by Cryptohopper | The Capital | Jun, 2022

by Automatic
June 30, 2022
0

In this article, we review the similarities and differences between crypto and stocks.Trading is one of the many ways to...

Load More

Enter your email address:

Delivered by FeedBurner

Quick Navigate

  • Home
  • Crypto
  • Crypto Technical Analysis
  • Top News
  • Thank You
  • Store

Top News

Starbucks joins NFT party, UK government seeks stablecoin regulations and Crypto Twitter rallies behind cancer fighter, Hodler’s Digest: Apr. 3-9

Katie Haun’s VC Firm Leads $11M Round for Web 3 Community Platform Highlight

We Still Know Nothing About the Metaverse

© 2021 mtrushmorecrypto - Crypto Related News Blog

No Result
View All Result
  • Home
  • Top News
  • Crypto
  • Crypto Technical Analysis
  • Store

© 2021 mtrushmorecrypto - Crypto Related News Blog

We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
Do not sell my personal information.
Cookie SettingsAccept
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT