• Latest
  • Trending
Incident Post Mortem: October 27, 2021

Incident Post Mortem: October 27, 2021

October 30, 2021
$170K of Stolen Ethereum Traced to North Korean Hackers

$170K of Stolen Ethereum Traced to North Korean Hackers

March 17, 2023
Web3 Offers a Cure to Toxic Pop Culture

Web3 Offers a Cure to Toxic Pop Culture

March 17, 2023
Metacade’s Presale Hits $10.9m in April 2023. 5 Reasons MCADE Tokens Are The Best Investments Today.

Metacade’s Presale Hits $10.9m in April 2023. 5 Reasons MCADE Tokens Are The Best Investments Today.

March 17, 2023
BTC, ETH, BNB, XRP, ADA, DOGE, MATIC, SOL, DOT, SHIB

BTC, ETH, BNB, XRP, ADA, DOGE, MATIC, SOL, DOT, SHIB

March 17, 2023
Silicon Valley Bank’s Bankruptcy: 100% guaranteed no Bailout because of the Dodd-Frank Act. | by Patrick Mehrhoff | The Capital Platform | Mar, 2023

Silicon Valley Bank’s Bankruptcy: 100% guaranteed no Bailout because of the Dodd-Frank Act. | by Patrick Mehrhoff | The Capital Platform | Mar, 2023

March 17, 2023
DOGE Price Analysis for March 17

DOGE Price Analysis for March 17

March 17, 2023
First Mover Americas: Bitcoin Busts Through $26K

First Mover Americas: Bitcoin Busts Through $26K

March 17, 2023
BitValue announced the Investment in Spield Algorithm, which is Shaping the Future of AI Quantitative Analysis

Stablz Launches Cannavest pools, with 30% APR on USDT deposits!

March 17, 2023
University students reveal new Web3 solutions at ETHDenver 2023

University students reveal new Web3 solutions at ETHDenver 2023

March 17, 2023
  • Privacy Policy
Friday, March 17, 2023
Mount Rushmore Crypto
  • Home
  • Top News
  • Crypto
  • Crypto Technical Analysis
  • About Us
  • Crypto Vouchers
  • Cryptocurrency and ANKR Price Prediction
  • Cosmos cryptocurrency price prediction
No Result
View All Result
  • Home
  • Top News
  • Crypto
  • Crypto Technical Analysis
  • About Us
  • Crypto Vouchers
  • Cryptocurrency and ANKR Price Prediction
  • Cosmos cryptocurrency price prediction
No Result
View All Result
Mount Rushmore Crypto
No Result
View All Result
Home Crypto

Incident Post Mortem: October 27, 2021

J_News by J_News
October 30, 2021
in Crypto, Top News
0
Incident Post Mortem: October 27, 2021
0
SHARES
6
VIEWS
Share on FacebookShare on Twitter


Summary

Between approximately 6:40 am and 10:42 am PT, and again between 12:20 pm and 2:32 pm PT on Wednesday, October 27th, we experienced intermittent outages on Coinbase.com, Coinbase mobile apps, and Coinbase Pro. During these outages, many users experienced slow loading times and errors while attempting to access Coinbase, or were unable to use features like buying, selling, and trading through our Retail and Pro websites and apps. The Exchange itself was not materially impacted. This post is intended to describe what occurred and the causes, and to discuss how we plan to avoid such problems in the future.

Related articles

$170K of Stolen Ethereum Traced to North Korean Hackers

$170K of Stolen Ethereum Traced to North Korean Hackers

March 17, 2023
Web3 Offers a Cure to Toxic Pop Culture

Web3 Offers a Cure to Toxic Pop Culture

March 17, 2023

We’re continuing to learn more about these events, and will continue to update this post with additional details that may be of interest.

The Incident

On the morning of October 27th PT, we experienced a significant increase in traffic. As traffic increased, our engineers were alerted about elevated error rates appearing across a number of services.

The following functionality was affected:

  • Logged-out experience: users that were not logged in experienced errors when visiting coinbase.com or our mobile apps.
  • Coinbase Pro: users were temporarily unable to log in to Coinbase Pro.
  • Transfers: There was a higher rate of cancelled and refunded transfers during this time, as well as delays in processing on-chain money movements. Users may have been unable to see their latest transfer history.

Root Cause Analysis

These issues were caused by two separate but related outages. Both were triggered by system bottlenecks caused by the elevated traffic.

Traffic to Coinbase — 10/27/2021

In the first outage, we observed traffic patterns that were several times greater than previous peaks. This increase in traffic began to overload a datastore responsible for our rewards functionality. As latency increased on this database, related services became saturated and started to deplete resources as well. This resulted in a chain of failures and a more widespread outage.

Query capacity to key database cluster

The second outage was also triggered by a spike in traffic levels. In the early afternoon, engineers were alerted that our payment processing was being similarly overloaded. Unfortunately, an automated maintenance event that was already underway slowed our ability to scale this cluster up to meet with demand, and a set of failures similar to those that occurred during the first outage followed.

Elevated query latency for Payments cluster

In this instance, the servers that power our logged-out experience were also affected. As these servers became overwhelmed, they were unable to serve new traffic and were ultimately marked by our load balancer as unhealthy and removed from its pool, causing coinbase.com to become unavailable to users who were logged out or who were attempting to log in. Other impacted functionality included the ability to buy, sell, and trade in both Coinbase’s retail application as well as Coinbase Pro.

At 2:32pm PT, our services returned to normal operation.

Resolution & Improvements

For the first outage, once the caching changes were deployed, the rewards database was scaled up, and additional replicas became available. Afterwards, our system was able to resume normal operation.

To resolve the second outage, we upgraded the under-capacity payments cluster to a larger instance size and introduced additional read-only replicas.

To prevent similar issues in the future, we are taking several additional actions:

  1. Reorganizing our largest services: we will continue to shard and isolate our largest services to avoid hitting limits like those mentioned previously.
  2. Enhanced load testing: we’re enhancing our load testing framework to be more representative of new traffic patterns that we saw during this event.
  3. Additional scaling: we are further scaling several of our databases that we observed operating close to limits at Wednesday’s elevated traffic levels.

We take the uptime and performance of our infrastructure very seriously, and we’re working hard to support the millions of customers that choose Coinbase to manage their cryptocurrency. If you’re interested in solving scaling challenges like those presented here, come work with us.


Incident Post Mortem: October 27, 2021 was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.



Source link

ShareTweetShareShare
Previous Post

Crypto.com Integrates with Solana, App Becomes 3rd Most Downloaded as Ad Features Matt Damon

Next Post

Starbucks Card Is Now Reloadable with Crypto, Coffee Giant Exploring Tokenizing Through Blockchain

Related Posts

$170K of Stolen Ethereum Traced to North Korean Hackers

$170K of Stolen Ethereum Traced to North Korean Hackers

by J_News
March 17, 2023
0

Alex Dovbnya Blockchain analytics firm Chainalysis has identified a North Korean-linked address that received approximately $170,000 worth of Ethereum stolen...

Web3 Offers a Cure to Toxic Pop Culture

Web3 Offers a Cure to Toxic Pop Culture

by J_News
March 17, 2023
0

A meaningful future of fandom in Web3 can mean many things: a culture of loyalty, financial freedom, collective ideals and...

Metacade’s Presale Hits $10.9m in April 2023. 5 Reasons MCADE Tokens Are The Best Investments Today.

Metacade’s Presale Hits $10.9m in April 2023. 5 Reasons MCADE Tokens Are The Best Investments Today.

by J_News
March 17, 2023
0

Newcomer Metacade has made a huge splash since appearing on the crypto scene, establishing itself as a high-potential investment opportunity...

BTC, ETH, BNB, XRP, ADA, DOGE, MATIC, SOL, DOT, SHIB

BTC, ETH, BNB, XRP, ADA, DOGE, MATIC, SOL, DOT, SHIB

by J_News
March 17, 2023
0

After a recovery on March 16, the United States equities markets are again down on March 17. Investors remain concerned...

Silicon Valley Bank’s Bankruptcy: 100% guaranteed no Bailout because of the Dodd-Frank Act. | by Patrick Mehrhoff | The Capital Platform | Mar, 2023

Silicon Valley Bank’s Bankruptcy: 100% guaranteed no Bailout because of the Dodd-Frank Act. | by Patrick Mehrhoff | The Capital Platform | Mar, 2023

by J_News
March 17, 2023
0

The Silicon Valley Bank’s bankruptcy had sent ripples across the financial landscape, striking fear into the hearts of its customers...

Load More

Enter your email address:

Delivered by FeedBurner

Quick Navigate

  • Home
  • Crypto
  • Crypto Technical Analysis
  • Top News
  • Thank You
  • Store
  • Crypto Vouchers
  • About Us
  • What Cryptocurrency Is and ANKR Price Prediction
  • Cosmos cryptocurrency price prediction

Top News

The 7 Best New ICOs To Buy in 2023

New York Attorney General sues CoinEx over failure to register

The Economic Crisis of 2023. The Cause and the Effects | by Opal A Roszell | The Capital | Nov, 2021

© 2021 mtrushmorecrypto - Crypto Related News Blog

No Result
View All Result
  • Home
  • Top News
  • Crypto
  • Crypto Technical Analysis
  • About Us
  • Crypto Vouchers
  • Cryptocurrency and ANKR Price Prediction
  • Cosmos cryptocurrency price prediction

© 2021 mtrushmorecrypto - Crypto Related News Blog

We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
Do not sell my personal information.
Cookie SettingsAccept
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT