• Latest
  • Trending
Misleading Metrics Like Accuracy Are Not Worth Your Time | by Dylan Cunningham | The Capital | Jun, 2022

Misleading Metrics Like Accuracy Are Not Worth Your Time | by Dylan Cunningham | The Capital | Jun, 2022

June 25, 2022
Institutions flocking to Ethereum for 7 straight weeks as Merge nears: Report

Institutions flocking to Ethereum for 7 straight weeks as Merge nears: Report

August 8, 2022
Is Tether ($USDT) The Biggest Scam In Human History? | by Pantera | The Capital | Aug, 2022

Is Tether ($USDT) The Biggest Scam In Human History? | by Pantera | The Capital | Aug, 2022

August 8, 2022
Ripple Named Among Best Medium Workplaces by Fortune Magazine

Ripple Named Among Best Medium Workplaces by Fortune Magazine

August 8, 2022
Crypto Options Trading on Solana Has Mostly Fizzled. A New DEX Has a Plan to Change That

Crypto Options Trading on Solana Has Mostly Fizzled. A New DEX Has a Plan to Change That

August 8, 2022

BikeN, A Bike-To-Earn crypto project is starting its presale in a week

August 8, 2022
Avalanche reclaims $14 support after June crash

Avalanche price prediction as AVAX crosses key resistance

August 8, 2022
Number Of Crypto ATMs Installed Worldwide Reaches New Milestone

Number Of Crypto ATMs Installed Worldwide Reaches New Milestone

August 8, 2022
Trend Micro calls out vulnerabilities in metaverse security development

Trend Micro calls out vulnerabilities in metaverse security development

August 8, 2022
Bitcoin Reaching $100,000 Is “Matter of Time,” Says Bloomberg’s Mike McGlone

Bitcoin Reaching $100,000 Is “Matter of Time,” Says Bloomberg’s Mike McGlone

August 8, 2022
  • Privacy Policy
Tuesday, August 9, 2022
Mount Rushmore Crypto
  • Home
  • Top News
  • Crypto
  • Crypto Technical Analysis
  • Store
No Result
View All Result
  • Home
  • Top News
  • Crypto
  • Crypto Technical Analysis
  • Store
No Result
View All Result
Mount Rushmore Crypto
No Result
View All Result
Home Crypto Technical Analysis

Misleading Metrics Like Accuracy Are Not Worth Your Time | by Dylan Cunningham | The Capital | Jun, 2022

Automatic by Automatic
June 25, 2022
in Crypto Technical Analysis, Top News
0
Misleading Metrics Like Accuracy Are Not Worth Your Time | by Dylan Cunningham | The Capital | Jun, 2022
0
SHARES
1
VIEWS
Share on FacebookShare on Twitter

Related articles

Institutions flocking to Ethereum for 7 straight weeks as Merge nears: Report

Institutions flocking to Ethereum for 7 straight weeks as Merge nears: Report

August 8, 2022
Is Tether ($USDT) The Biggest Scam In Human History? | by Pantera | The Capital | Aug, 2022

Is Tether ($USDT) The Biggest Scam In Human History? | by Pantera | The Capital | Aug, 2022

August 8, 2022


Do not invest in a model tuned to accuracy: probability weighted accuracy, and others, are more trustworthy.

Accuracy as a metric, in trading, is misleading. The definition of accuracy, as it relates to a single trading strategy, is the total number of days a strategy makes money over the total number of days the strategy took a position. This misses at least one key point in trading applications: profits and losses.

Photo by Ricardo Arce on Unsplash

In his recent books, Marcos Lopez de Prado teaches key principles when using metrics to evaluate a trading strategy. The points below are summaries of principles pulled from his work and my own thoughts on the matter.

(The code chunks are pulled from his work too, but I adjusted them only slightly to make them make sense to me. Hopefully my minor adjustments help you too.)

For machine learning training purposes, we simply need a metric that mirrors how we want to evaluate a strategy’s performance.

Sizing your bets incorrectly will get you into more trouble than incorrectly guessing the direction of price moves. Yes, if you make a bet and are wrong, then you will lose money, but the goal is to lose small amounts when you are wrong and to make large amounts when you are right. Savvy?

Log Loss

Compared to accuracy, a better metric is log loss. Log loss “rewards” correct predictions assigned higher confidence, while it “punishes” incorrect predictions assigned higher confidence. This is to say, a good log loss score is one where long and short bets are made accurately with predicted probabilities that are high and where long and short bets are made incorrectly with predicted probabilities that are low. My complaint with log loss is that it is difficult to interpret.

Log loss is difficult to interpret, but you should check out this Stack Exchange question and the answer given by Fed Zee. Simply put, Fed Zee shows some complexities to log loss as he compares log loss scores to accuracy.

A lower log loss score is better, but the best way to use log loss is by negating it; then, higher values are better, just like all other metrics (e.g., accuracy, recall, F1, etc.). scikit-learn’s implementation is sufficient.

from sklearn.metrics import log_loss...probabilities = clf.predict_proba(X_test)neg_log_loss = -log_loss(
y_test, probabilities, w_test, labels=clf.classes_)

Weighted Accuracy

To retain the interpretability of accuracy and to extend its functionality, you can use weighted accuracy. Weighted accuracy will compute accuracy but give higher or lower weights based on your input.

One way to make this metric valuable is to pass in returns as weights. Correct predictions which would have made more money receive higher weight. Incorrect predictions, which would have lost more money also receive higher weights. Correct and incorrect predictions which would have returned low profits or losses receive lower weights. This emulates how we want to evaluate strategies.

def weighted_accuracy(yn, wght, normalize=False):
"""
Weighted accuracy (normalize=True), or weighted sum
(normalize=False)

:param yn: indicator function yn E 0,1 where yn = 1 when
prediction was correct, yn = 0 otherwise
:param: wght: sample weights
"""

if normalize:
return np.average(yn, weights=wght)
elif wght is not None:
return np.dot(yn, wght)
else:
return yn.sum()

Probability Weighted Accuracy

You can probably guess, by its name, what this metric is measuring. Yes! You are correct. Probability weighted accuracy, introduced by Marcos Lopez de Prado, in Machine Learning for Asset Managers, uses probabilities to weight accuracy, similarly to log loss, however, it is much more interpretable.

Probability weighted accuracy punishes bad predictions made with high confidence more severely than accuracy, but less severely than log-loss. — Marcos Lopez de Prado

def probability_weighted_accuracy(yn, pn, K):
"""
PWA punishes bad predictions made with high confidence more
severely than accuracy, but less severely than log-loss

:param yn: indicator function yn E 0,1 where yn = 1 when
prediction was correct, yn = 0 otherwise
:param pn: max k pn,k where pn,k is the prob associated
with prediction n of label k
:param K: num of classes
"""

return np.sum(
(yn * (pn.max(axis=1) - (K * 1.) ** -1))[:-1]) / np.sum(
(pn.max(axis=1) - (K * 1.) ** -1)[:-1])



Source link

ShareTweetShareShare
Previous Post

Why Solana Has Outperformed Ethereum, SOL Up 36% In One Week

Next Post

Crucial Decision on Testnet Hard Fork to Be Made Next Week

Related Posts

Institutions flocking to Ethereum for 7 straight weeks as Merge nears: Report

Institutions flocking to Ethereum for 7 straight weeks as Merge nears: Report

by Automatic
August 8, 2022
0

Institutional investors are piling into Ether-based digital asset funds, which have recorded seven straight weeks of positive inflows, according to...

Is Tether ($USDT) The Biggest Scam In Human History? | by Pantera | The Capital | Aug, 2022

Is Tether ($USDT) The Biggest Scam In Human History? | by Pantera | The Capital | Aug, 2022

by Automatic
August 8, 2022
0

StablecoinsWhy Tether Will Never Be AuditedPhoto Background by “WikiImages” on PixabayTether ($USDT) reached close to the market cap the CTO...

Ripple Named Among Best Medium Workplaces by Fortune Magazine

Ripple Named Among Best Medium Workplaces by Fortune Magazine

by Automatic
August 8, 2022
0

Alex Dovbnya Fortune has recognized Ripple, a prominent blockchain company, as one of the best medium workplaces once again Blockchain...

Crypto Options Trading on Solana Has Mostly Fizzled. A New DEX Has a Plan to Change That

Crypto Options Trading on Solana Has Mostly Fizzled. A New DEX Has a Plan to Change That

by Automatic
August 8, 2022
0

The appeal is that it can make trading less capital intensive. In derivatives, it’s typical to make traders post collateral...

BikeN, A Bike-To-Earn crypto project is starting its presale in a week

by Automatic
August 8, 2022
0

BikeN is a Web3 bike-to-earn lifestyle app with metaverse integration focused on biking and fitness.  The project aims to promote...

Load More

Enter your email address:

Delivered by FeedBurner

Quick Navigate

  • Home
  • Crypto
  • Crypto Technical Analysis
  • Top News
  • Thank You
  • Store

Top News

$SHAZU Sets Official Listing Date on May 30 for it’s BSC Token | by The Capital | The Capital | May, 2022

Crypto exec says India needs to ‘rethink’ crypto tax policy

Homo Economicus and the Rational Expectations Theory | by Dimitrios Gourtzilidis | The Capital | May, 2022

© 2021 mtrushmorecrypto - Crypto Related News Blog

No Result
View All Result
  • Home
  • Top News
  • Crypto
  • Crypto Technical Analysis
  • Store

© 2021 mtrushmorecrypto - Crypto Related News Blog

We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
Do not sell my personal information.
Cookie SettingsAccept
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT