Cohen’s Kappa

class: center, middle, inverse, title-slide

# Cohen’s Kappa
## EDP 612 Week 8
### Dr. Abhik Roy

---

section {
    height: 600px;
    width: 60%;
    margin: auto;
    border-radius: 21px;
    background-color: #212121;
}

section p {
    text-align: center;
    font-size: 30px;
    background-color: #212121;
    border-radius: 21px;
    font-family: Roboto Condensed;
    font-style: bold;
    padding: 12px;
    color: #bff4ee;
    margin: auto;
}

#center {

text-align: center;

}

.center p {
  margin: 0;
  position: absolute;
  top: 50%;
  left: 50%;
  -ms-transform: translate(-50%, -50%);
  transform: translate(-50%, -50%);
}

.center2 {
  margin: 0;
  position: absolute;
  top: 50%;
  left: 50%;
  -ms-transform: translate(-50%, -50%);
  transform: translate(-50%, -50%);
}

.tab {
    display: inline-block;
    margin-left: 40px;
}

td, th, tr, table {
 border: 0 !important;
 border-spacing:0 !important;
 }
</style>

<div>
<style type="text/css">.xaringan-extra-logo {
width: 110px;
height: 128px;
z-index: 0;
background-image: url(/Users/skynet/Documents/WVU/Teaching/GitHub.nosync/Workspace/edp612/static/img/course_hex.png);
background-size: contain;
background-repeat: no-repeat;
position: absolute;
top:1em;right:1em;
}
</style>
<script>(function () {
 let tries = 0
 function addLogo () {
 if (typeof slideshow === 'undefined') {
 tries += 1
 if (tries < 10) {
 setTimeout(addLogo, 100)
 }
 } else {
 document.querySelectorAll('.remark-slide-content:not(.title-slide):not(.inverse):not(.hide_logo)')
 .forEach(function (slide) {
 const logo = document.createElement('a')
 logo.classList = 'xaringan-extra-logo'
 logo.href = 'https://edp612.asocialdatascientist.com'
 slide.appendChild(logo)
 })
 }
 }
 document.addEventListener('DOMContentLoaded', addLogo)
})()</script>
</div>

---

class: highlight-last-item
layout: true

---

# Reliability and Validity 
--

.pull-left[
<center>
Reliability being consistent
</center>
]

.pull-right[
<center>
Validity on target
</center>
]

---

# Recall Scenarios

.pull-left[
<center>
Not Reliable and not Valid
 
<img src="img/image10.png" alt="norelnoval" style="width: 150px;"/>
</center>
]

.pull-right[
<center>
Reliable but not Valid
 
<img src="img/image12.png" alt="relnoval" style="width: 150px;"/>
</center>
]

.pull-left[
<center>
Not Reliable but Valid
 
<img src="img/image13.png" alt="norelval" style="width: 150px;"/>
</center>
]

.pull-right[
<center>
Reliable and Valid
 
<img src="img/image14.png" alt="relval" style="width: 150px;"/>
</center>
]

---

# Basic Tenant

.pull-left[
A test can be Reliable without being Valid
]

.pull-right[
A test cannot be Valid unless it is Reliable
]

---

# Reliability

.pull-left[
<center>
Intra-rater Reliability the degree of agreement between different measurements done by the same person
</center>
]

.pull-right[
<center>
Inter-rater Reliability degree of agreement between between different measurements done by multiple people
</center>
]

---

# Cohen's Kappa `$\kappa$`

+ *Officially*. Measure of the agreement between two raters who each classify `$N$` items into `$C$` mutually exclusive categories

+ *Basic idea*.

+ Quantitative measure of reliability for two raters that are rating the same thing

+ With a correction for how often that the raters may agree by chance
	  
--

+ *Lay terms*. Measure of how well do different people agree

---

## Evaluating

.pull-left[
<center>
 `\kappa < 0` 
random agreement among raters
</center>
]

.pull-right[
<center>
 `\kappa = 1` 
complete agreement among raters
</center>
]

---

## Decision Matrix

.pull-left[
<center>
 `A` 
Agreement 
The total number of instances that both Raters said were correct
</center>
]

.pull-right[
<center>
 `B` 
Disagreement 
The total number of instances that Rater 2 said was incorrect, but Rater 1 said were correct
</center>
]

.pull-left[
<center>
 `C` 
Disagreement 
The total number of instances that Rater 1 said was incorrect, but Rater 2 said were correct
</center>
]

.pull-right[
<center>
 `D` 
Agreement 
The total number of instances that both Raters said were incorrect
</center>
]

---

## Calculations

---

## Probability of Agreement `$P_0$`

<center>
Number in Full Agreement / Total
</center>
.center2[
`$$\dfrac{A+D}{A+B+C+D}$$`
]

---

## Probability of Correct Random Agreement `$P_{correct}$`

<center>
Number Correct in Full or Partial Agreement / Total
</center>
.center2[
`$$\dfrac{A+B}{A+B+C+D}\cdot\dfrac{A+C}{A+B+C+D}$$`
]

---

## Probability of Incorrect Random Agreement `$P_{incorrect}$`

<center>
Number Incorrect in Full or Partial Agreement / Total
</center>
.center2[
`$$\dfrac{C+D}{A+B+C+D}\cdot\dfrac{B+D}{A+B+C+D}$$`
]

---

## Probability of Random Agreement

.center2[
`$$P_e = P_{correct} + P_{incorrect}$$`
]

---

## Kappa `$\kappa$`

.center2[
`$$\kappa = \dfrac{P_0 - P_e}{1-P_e}$$` 
]

---

## Interpretation

.center2[
<table class=" lightable-paper" style='font-family: "Arial Narrow", arial, helvetica, sans-serif; width: auto !important; margin-left: auto; margin-right: auto;'>
 <thead>
 <tr>
 <th style="text-align:center;color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> Value of `kappa` </th>
 <th style="text-align:left;color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> Strength of Agreement </th>
 </tr>
 </thead>
<tbody>
 <tr>
 <td style="text-align:center;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> `leq 0.20` </td>
 <td style="text-align:left;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> Poor </td>
 </tr>
 <tr>
 <td style="text-align:center;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> `0.21-0.40` </td>
 <td style="text-align:left;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> Fair </td>
 </tr>
 <tr>
 <td style="text-align:center;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> `0.41-0.60` </td>
 <td style="text-align:left;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> Moderate </td>
 </tr>
 <tr>
 <td style="text-align:center;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> `0.61-0.80` </td>
 <td style="text-align:left;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> Good </td>
 </tr>
 <tr>
 <td style="text-align:center;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> `geq 0.81` </td>
 <td style="text-align:left;width: 10em; color: #ffffff !important;background-color: #212121 !important;vertical-align: middle !important;"> Very Good </td>
 </tr>
</tbody>
</table>
]

---

#  That’s it!

Any questions?