Reliability, bear in mind and F1-get a variety of categories of elements removed by the fantasy operating product up against the hands-coded establishes
4.cuatro. Assessment
We analyzed our very own tool-using a couple groups of fantasy records you to definitely have been give-coded by the fantasy masters utilizing the Hallway–Van de- Castle program (§cuatro.2.1): (i) the fresh new annotated number of dream account, and (ii) the latest normative set of which brand new norms found in this new literature were computed. For all those fantasy accounts, we mentioned the fresh new the total amount that this new categories of emails, correspondence and thoughts projected because of the dream operating product coordinated the newest associated floor-details establishes; desk cuatro summarizes new resulting reliability, keep in mind and F1-score.
I upcoming went on to compare brand new brand new Hallway–Van de- Palace indications calculated by the unit (desk step 1) toward relevant surface-information philosophy. Because of the floor-truth value v together with tool’s value v ? , i calculated the error given that elizabeth = | v ? v ? | .
Complete, the typical mistake across categories is 0.twenty-four (shape 3b), which is limited due to the higher variability regarding textual appearance when you look at the brand new corpus, additionally the built-in difficulty of a few of your own methods. In order to understand the magnitude of your own error, you ought to envision you to, in practice, the evidence undertake philosophy that will be typically when you look at the the brand new [0,1] range on this subject particular sample set of fantasy reports. This new measure you to definitely deviates really out of this variety is the A / C Index : it’s higher than one in six% of one’s circumstances in the crushed-knowledge plus in 3% of one’s times predicated on our product.