ROC curve for binary measurements...?

[GWestphal]

So I have been tasked with making some graphs for some data. I was asked to make some ROC curves for the data and I've seen the graphs in papers and understand what they mean, but I've never made one and I'm not sure I can with the data I have.

I have a signal with spikes in, I have a vector Ref with all the true times of those spikes, and I have vector Test that is the the times output by an algorithm whose purpose is to automatically detect said spikes.

Comparing the Ref and the Test vectors I can get TP, FN, FP, which I can calculate Sensitivity and Specificity...but isn't that just one point? Can I get more points for the ROC curve somehow from two essentially binary vectors?

As I understand it with two binary vectors,I would only have three points (0,0)-->(Specificity, Sensitivity)-->(1,1).

Am I right? Wrong? What am I missing?

 

[Mark R]

An ROC curve requires that you be able to calculate sens/spec for multiple different thresholds.

For example, you vary the theshold on your spike detector. For each different threshold, you calculate spec and sens, and then plot on a chart of spec vs (1-sens)

 

[GWestphal]

Wouldn't each of those threshold changes be points on a different ROC curves? I seem to recall when you'd have a chart of several ROCs on one plot it each one would be associated with a given value of a parameter (curve 1 is P=1, curve 2 is P=1.4, curve 3 is P=4.5 etc)...or is only ever a one dimensional investigation? The algorithm is pretty complicated and has about 50 parameters, so it would be quite arduous to make ROCs at 20 different parameters levels for each of 50 parameters, especially since it takes about 30 mins per run and is a multi level function style code so it's harder to script the parameter changes in an efficient manner without ending up with a heck of a lot of little code changes. Then how do you optimize it to multiple parameters if their are interaction effects between parameters? Oi...

Hmm.

 

[Mark R]

The ROC is classically derived from a threshold. You have a continuous measurement, which is threshold filtered, to yield a binary signal.

The ROC curve is inherently a non-parametric measure designed to measure the performance of a specific system.

If there are adjustable parameters, then you can plot an ROC for different values of that parameter. Alternatively, you can reduce the ROC curve to an AUC.

You can then explore AUC for various parameter values, or combinations thereof.