I offer my impressions on the Signature Diagnostics Detector C blood test for CRC screening. Bottom line: one of the better attempts but not a serious threat to EXAS IMO due to severe specificity problems not reflected in their reported numbers.
Given the host of exaggerated claims about CRC screening, and the profound disadvantages of blood vs stool, the burden is on them to show they’ve achieved the impossible, and they have not. Scrutiny reveals they don’t actually claim to have achieved performance that would be a serious threat to EXAS, IMO.
I offer some facts and opinions and did some reading between the lines. I feel confident of my conclusions, but encourage everyone to do their own DD. Links:
They took 572 blood samples, isolated RNA, and analyzed it in their array. If “RNA Integrity” was insufficient, they tried extraction again, and rejected the sample if still inadequate. They likewise used up to two tries followed by rejection as necessary to ensure good performance in their array. This gave a no-call rate over 19%, a serious limitation not reflected in the reported sens/spec. In addition, I suspect they also avoided subjects with inflammation, which is very common (arthritis etc.) and has been a severe problem for similar tests (e.g., the Arber test, previously discussed on this board. Arber reported impressive performance by first excluding 30% or more of subjects with inflammation.) Sig D used “healthy” normal subjects, and said “Detector C measures host response (…inflammation) of white blood cells to tumor lesions”.
They used 55 CRC and 64 normal samples for a discovery trial to design a test, which they then validated with the remaining samples (including 54 Stage 1, and 133 normal). They used some sophisticated (but well-known for years) algorithms to find a CRC gene signature (using numerous genes), and designed a test to detect it. They report val results of 89/88 sens/spec for Stage 1 CRC, very similar results at all stages, and 66% pre-cancer sensitivity. Very impressive numbers if not for the no-calls, and if obtained for general population. But, they have not confirmed general population, and I believe general population specificity would be far lower, even beyond the substantial effect of the no-calls.
They say their test could help many, and false positives are currently recommended for a colo anyway. They brag about high sensitivity (low false neg), but are strangely silent about specificity during concluding and summary remarks. This is all consistent with putting a good spin on a test with very high no-call and false positive rates, only applicable to a subset of the population. When I read their material with this in mind, it seems pretty clear to me. People don’t parse without a reason.
By the way, Health Discovery failed in a similar attempt a while back (previously discussed on this board).
Technical remarks: they reported using a random forest algorithm (to find gene signature using a model made up of many decision trees and incorporating random search, RFA has a reputation for being robust to noise and to over-modeling). Over-modeling is being too customized for the specific subjects in the training trial, and is more of a problem when there are fewer samples and more markers. Sig D used a support vector machine (SVM), presumably to partition their decision space with a hyperplane.
I am perplexed by performance very independent of cancer stage. Also, the val performance was surprisingly similar to training data. This is an indication of no severe over-modeling, but it was almost too good (it’s normal for training results to be better than val). Tests with lots of parameters (like this one) are more vulnerable to over-modeling.
I got carried away again, that’s enough.
True, lots of genes, and many samples processed twice. And, very limited additional performance for additional genes after the first 8 or so. Doesn't seem cost effective. Also strange the graph of performance vs number of genes dipped at one point (more genes gave worse performance).
completely agree (and reposting) --
this is the fourth blood-based colon cancer screening test that has come along. So far, all have been flops and I fully expect this one will as well.
It's simply not possible to obtain enough mutated DNA from colon cells via the blood stream. If it were possible, Conroy's Invader technology would probably be the best situated - i.e. EXAS already has a leg up on everyone else when the technology finally gets to a point where blood based tests can detect incredibly small amounts of mutant DNA against a huge background of normal DNA.
And some food for thought on using inflammation markers - Aspen Biopharma has been trying to using a panel of "inflammation markers" for their appendicitis test for years. This is a setting that would seem ideal for inflammation markers, but even Aspen is getting piss poor results. Inflammation is just way too of a generic marker family.
We're on the same page. Lots of people have been looking at the problem for years, the best they seem to do is about match a good FIT test (for much higher cost). Whenever a new one comes along, I always take a look, but it always turns out the same way. The exaggerated claims wilt under scrutiny. EXAS is the real deal.
The Sig D guys seem pretty good at algorithms, but no algorithm can extract information that's not there. As KC said, the fundamental limitation for blood tests is biology, not technology.