Word count vectors as features for classification, overlapping k-mers
| Confusion matrix | ||||
|---|---|---|---|---|
| 5453.0 | 8.0 | 6.0 | 1.0 | 0.0 |
| 135.0 | 2754.0 | 1.0 | 1.0 | 0.0 |
| 211.0 | 4.0 | 297.0 | 4.0 | 0.0 |
| 115.0 | 0.0 | 3.0 | 275.0 | 0.0 |
| 42.0 | 1.0 | 2.0 | 3.0 | 48.0 |
| 7.0 | 0.0 | 0.0 | 2.0 | 0.0 |
Summary Statistics * Accuracy = 0.942
Word count vectors as features for classification, overlapping k-mers
| Confusion matrix | ||||
|---|---|---|---|---|
| 5446.0 | 8.0 | 10.0 | 4.0 | 0.0 |
| 137.0 | 2746.0 | 0.0 | 1.0 | 0.0 |
| 186.0 | 7.0 | 314.0 | 9.0 | 0.0 |
| 112.0 | 2.0 | 9.0 | 270.0 | 0.0 |
| 43.0 | 3.0 | 1.0 | 3.0 | 46.0 |
| 10.0 | 0.0 | 0.0 | 0.0 | 0.0 |
Summary Statistics * Accuracy = 0.941
Word count vectors as features for classification, overlapping k-mers
| Confusion matrix | ||||
|---|---|---|---|---|
| 5446.0 | 8.0 | 10.0 | 4.0 | 0.0 |
| 137.0 | 2746.0 | 0.0 | 1.0 | 0.0 |
| 186.0 | 7.0 | 314.0 | 9.0 | 0.0 |
| 112.0 | 2.0 | 9.0 | 270.0 | 0.0 |
| 43.0 | 3.0 | 1.0 | 3.0 | 46.0 |
| 10.0 | 0.0 | 0.0 | 0.0 | 0.0 |
Summary Statistics * Accuracy = 0.941
Number of topics:10 Number of iterations: 100 Nonoverlapping accuracy table:
| Confusion matrix | ||||
|---|---|---|---|---|
| 5167.0 | 288.0 | 0.0 | 11.0 | 0.0 |
| 2414.0 | 472.0 | 0.0 | 1.0 | 0.0 |
| 496.0 | 20.0 | 0.0 | 0.0 | 0.0 |
| 217.0 | 5.0 | 0.0 | 170.0 | 0.0 |
| 94.0 | 2.0 | 0.0 | 0.0 | 0.0 |
| 28.0 | 0.0 | 0.0 | 3.0 | 0.0 |
Summary Statistics * Accuracy = 0.618 * fm0 = 0.744 * fm1 = 0.257 * fm2 = 0.0 * fm3 = 0.588 * fm4 = 0.0
Number of topics:10 Number of iterations: 1000 Nonoverlapping accuracy table:
| Confusion matrix | ||||
|---|---|---|---|---|
| 5290.0 | 113.0 | 10.0 | 3.0 | 0.0 |
| 1012.0 | 1667.0 | 22.0 | 0.0 | 3.0 |
| 430.0 | 10.0 | 76.0 | 0.0 | 0.0 |
| 213.0 | 6.0 | 3.0 | 169.0 | 0.0 |
| 93.0 | 2.0 | 1.0 | 0.0 | 0.0 |
| 31.0 | 0.0 | 0.0 | 0.0 | 0.0 |
Summary Statistics * Accuracy = 0.787 * fm0 = 0.847 * fm1 = 0.740 * fm2 = 0.242 * fm3 = 0.600 * fm4 = 0.0
Number of topics:20 Number of iterations: 100 Nonoverlapping accuracy table:
| Confusion matrix | ||||
|---|---|---|---|---|
| 4174.0 | 1286.0 | 0.0 | 8.0 | 0.0 |
| 829.0 | 2056.0 | 0.0 | 6.0 | 0.0 |
| 408.0 | 108.0 | 0.0 | 0.0 | 0.0 |
| 189.0 | 34.0 | 0.0 | 170.0 | 0.0 |
| 48.0 | 48.0 | 0.0 | 0.0 | 0.0 |
| 29.0 | 1.0 | 0.0 | 1.0 | 0.0 |
Summary Statistics * Accuracy = 0.681 * fm0 = 0.749 * fm1 = 0.640 * fm2 = 0.0 * fm3 = 0.588 * fm4 = 0.0
Number of topics:20 Number of iterations: 20000 Nonverlapping accuracy table:
| Confusion matrix | ||||
|---|---|---|---|---|
| 5373.0 | 89.0 | 3.0 | 3.0 | 0.0 |
| 368.0 | 2522.0 | 0.0 | 1.0 | 0.0 |
| 478.0 | 15.0 | 20.0 | 3.0 | 0.0 |
| 193.0 | 8.0 | 0.0 | 192.0 | 0.0 |
| 89.0 | 5.0 | 0.0 | 1.0 | 1.0 |
| 14.0 | 0.0 | 0.0 | 0.0 | 0.0 |
Summary Statistics * Accuracy = 0.865 * fm0 = 0.897 * fm1 = 0.912 * fm2 = 0.074 * fm3 = 0.648 * fm4 = 0.021
Number of topics:20 Number of iterations: 15000 Overlapping accuracy table:
| Confusion matrix | ||||
|---|---|---|---|---|
| 5419.0 | 25.0 | 12.0 | 12.0 | 0.0 |
| 190.0 | 2687.0 | 7.0 | 7.0 | 0.0 |
| 398.0 | 13.0 | 89.0 | 15.0 | 0.0 |
| 193.0 | 2.0 | 2.0 | 196.0 | 0.0 |
| 89.0 | 2.0 | 3.0 | 1.0 | 1.0 |
| 11.0 | 0.0 | 0.0 | 0.0 | 0.0 |
Summary Statistics * Accuracy = 0.895 * fm0 = 0.921 * fm1 = 0.956 * fm2 = 0.283 * fm3 = 0.628 * fm4 = 0.021
Number of topics:50 Number of iterations: 100 Nonoverlapping accuracy table:
| Confusion matrix | ||||
|---|---|---|---|---|
| 5250.0 | 217.0 | 0.0 | 1.0 | 0.0 |
| 2667.0 | 224.0 | 0.0 | 0.0 | 0.0 |
| 503.0 | 13.0 | 0.0 | 0.0 | 0.0 |
| 220.0 | 3.0 | 0.0 | 170.0 | 0.0 |
| 94.0 | 2.0 | 0.0 | 0.0 | 0.0 |
| 30.0 | 1.0 | 0.0 | 0.0 | 0.0 |
Summary Statistics * Accuracy = 0.601 * fm0 = 0.738 * fm1 = 0.134 * fm2 = 0.0 * fm3 = 0.603 * fm4 = 0.0
Summary Tables for (LDA + Classification) and Classification
- (LDA + Classification) Varying number of topics and fixed number of iterations = 100:
| # topics | Accuracy |
|---|---|
| 10 | 0.618 |
| 20 | 0.681 |
| 50 | 0.601 |
Conclusion: 20, approximately the number of aminoacids, is a good candidate for the number of topics
- We tried both nonoverlapping and overlapping k-mer features directly on classifier:
| Data type | Accuracy |
|---|---|
| Nonoverlapping k-mers | 0.941 |
| Overlapping k-mers | 0.942 |
- Also, on (LDA + classification) for number of topics = 20. While for nonoverlapping the number of iterations are 20000, for overlapping the number of iterations are 15000. Due to 'unexpected shutdown' of clusters we couldn't run LDA on overlapping clusters for 20000 iterations. However results suggests that overlapping k-mers helps LDA to learn structure better in terms of prediction power (but not efficiently) :
| Data type | Accuracy |
|---|---|
| Nonoverlapping k-mers | 0.865 |
| Overlapping k-mers | 0.895 |
Conclusion: For classifier, there is not much difference on overlapping or nonoverlapping k-mer features; however for LDA having overlapping helps to learn structure more. But increased time complexity of overlapping k-mers requires LDA to have more iterations to learn.
- (LDA + Classification) Varying number of iterations and topics :
| (# iterations, # topics, overlapping) | Accuracy |
|---|---|
| (100, 10, false) | 0.618 |
| (1000, 10, false) | 0.787 |
| (100, 20, false) | 0.681 |
| (15000, 20, true) | 0.895 |
| (20000, 20, false) | 0.865 |
Conclusion: Also considering the topic summaries, we can conclude that the number of iterations highly affect the topic diversity, and thus classifier performance. Although we can check convergence of EM algorithm to stop, this migh be problematic due to increasing computation time with the number of iterations.
- The best of (LDA + Classification) where the number of topics is 20 and the number of iterations is 15000 and vs direct Classification on k-mers performance comparision.
| Method | Accuracy |
|---|---|
| (LDA + Classification) | 0.895 |
| Direct Classification | 0.942 |
Conclusion: We couldn't perform higher number of iterations (due to limited time and cluster restarts); however, this result shows that LDA is capable of summarizing k-mer features. Once, we learn the mapping from k-mers to reduced topic distribution space via LDA, then we can use this reduced number of features to train classifier. This makes the data more scalable and may save computation time in the long run.
Our conclusions
- What we have tried and failed? What we have learned?
- Overlapping k-mers with low iteration led poor diversity in topics
- With expectation maximization LDA required iteration increases significantly. Where to stop the iteration becomes a problem due to computation time concerns.
- Changing doc or term concentration did not lead better results
- We tried several number of topics (10, 20, 50) and #topics = 20, which is almost equal to number of aminoacids, yields the best result (coincidence or the number of aminoacids is a good choice for topic number?)
- Using a fixed vocabulary of k-mers (using nucleotide alphabet A-T-G-C) yields poor topic diversity [since there are sequencing errors in genomes, there are different k-mers such as TAK, TAR. And, we concluded that these errors actually give some insight on the virus]
- The comparison between LDA-based classifier and directly-k-mers classifier demonstrates that with enough iterations LDA is capable of summarising the data.
- There is not much difference between overlapping and nonoverlapping k-mer features when we directly give them to classifier. However, for LDA overlapping features require much higher number of iterations for convergence [or topic divergence]
- Very final conclusion: Directly giving k-mers to classifier works better. However, once we learn from LDA with reduced number of features (from k-mers to topic distributions), we can have a good result on classification which can save computation time and make it scalable (by reducing the number of features to the number of topics chosen).
“Everything should be made as simple as possible, but no simpler.”
