Thank you for your code, Dan! And sorry for my delayed reply – I was too busy analyzing Trump tweets . Some comments follow.
1. I used NNs based image classifiers for couple of my consultancy projects. In my opinion, the use of NNs methods is underestimated in image classification.
2. Using the function CrossTabulate I visualized the confusion matrices for several experiments using different values for keep. (The variable keep is not defined in your code.) Pretty good overall accuracy is achieved with keep = 40 and reasonably fast.
3. Instead of using Sharpen I was thinking to do the opposite – use Gaussian blur.
4. It might be a good idea to look in ROC application to your approach. (I might do that later this week.)
Here is experimental code for over a set of keep values:
Table[(
trainingTime =
AbsoluteTiming[
{nf, vv} =
nearestImages[trainingImages, trainingLabels, dn, dst, keep];
testvecs = processInput[testImages, vv, dn, dst];
][[1]];
classificationTime =
AbsoluteTiming[guessed = guesses[nf, testvecs, 5];][[1]];
ColumnForm[{
Row[{"keep = ", keep}],
Row[{"Training time, (s): ", trainingTime}],
Row[{"Classification time, (s): ", classificationTime}],
Row[{"Overall accuracy: ",
correct[guessed, testLabels]/Length[testLabels] // N}],
MatrixForm@CrossTabulate[Transpose[{guessed, testLabels}]]
}]
), {keep, {5, 10, 15, 20, 30, 40, 60, 100}}]
And here is the output:
Note that increasing keep after 30 does not produce significantly better results.