Why?

5.4 million people are bitten by snakes every year and 81-138 million people die due to snake bites each year. Preventing snake bites is clearly a major issue, in need of preventative measures to save lives. The project Snaked demonstrates a potential solution to alleviate the problem of struggling to identify the snake which has bitten a person. A proof of concept app is available on the Google Play Store which uses this model.

Challenge

• Each snake species varies in shape, colour, size, texture and more.
• Over 3000 snake species have already been discovered worldwide!
• Different snake species may look nearly identical, however, vary significantly

So it is clearly not an easy job to identify one snake from another, even though it's quintessential that we do.

Frameworks and Methods Utilized

PyTorch is used for ALL deep learning code and Numpy for numerical computations. The code is separated into the main file outlining the chosen algorithms (can be swapped/modified easily) and abstracted code to help train, evaluate and create an executable for any model easily.

There are several novel ideas/techniques used here which help to create neater/more readable pythonic code. The three primary examples of new PyTorch techniques:

• The Item tuple class which allows modular and further extensible code (when extra data needs to be processed)
• Use of super-convergence/the one-cycle policy in pure PyTorch instead of a highly abstracted library (i.e. Fast.AI)/bare python/numpy
• Use of a dictionary to dictate how different data sets should be split up to allow easy modifications of data proportions (i.e. switching between full dataset for training and small batches for ensuring all code runs without errors)

Although several models were trialed out on the dataset, in the end, a MobileNetV2 model provided the best results, whilst also remaining relatively lightweight and so able to run on low-power devices like phones (essential for the app). The final model was trained using LDAM loss instead of cross-entropy loss due to the dataset being imbalanced (some classes having far more samples than others). Note that the codebase has support for classical rebalancing, however, experimental trials show that this method causes overfitting extremely early on. The model manages to achieve around a 70% accuracy and F1 score. More details about the choice of model, how it was improved and lessons learnt from this project can be found on The Data Science Swiss Army Knife blog.

Android Application

The android application created for this project was written with Kotlin, using Fotoapparat (for easy camera support) and PyTorch (for utilizing the chosen PyTorch model) libraries. Due to the lightweight MobileNetV2 model no network is required to connect to a server (which would normally run the computations itself). This is intentionally done to facilitate use within remote locations! Please note that this is only a sample proof of concept app and if you're bitten consult a medical expert immediately.

Sources

All data currently used for the project comes from AIcrowd's Snake Species Identification Challenge. The images and labels have been used, however, geographic locations have been ignored to allow easy usage for any image, even if it hasn't been tagged. The dataset allows 85 species to be labelled. A Jupyter Notebook is also provided which demonstrates how to collect further data using Google Image searches! All statistics used here, or within the repository are from the World Health Organisation (unless otherwise stated) or pertaining specifically to the Snaked source code.

For further information please see the following:

• Snakebite envenoming
• Venomous Snakes Distribution and Species Risk Categories
• AIcrowd Snake Species Identification Challenge

Metrics

Throughout this report, I'll refer to F1 scores as my primary metric. This is because of the class imbalance. Please ensure to note that there is a major difference between the support number and number of present samples for a class. The first refers to the count in the validation dataset, whereas the latter in the training dataset. The number present in the training set will be primarily used to judge the effect of a skewed dataset on model predictions.

Training and Validation Graphs

Train epoch vs accuracy

Train epoch vs loss

Validation epoch vs accuracy

Validation epoch vs loss

Model trained for 51 epochs

Classification Report

species	precision	recall	f1-score	support
agkistrodon-contortrix	0.8452380952380952	0.8765432098765432	0.8606060606060606	81.0
agkistrodon-piscivorus	0.676923076923077	0.6027397260273972	0.6376811594202899	73.0
ahaetulla-prasina	0.8	0.6666666666666666	0.7272727272727272	6.0
arizona-elegans	0.631578947368421	0.5714285714285714	0.6	21.0
boa-imperator	0.6	0.5294117647058824	0.5625	17.0
bothriechis-schlegelii	0.8888888888888888	0.7619047619047619	0.8205128205128205	21.0
bothrops-asper	0.875	0.5384615384615384	0.6666666666666667	13.0
carphophis-amoenus	0.6551724137931034	0.6551724137931034	0.6551724137931034	29.0
charina-bottae	0.8947368421052632	0.7391304347826086	0.8095238095238095	23.0
coluber-constrictor	0.5689655172413793	0.559322033898305	0.5641025641025641	59.0
contia-tenuis	0.52	0.6190476190476191	0.5652173913043478	21.0
coronella-austriaca	0.3333333333333333	0.1	0.15384615384615383	10.0
crotalus-adamanteus	0.6153846153846154	0.7272727272727273	0.6666666666666667	11.0
crotalus-atrox	0.7747252747252747	0.8392857142857143	0.8057142857142857	168.0
crotalus-cerastes	0.7857142857142857	0.8461538461538461	0.8148148148148148	13.0
crotalus-horridus	0.7741935483870968	0.8135593220338984	0.7933884297520662	59.0
crotalus-molossus	0.6363636363636364	0.7	0.6666666666666666	10.0
crotalus-oreganus	0.5333333333333333	0.6666666666666666	0.5925925925925926	12.0
crotalus-ornatus	1.0	0.8181818181818182	0.9	11.0
crotalus-pyrrhus	0.8571428571428571	0.5454545454545454	0.6666666666666665	22.0
crotalus-ruber	0.7894736842105263	0.7142857142857143	0.7500000000000001	21.0
crotalus-scutulatus	0.8620689655172413	0.7575757575757576	0.8064516129032258	33.0
crotalus-viridis	0.5384615384615384	0.6363636363636364	0.5833333333333334	22.0
diadophis-punctatus	0.8266666666666667	0.7948717948717948	0.8104575163398693	78.0
epicrates-cenchria	1.0	0.5	0.6666666666666666	2.0
haldea-striatula	0.5454545454545454	0.47058823529411764	0.5052631578947367	51.0
heterodon-nasicus	0.7	0.5384615384615384	0.608695652173913	13.0
heterodon-platirhinos	0.7058823529411765	0.75	0.7272727272727272	48.0
hierophis-viridiflavus	0.5	0.4375	0.4666666666666667	16.0
hypsiglena-jani	0.5238095238095238	0.6111111111111112	0.5641025641025642	18.0
lampropeltis-californiae	0.8701298701298701	0.8170731707317073	0.8427672955974842	82.0
lampropeltis-getula	0.8333333333333334	0.625	0.7142857142857143	16.0
lampropeltis-holbrooki	0.6923076923076923	0.6428571428571429	0.6666666666666666	14.0
lampropeltis-triangulum	0.8166666666666667	0.8305084745762712	0.8235294117647058	59.0
lichanura-trivirgata	0.9333333333333333	0.8235294117647058	0.8749999999999999	17.0
masticophis-flagellum	0.5294117647058824	0.6428571428571429	0.5806451612903226	42.0
micrurus-tener	1.0	0.95	0.9743589743589743	20.0
morelia-spilota	0.4	0.2222222222222222	0.2857142857142857	9.0
naja-naja	0.8333333333333334	0.7142857142857143	0.7692307692307692	7.0
natrix-maura	0.25	0.125	0.16666666666666666	8.0
natrix-natrix	0.7857142857142857	0.4074074074074074	0.5365853658536585	27.0
natrix-tessellata	0.5	0.36363636363636365	0.4210526315789474	11.0
nerodia-cyclopion	0.5833333333333334	0.4666666666666667	0.5185185185185186	15.0
nerodia-erythrogaster	0.53125	0.4358974358974359	0.47887323943661975	78.0
nerodia-fasciata	0.6190476190476191	0.43333333333333335	0.5098039215686274	30.0
nerodia-rhombifer	0.7213114754098361	0.6567164179104478	0.6875	67.0
nerodia-sipedon	0.5084745762711864	0.5309734513274337	0.5194805194805195	113.0
nerodia-taxispilota	0.8461538461538461	0.55	0.6666666666666667	20.0
opheodrys-aestivus	0.9042553191489362	0.9444444444444444	0.9239130434782609	90.0
opheodrys-vernalis	0.8235294117647058	0.6666666666666666	0.7368421052631577	21.0
pantherophis-alleghaniensis	0.36585365853658536	0.4	0.38216560509554137	75.0
pantherophis-emoryi	0.5714285714285714	0.6666666666666666	0.6153846153846153	36.0
pantherophis-guttatus	0.9183673469387755	0.8035714285714286	0.8571428571428571	56.0
pantherophis-obsoletus	0.5395348837209303	0.6041666666666666	0.5700245700245701	192.0
pantherophis-spiloides	0.43478260869565216	0.25	0.3174603174603175	40.0
pantherophis-vulpinus	0.5957446808510638	0.8484848484848485	0.7	33.0
phyllorhynchus-decurtatus	0.6956521739130435	0.9411764705882353	0.7999999999999999	17.0
pituophis-catenifer	0.7073170731707317	0.7016129032258065	0.7044534412955465	124.0
pseudechis-porphyriacus	0.6666666666666666	0.2857142857142857	0.4	7.0
python-bivittatus	0.875	0.7777777777777778	0.823529411764706	9.0
python-regius	0.0	0.0	0.0	3.0
regina-septemvittata	0.56	0.6666666666666666	0.6086956521739131	21.0
rena-dulcis	0.6666666666666666	0.8	0.7272727272727272	10.0
rhinocheilus-lecontei	0.8292682926829268	0.85	0.8395061728395061	40.0
sistrurus-catenatus	1.0	0.8	0.888888888888889	5.0
sistrurus-miliarius	1.0	0.6666666666666666	0.8	6.0
sonora-semiannulata	0.3333333333333333	0.4	0.3636363636363636	5.0
storeria-dekayi	0.76	0.8465346534653465	0.8009367681498829	202.0
storeria-occipitomaculata	0.631578947368421	0.6486486486486487	0.64	37.0
tantilla-gracilis	0.5454545454545454	0.6	0.5714285714285713	10.0
thamnophis-cyrtopsis	0.5714285714285714	0.4444444444444444	0.5	9.0
thamnophis-elegans	0.47058823529411764	0.2962962962962963	0.3636363636363636	27.0
thamnophis-hammondii	0.5555555555555556	0.35714285714285715	0.43478260869565216	14.0
thamnophis-marcianus	0.8787878787878788	0.8285714285714286	0.8529411764705883	35.0
thamnophis-ordinoides	0.375	0.16666666666666666	0.23076923076923078	18.0
thamnophis-proximus	0.7543859649122807	0.7413793103448276	0.7478260869565219	58.0
thamnophis-radix	0.8181818181818182	0.47368421052631576	0.6	38.0
thamnophis-sirtalis	0.7167070217917676	0.896969696969697	0.7967698519515478	330.0
tropidoclonion-lineatum	0.7777777777777778	0.6363636363636364	0.7000000000000001	11.0
vermicella-annulata	0.0	0.0	0.0	3.0
vipera-aspis	0.4	0.5714285714285714	0.47058823529411764	7.0
vipera-berus	0.75	0.75	0.75	12.0
virginia-valeriae	0.2	0.14285714285714285	0.16666666666666666	7.0
xenodon-rabdocephalus	1.0	1.0	1.0	8.0
zamenis-longissimus	0.4	0.3333333333333333	0.3636363636363636	12.0
accuracy	0.6930662557781202	0.6930662557781202	0.6930662557781202	0.6930662557781202
macro avg	0.6659430597272402	0.6050948460385794	0.6247619446039016	3245.0
weighted avg	0.6925634979734435	0.6930662557781202	0.6866874083865039	3245.0

Despite a skewed dataset, the majority of snakes had a similar precision and recall score.

Number of Samples vs F1 score

The above graph of the number of samples the neural network was trained over and F1 score per class indicates two significant points:

• Having more samples of a class will increase the probability of a model robustly identifying the snake species correctly
• However, this does not mean that an inability to collect a large number of images for all classes will create a major imbalance in a model's predictions

The latter takeaway shows that LDAM loss has been largely successful! On a side note though, when a class has few samples, the F1 score may not be completely representative of how the model will generalise. This is primarily because, in a small number of images, only a small number of conditions can be shown and tested. Yet, in reality, an image can be taken in any environment and transformed in a very large number of ways.

Confusion Matrix

The test dataset had too few images to clearly judge anything from the confusion matrix. This is due to a combination of having 85 classes, and classes with few samples being quite dark.

Improvements

If this model was going to be retrained, a good idea would be to use a 90-5-5, or better a 80-10-10 split. This would ensure that there would be enough data in the testing dataset to judge where exactly the model's confusion stems from (using a confusion matrix).

Combining the current classification model with an additional preprocessing pipeline which includes segmentation may also allow far higher F1 scores. This is because segmentation is able to remove extraneous noise around a snake (subtle environmental cues which prevent the model from generalising). Additionally, this would completely avoid the chance of a snake being cropped out of an image (currently unlikely but still possible for small snakes).