What this project is
SkinScan is a research and learning project. Given a face photo, the system finds visible skin concerns, maps them to face regions, estimates skin tone, and recommends products through reviewed rules.
It is not medical software. The language is cosmetic-concern only. There is no diagnosis.
The production path today is: native 1024px tiles of the photo, a required external SA-RPN HTTP service for lesion identification, MediaPipe face regions, ITA-based tone estimation, then a v3 care decision and product eligibility stack. There is no local-model fallback if the SA-RPN service fails.
This post walks through the work in the order I did it, including the paths that failed.
Step 0. Contracts before models
Before training anything, I locked the data contracts. The recommender needed a fixed vocabulary so it would never chase moving model outputs.
I defined a closed concern list, a closed face-region list, ordinal severity from 0 to 4, and a catalog schema. The CV stages were then built to fill those contracts.
- Concerns: acne_comedonal, acne_inflammatory, acne_cystic, acne_scarring, hyperpigmentation, dryness
- Regions: forehead, nose, left_cheek, right_cheek, chin_jaw, perioral
- Datasets: ACNE04 for detection, Kaggle acne types for classification, FFHQ for negatives, Sephora products and reviews for ranking, self photos for test only
Step 1. Lesion detector (YOLOv8)
I trained a single-class lesion detector on ACNE04. That dataset has 1,457 dermatologist-boxed faces and 18,983 boxes. Training started from COCO weights on a Colab T4.
I began with YOLOv8-nano. Recall on small dense lesions was weak. The model was too small. I upgraded to YOLOv8m. That became decision D-018.
The first check was always visual label review, not metrics. After that I swept confidence thresholds and locked an operating point that favored recall over precision. A missed spot is gone forever. An extra crop can still be rejected later.
- Locked point: conf 0.07, IoU 0.2, imgsz 1024
- Result: precision 0.697, recall 0.750, F1 0.722
Step 2a. First classifier attempt, self-labeled crops
I trained a first classifier on detector crops from ACNE04 that I labeled myself. Classes were comedonal, cystic, inflammatory, not_acne, and post_acne_mark.
It did not hold up. The confusion matrix diagonal was weak almost everywhere. Inflammatory recall was 20 of 56. Cystic recall was 10 of 30.
The error audit showed why. At crop scale, fading inflammatory lesions and post-acne marks look nearly the same. Self-labeled concern-level crops were too noisy.
One class did work. Not_acne was the strongest row at 23 of 36. That was the first evidence that a learned reject class was viable. Those negatives came from harvested detector false positives: hair, shadows, fabric, and plain skin.
Step 2b. Second classifier attempt, Kaggle typed dataset
I switched to the Kaggle acne-type dataset. Five real types: Blackheads, Cyst, Papules, Pustules, Whiteheads. I retrained EfficientNetB0 on a Colab T4.
Again I reviewed labels first. Training used Adam at 1e-5 for 150 epochs with a best-validation checkpoint. Split sizes were train 2778, valid 921, test 918.
Test accuracy was about 92 percent. Macro F1 was 0.92 to 0.93. The diagonal was clean. Papules and Pustules remained the soft spot because both look inflammatory. Whiteheads had the smallest test set but scored highest.
I then wired detector to context crop to classifier. On a self-collected photo with real acne, the detector found lesions and the classifier typed them as pustules. That matched the photo. Sixteen detections, all Pustules, confidence from 0.47 to 1.00.
The remaining worry was structural. A 5-way softmax still cannot say none of these. Weak detector boxes on shadows, pores, or hair get forced onto an acne type. That is what Not_acne was supposed to fix next.
Step 3. Adding Not_acne. Right idea, wrong dataset
Not_acne had already worked on the small self-labeled model, so I brought it to the Kaggle model as a sixth class. Design analysis preferred a sixth class over a softmax probability threshold, which is miscalibrated on out-of-distribution crops, and over a two-stage binary gate, which was over-engineered.
Negatives were FFHQ clear-skin detector false positives plus non-lesion ACNE04 regions, harvested through the same crop_with_context transform the pipeline uses.
On paper the 6-class retrain looked perfect. Test accuracy 91.72 percent. Macro F1 0.93. Only 2 of 918 real lesion crops were misrouted to Not_acne. FFHQ clear-skin reject rate was 99.7 percent.
Then the end-to-end run fell apart. That became decision D-025. On real faces the model classified essentially every detector crop as Not_acne, including crops that were unmistakably pustules, at confidence 1.00.
Root cause: a crop-domain confound. The acne positives were 640 by 640 Roboflow mosaic images. The negatives were 224px upscaled pipeline crops. The model learned crop style, not acne. Every dataset-level acceptance gate still passed, because none of them fed real pipeline crops of a known-acne face through the model.
- Lesson: dataset-level metrics can be perfect while the deployed model is fully broken
- Fix: reverted weights to the 5-class model
- New gate: Not_acne share on a known-acne image must stay under 50 percent
Step 4. Face regions and skin tone
MediaPipe FaceLandmarker with 468 landmarks assigns each lesion to a face region by point-in-polygon. That became decision D-020.
Skin tone is estimated by ITA in CIELAB over non-lesional forehead and cheek pixels, bucketed into light, medium, or deep. Self-report always overrides the photo estimate. That became decision D-021.
I also wrote a fairness-eval design for disaggregating pooled scalars by Fitzpatrick group. ACNE04 ships no tone labels, so ITA is the estimator for that work.
Step 5. Recommendation layer
The recommendation stack went through several versions. The current v3 path is decision, therapy plan, hard eligibility, ranking, composition, then whole-regimen validation.
Triage and therapy disposition are independent. High counts, scarring, or pigment concern may add review language without suppressing active treatment. Raw detector confidence is never treated as a calibrated probability. An uncalibrated nodule signal causes an explicit abstention.
Missing reviewed policy or verified catalog data is represented as deferred or unavailable. The system does not guess. Production ships no clinician-reviewed therapy policy, so active therapy is deferred by default.
Ranking is subordinate to eligibility. Concern-specific outcome evidence, tolerability, evidence completeness, and usable budget come first. Pooled review statistics are only a final tie-break among already eligible equivalents.
Step 5b. Learned ranker lost to a simple baseline
I trained a HistGradientBoosting ranker on about 1.1 million Sephora reviews. Pairwise ordering score was 0.584.
A Bayesian-smoothed rating baseline scored 0.609. Global popularity scored 0.597. The learned model lost the bake-off.
I also tried per-skin-type ranking cells. Those scored 0.606 and 0.596, worse than the pooled baseline. The cells were too sparse.
Seven probes confirmed the structural issue. Product-anonymous features cannot beat per-product memorized stats on this setup. I shipped the StatsRanker champion. That became decision D-022. The learned model never shipped.
| Method | ROC-AUC | Pairwise ordering |
|---|---|---|
| Learned HistGB | 0.659 | 0.584 |
| Global popularity | 0.672 | 0.597 |
| Bayesian-smoothed rating | 0.666 | 0.609 |
Step 6. SA-RPN detector
I researched Zhang et al., Learning High-quality Proposals for Acne Detection. SA-RPN adds a spatial-aware RPN with NWD proposal scoring and deformable convolutions on Mask R-CNN R50-FPN.
I replicated it on AcneSCU with MMDetection. 275 clinical faces were tiled into 1024 by 1024 crops, giving 4,680 train and 529 val tiles. Fine-tuning from COCO ran for 15 epochs on a Lightning AI A100.
Unlike the YOLOv8m detector, which was a single lesion class, SA-RPN predicts 10 fine-grained lesion types with per-lesion masks.
Validation mAP jumped from about 0.43 to 0.50 exactly at the epoch-9 learning-rate drop, then plateaued. The model converged. More epochs at that recipe would not help.
- Caveat: strict-IoU localization is weak. mAP at 0.75 is about 0.08. The next lever is anchor and mask-head tuning, not more training.
- Caveat: the available dataset mirror lacks patient IDs, so the paper patient-disjoint split cannot be reproduced exactly. The split is image-level.
| Metric | bbox | segm | Paper target |
|---|---|---|---|
| mAP @ 0.50 | 0.499 | 0.498 | AP 0.507 |
| Recall @ 0.50 | 0.743 | 0.742 | AR 0.775 |
| mAP @ [.50:.95] | 0.174 | 0.169 | — |
Step 7. Pipeline A/B: how should SA-RPN see a full photo?
The model trains on 1024px tiles of clinical photos. A full face photo has to be funneled into that input somehow. I A/B tested two funnels on the same served checkpoint against 5 held-out validation images with 756 annotated lesions, scored at IoU at least 0.3.
Zoom funnel: the historical YOLOv8m detector finds acne areas. Each cluster is cropped and upscaled to 1024px. SA-RPN re-detects inside those crops.
Tile funnel: the photo is chunked into native-resolution 1024px tiles with guaranteed minimum overlap. Every tile runs. Seams are deduped client-side.
| Funnel | Recall | Precision | Exact-label acc | Concern acc |
|---|---|---|---|---|
| Zoom | 4% (32/756) | 52% | 44% | 62% |
| Tile | 70% (530/756) | 68% | 92% | 95% |
On one proof image with 282 ground-truth lesions, zoom surfaced 8 and tile surfaced 254 with 91 percent correct types.
Zoom fails twice. YOLO at imgsz 1024 downscales a 3448 by 4600 photo by about 4x, so the gatekeeper forwards almost nothing. Then the upscaled blurry crops cut SA-RPN label accuracy roughly in half versus native pixels.
Decision: native-resolution tiling. Locked as D-026. Production cutover to SA-RPN as the sole default was locked as D-027. The YOLOv8m plus EfficientNetB0 path stays in the repo as a historical evaluation-only reference. The default CLI no longer calls it.
Step 8. Mapping SA-RPN labels to concerns
After tiling and detection, labels map into the locked concern vocabulary. Comedones map to acne_comedonal. Papules and pustules map to acne_inflammatory. Nodules map to acne_cystic. Atrophic and hypertrophic scars map to acne_scarring. Melasma maps to hyperpigmentation. That was the first CV path to emit a non-acne concern.
Nevus and other are never concerns. They surface as visible safety observations, gated by per-label count and confidence thresholds. They are not silently dropped.
Step 9. Still in progress: Stage 2 v2 dataset
The D-025 confound motivated a Stage 2 v2 dataset plan. Auditing the Roboflow training data revealed a second problem. Augmented copies of the same source face leak across train, val, and test. For Blackheads, there were 735 train files but only about 91 source images, and 61 of those appeared in all three splits.
The v2 plan harvests crops by running the deployed detector on AcneSCU faces so train crops match deployment crops. Every label gets human review. Splits are by source face before any augmentation. Seven gates must pass before v2 replaces the shipped 5-class model.
Wrong paths kept on the record
Half the value of the project is the dead ends. I kept them on purpose.
| Dead end | Symptom | Root cause | Resolution |
|---|---|---|---|
| YOLOv8-nano | Weak recall on small dense lesions | Model too small | Upgraded to YOLOv8m |
| Self-labeled classifier | Weak diagonal; inflammatory confused with post-acne marks | Noisy hand labels; crop too small for concern-level classes | Switched to typed Kaggle dataset |
| 6-class retrain v1 | Real pustule crops classified Not_acne at confidence 1.00 | Crop-domain confound between mosaic positives and pipeline-crop negatives | Reverted to 5-class; added real-pipeline gate |
| Roboflow split trust | Same source face in train and test | Augmentation done before splitting | v2 splits by source face first |
| Learned product ranker | Pairwise 0.584 vs baseline 0.609 | Product-anonymous features cannot beat memorized product stats | Shipped StatsRanker |
| Per-skin-type ranking cells | Worse than pooled baseline | Cells too sparse | Pooled stats; cells evidence-only |
| Zoom funnel into SA-RPN | Recall 4% vs tiling 70% | YOLO downscales hi-res photos; upscaled crops blur detail | Native-res tiling locked |
What I take away
Every proxy metric can pass while the real pipeline is broken. That is why each stage now has a visual proof sheet and a real-pipeline acceptance gate, not just held-out accuracy.
Contracts first helped. The recommender never had to chase moving model outputs because the concern and region vocabularies were locked early.
Simpler baselines can beat learned models when the feature setup is wrong for the task. Ranking only after hard eligibility kept recommendations from repairing bad catalog or policy data.
Production identification depends on a required SA-RPN service with no local fallback. Detector-only recall at 0.50 is about 0.743. There is still no single measured end-to-end accuracy number for the full SA-RPN production pipeline. That gap is recorded as D-027.
The work continues on the Stage 2 v2 dataset, localization tuning for SA-RPN, and external release gates that code and synthetic fixtures cannot satisfy alone.