Johns Hopkins case study

TechTrack

Real-time object detection for warehouse robots, on CPU.

Course
EN.705.603 Creating AI-Enabled Systems, Johns Hopkins, fall 2026
Domain
Computer vision for warehouse robotics
Stack
Python, OpenCV, YOLOv4-tiny, Docker
01

The problem

TechTrack is the first full case study in EN.705.603 (Creating AI-Enabled Systems), which I am taking at Johns Hopkins in fall 2026 as part of my MS in Artificial Intelligence. The scenario is a fleet of warehouse robots that have to find and label twenty kinds of objects (people, forklifts, pallets, boxes, safety gear, vehicles, signs) in a 30 frames per second camera feed, on CPU only, with a human review loop that turns the model's mistakes into retraining data.

The robot's planner stops or reroutes on every box it receives. A missed person is the most expensive error the system can make, and I set every threshold in the analysis against person recall first.

02

What I built

I built the inference service in Python and OpenCV, then the evaluation and retraining pieces around it.

  • Preprocessing, a YOLOv4-tiny detector, and non-maximum suppression (NMS), all running on CPU
  • IoU matching, per-class precision-recall curves, and 11-point mean average precision (mAP)
  • The YOLO loss, split into its localization, objectness, no-object, and classification terms
  • A hard negative miner that ranks images by that loss
  • Augmentation transforms (blur, brightness, contrast, flips) and the Docker packaging
03

The design analysis

Both candidate models arrived trained, so the analysis tunes the system around them rather than the networks. I ran each model once over the 9,525-image logistics dataset, cached every candidate box above 0.01 objectness, and made each later experiment a re-read of that cache. An evaluation went from two minutes to about two seconds on my laptop.

  1. Model selection

    Model 2 led model 1 on mAP (0.583 against 0.544), was ahead or level on all 20 classes, and kept the lead under every match IoU and score threshold I tried. I set the deployed objectness threshold at 0.25, where person recall stays within 0.02 of its best and the planner gets 3.5 boxes per frame instead of 8.6.

    Per-class AP for model 1 and model 2 Dumbbell chart of 20 classes. Model 2 is ahead or level on every class; the largest gains are forklift, barcode, person, smoke and freight container. QR code is the one class where model 1 is ahead, by 0.001. 0.0 0.2 0.4 0.6 0.8 1.0 AP at match IoU 0.5 wood pallet wood pallet: model 1 0.273, model 2 0.307, 9,330 boxes person person: model 1 0.511, model 2 0.588, 6,368 boxes +0.08 cardboard box cardboard box: model 1 0.723, model 2 0.782, 4,995 boxes +0.06 fire fire: model 1 0.253, model 2 0.315, 2,793 boxes +0.06 helmet helmet: model 1 0.470, model 2 0.528, 2,170 boxes +0.06 smoke smoke: model 1 0.455, model 2 0.528, 1,495 boxes +0.07 car car: model 1 0.674, model 2 0.684, 1,379 boxes safety vest safety vest: model 1 0.562, model 2 0.586, 1,260 boxes traffic light traffic light: model 1 0.782, model 2 0.793, 1,193 boxes forklift forklift: model 1 0.695, model 2 0.778, 1,103 boxes +0.08 truck truck: model 1 0.765, model 2 0.768, 782 boxes van van: model 1 0.784, model 2 0.790, 765 boxes road sign road sign: model 1 0.213, model 2 0.219, 720 boxes traffic cone traffic cone: model 1 0.681, model 2 0.691, 506 boxes qr code qr code: model 1 0.760, model 2 0.759, 369 boxes model 1 ahead, 0.001 license plate license plate: model 1 0.353, model 2 0.414, 359 boxes +0.06 freight container freight container: model 1 0.236, model 2 0.305, 318 boxes +0.07 barcode barcode: model 1 0.676, model 2 0.754, 283 boxes +0.08 ladder ladder: model 1 0.385, model 2 0.436, 277 boxes +0.05 gloves gloves: model 1 0.630, model 2 0.636, 256 boxes
    model 1model 2
    I measured per-class AP at a 0.5 match IoU on all 9,525 images and ordered the classes by how many labeled boxes each one has, with the ten classes where model 2 gains more than 0.05 drawn heavier so the shape of the lead is visible at a glance. The gains land where I need them. Person and helmet and forklift matter for safety and cardboard box and freight container matter for the shipment, while the vehicle classes and the small markers sit within 0.01 of each other. QR code is the one class model 1 wins, by a rounding error.
    Show the numbers
    classboxesAP model 1AP model 2delta
    wood pallet9,3300.2730.307+0.034
    person6,3680.5110.588+0.077
    cardboard box4,9950.7230.782+0.059
    fire2,7930.2530.315+0.062
    helmet2,1700.4700.528+0.058
    smoke1,4950.4550.528+0.073
    car1,3790.6740.684+0.010
    safety vest1,2600.5620.586+0.024
    traffic light1,1930.7820.793+0.011
    forklift1,1030.6950.778+0.083
    truck7820.7650.768+0.003
    van7650.7840.790+0.006
    road sign7200.2130.219+0.006
    traffic cone5060.6810.691+0.010
    qr code3690.7600.759-0.001
    license plate3590.3530.414+0.061
    freight container3180.2360.305+0.069
    barcode2830.6760.754+0.078
    ladder2770.3850.436+0.051
    gloves2560.6300.636+0.006
    mAP36,7210.5440.583+0.039
    mAP under three match IoU and three score thresholds, both models Two three by three grids. Model 2 is darker than model 1 in every cell; tightening the match IoU from 0.5 to 0.7 costs about 0.16 mAP and raising the score threshold to 0.5 costs 0.06 to 0.08. model 1 score 0.05 score 0.25 score 0.50 match IoU 0.3 model 1, match IoU 0.3, score 0.05: mAP 0.5850.585 model 1, match IoU 0.3, score 0.25: mAP 0.5800.580 model 1, match IoU 0.3, score 0.50: mAP 0.4850.485 match IoU 0.5 model 1, match IoU 0.5, score 0.05: mAP 0.5440.544 model 1, match IoU 0.5, score 0.25: mAP 0.5490.549 model 1, match IoU 0.5, score 0.50: mAP 0.4620.462 match IoU 0.7 model 1, match IoU 0.7, score 0.05: mAP 0.3850.385 model 1, match IoU 0.7, score 0.25: mAP 0.3930.393 model 1, match IoU 0.7, score 0.50: mAP 0.3640.364 model 2 score 0.05 score 0.25 score 0.50 model 2, match IoU 0.3, score 0.05: mAP 0.6120.612 model 2, match IoU 0.3, score 0.25: mAP 0.6110.611 model 2, match IoU 0.3, score 0.50: mAP 0.5410.541 model 2, match IoU 0.5, score 0.05: mAP 0.5830.583 model 2, match IoU 0.5, score 0.25: mAP 0.5710.571 model 2, match IoU 0.5, score 0.50: mAP 0.5240.524 model 2, match IoU 0.7, score 0.05: mAP 0.4270.427 model 2, match IoU 0.7, score 0.25: mAP 0.4230.423 model 2, match IoU 0.7, score 0.50: mAP 0.4020.402
    I ran the full dataset through a grid of three match IoU values and three score thresholds before I chose a model, because a single mAP figure is a choice of two thresholds as much as a property of the network. Model 2 is darker in every cell. Tightening the match IoU from 0.5 to 0.7 costs about 0.16 mAP and raising the score threshold to 0.5 costs 0.06 to 0.08, so I state both settings every time I report a number.
    Show the numbers
    match IoUscore thresholdmodel 1model 2
    0.30.050.5850.612
    0.30.250.5800.611
    0.30.500.4850.541
    0.50.050.5440.583
    0.50.250.5490.571
    0.50.500.4620.524
    0.70.050.3850.427
    0.70.250.3930.423
    0.70.500.3640.402
    Precision and recall against the objectness threshold Two panels, person and all 20 classes. As the threshold rises from 0.05 to 0.5 precision climbs and recall falls; at the chosen 0.25 person recall is 0.620 against 0.638 at 0.05 and person precision is 0.673. person 0.4 0.5 0.6 0.7 0.8 0.05 0.25 0.50 objectness threshold deployed 0.25 person, threshold 0.05: precision 0.637 person, threshold 0.10: precision 0.637 person, threshold 0.25: precision 0.673 person, threshold 0.50: precision 0.808 0.673 person, threshold 0.05: recall 0.638 person, threshold 0.10: recall 0.638 person, threshold 0.25: recall 0.620 person, threshold 0.50: recall 0.489 0.620 all 20 classes 0.4 0.5 0.6 0.7 0.8 0.05 0.25 0.50 objectness threshold deployed 0.25 all 20 classes, threshold 0.05: precision 0.591 all 20 classes, threshold 0.10: precision 0.591 all 20 classes, threshold 0.25: precision 0.634 all 20 classes, threshold 0.50: precision 0.786 0.634 all 20 classes, threshold 0.05: recall 0.586 all 20 classes, threshold 0.10: recall 0.586 all 20 classes, threshold 0.25: recall 0.564 all 20 classes, threshold 0.50: recall 0.453 0.564
    precisionrecall
    I measured precision and recall for model 2 against the objectness threshold on the full dataset at NMS 0.5, with person in the left panel and all 20 classes in the right, and I wrote the selection rule down before I read the numbers. Deploy at the highest threshold that keeps person recall within 0.02 of its value at 0.05. That is 0.25. There the planner gets 3.5 boxes per frame instead of 8.6 and almost none of them arrive without a class label.
    Show the numbers
    thresholdperson Pperson Rall Pall Rboxes per image
    0.050.6370.6380.5910.5868.58
    0.100.6370.6380.5910.5865.78
    0.250.6730.6200.6340.5643.46
    0.500.8080.4890.7860.4532.22
    Boxes per image the service would emit at each objectness threshold Stacked columns. At 0.05 the service emits 8.6 boxes per image and 4.8 of them carry no class label; at 0.25 it emits 3.5 with 0.03 unlabeled, and person recall is 0.620 there against 0.638 at 0.05. 0 2 4 6 8 10 threshold 0.05: 8.58 boxes per image, 4.76 unlabeled, person recall 0.638 8.6 0.05 0.64 threshold 0.10: 5.78 boxes per image, 1.95 unlabeled, person recall 0.638 5.8 0.10 0.64 threshold 0.20: 3.93 boxes per image, 0.11 unlabeled, person recall 0.638 3.9 0.20 0.64 threshold 0.25: 3.46 boxes per image, 0.03 unlabeled, person recall 0.620 3.5 0.25 0.62 deployed threshold 0.30: 3.11 boxes per image, 0.02 unlabeled, person recall 0.595 3.1 0.30 0.59 threshold 0.50: 2.22 boxes per image, 0.0 unlabeled, person recall 0.489 2.2 0.50 0.49 threshold person recall boxes per image
    boxes with a class labelboxes with no class label
    I counted the boxes the service would hand the planner at each objectness threshold on the full dataset, and how many of those boxes would arrive with no class label because the Darknet importer zeroes class scores below 0.2 objectness. At 0.05 half the boxes are unlabeled. At 0.25 the planner gets 3.5 boxes per image with 0.03 of them unlabeled and person recall is still 0.620, which is why I deploy there rather than at 0.05 or 0.5.
    Show the numbers
    objectness thresholdboxes per imageunlabeled boxes per imageperson recall
    0.058.584.760.638
    0.105.781.950.638
    0.203.930.110.638
    0.253.460.030.620
    0.303.110.020.595
    0.502.220.000.489

  2. Sampling strategy

    I drew a 5,000-image working sample stratified by the rarest class in each image, using label counts only. It keeps every ladder, gloves, and freight container image. On the ten rarest classes its AP error against the full set is half that of a uniform sample for model 2.

    Share of each class's images kept by the stratified and the uniform sample Dumbbell chart of 20 classes ordered from rarest to most common by image count. The stratified sample keeps every ladder, freight container and gloves image and thins the common classes to 27 to 63 percent, while the uniform sample keeps about half of every class. 0% 25% 50% 75% 100% share of the class's images kept ladder ladder: 183 images in full set, 183 stratified (100%), 86 uniform (47%) 183 of 183 freight container freight container: 192 images in full set, 192 stratified (100%), 98 uniform (51%) 192 of 192 gloves gloves: 226 images in full set, 226 stratified (100%), 115 uniform (51%) 226 of 226 barcode barcode: 272 images in full set, 263 stratified (97%), 144 uniform (53%) traffic cone traffic cone: 291 images in full set, 268 stratified (92%), 175 uniform (60%) license plate license plate: 292 images in full set, 286 stratified (98%), 155 uniform (53%) qr code qr code: 299 images in full set, 278 stratified (93%), 162 uniform (54%) road sign road sign: 372 images in full set, 331 stratified (89%), 210 uniform (56%) cardboard box cardboard box: 443 images in full set, 330 stratified (74%), 211 uniform (48%) safety vest safety vest: 479 images in full set, 297 stratified (62%), 254 uniform (53%) helmet helmet: 631 images in full set, 485 stratified (77%), 329 uniform (52%) wood pallet wood pallet: 689 images in full set, 292 stratified (42%), 370 uniform (54%) van van: 695 images in full set, 326 stratified (47%), 360 uniform (52%) truck truck: 706 images in full set, 279 stratified (40%), 390 uniform (55%) traffic light traffic light: 866 images in full set, 370 stratified (43%), 429 uniform (50%) forklift forklift: 949 images in full set, 303 stratified (32%), 487 uniform (51%) smoke smoke: 975 images in full set, 263 stratified (27%), 538 uniform (55%) car car: 1000 images in full set, 456 stratified (46%), 534 uniform (53%) fire fire: 1133 images in full set, 361 stratified (32%), 603 uniform (53%) person person: 1933 images in full set, 1220 stratified (63%), 1001 uniform (52%)
    stratified by rarest classuniform random
    I bucketed every image under the rarest class it contains and capped each bucket at 263 images, which a binary search found as the cap that lands the total on 5,000, and I compared that rule with a uniform draw of the same size. The stratified rule keeps every ladder, freight container and gloves image. It thins person to 63 percent and smoke to 27 percent, while the uniform draw keeps about half of every class and so holds 86 ladder images against my 183.
    Show the numbers
    classimages (full)images (stratified)images (uniform)share kept (stratified)share kept (uniform)
    ladder183183861.000.47
    freight container192192981.000.51
    gloves2262261151.000.51
    barcode2722631440.970.53
    traffic cone2912681750.920.60
    license plate2922861550.980.53
    qr code2992781620.930.54
    road sign3723312100.890.56
    cardboard box4433302110.740.48
    safety vest4792972540.620.53
    helmet6314853290.770.52
    wood pallet6892923700.420.54
    van6953263600.470.52
    truck7062793900.400.55
    traffic light8663704290.430.50
    forklift9493034870.320.51
    smoke9752635380.270.55
    car1,0004565340.460.53
    fire1,1333616030.320.53
    person1,9331,2201,0010.630.52

  3. NMS threshold

    A sweep from 0.1 to 0.9 found a flat top between 0.4 and 0.6 that held on the 4,525 images outside the sample and at the score thresholds the service would run, so I kept 0.5 in the middle of the plateau.

    mAP against the NMS IoU threshold Line chart. mAP rises from 0.545 at an NMS threshold of 0.1 to a flat top of 0.572 at 0.5 and 0.571 at 0.6, then falls to 0.468 at 0.9. 0.600.550.500.45 0.10.20.30.40.50.60.70.80.9 NMS IoU threshold plateau 0.4 to 0.6 NMS 0.1: mAP 0.545 NMS 0.2: mAP 0.548 NMS 0.3: mAP 0.564 NMS 0.4: mAP 0.567 NMS 0.5: mAP 0.572 NMS 0.6: mAP 0.571 NMS 0.7: mAP 0.559 NMS 0.8: mAP 0.527 NMS 0.9: mAP 0.468 0.572 0.468
    mAP at a 0.5 match IoU against the NMS threshold, model 2, on the 5,000-image sample with a 0.05 score threshold.
    Show the numbers
    NMS IoUmAP
    0.10.545
    0.20.548
    0.30.564
    0.40.567
    0.50.572
    0.60.571
    0.70.559
    0.80.527
    0.90.468
    mAP against the NMS threshold at three score thresholds Three lines. At every score threshold the best NMS value is 0.5, with 0.6 within 0.002 to 0.005 of it, and the whole curve shifts down as the score threshold rises from 0.05 to 0.5. 0.50 0.52 0.54 0.56 0.58 0.3 0.4 0.5 0.6 0.7 NMS IoU threshold score 0.05, NMS 0.3: mAP 0.564 score 0.05, NMS 0.4: mAP 0.567 score 0.05, NMS 0.5: mAP 0.572 score 0.05, NMS 0.6: mAP 0.571 score 0.05, NMS 0.7: mAP 0.559 score 0.05 0.572 score 0.25, NMS 0.3: mAP 0.555 score 0.25, NMS 0.4: mAP 0.562 score 0.25, NMS 0.5: mAP 0.565 score 0.25, NMS 0.6: mAP 0.563 score 0.25, NMS 0.7: mAP 0.551 score 0.25 0.565 score 0.50, NMS 0.3: mAP 0.506 score 0.50, NMS 0.4: mAP 0.510 score 0.50, NMS 0.5: mAP 0.520 score 0.50, NMS 0.6: mAP 0.518 score 0.50, NMS 0.7: mAP 0.512 score 0.50 0.520
    score 0.05score 0.25score 0.50
    I repeated the NMS sweep at the two score thresholds the service could run, because the first sweep used the 0.05 evaluation score and I wanted the choice to hold where the service operates. The best NMS value is 0.5 at every score threshold. The 0.6 setting is within 0.005 of it everywhere and the difference in boxes per image shrinks to a quarter of a box at the deployed score, so the choice between the two is immaterial on every measure I have.
    Show the numbers
    score thresholdNMS 0.3NMS 0.4NMS 0.5NMS 0.6NMS 0.7boxes per image at 0.5at 0.6
    0.050.5640.5670.5720.5710.5598.9510.46
    0.250.5550.5620.5650.5630.5513.774.02
    0.500.5060.5100.5200.5180.5122.502.57
    Per-class AP against the NMS threshold for seven classes Seven small panels on a shared zero to 0.8 axis. Person and helmet gain from 0.1 to 0.3 and hold; car and gloves are flat until 0.7 then fall; wood pallet barely moves at any setting; ladder peaks at 0.5. wood pallet 13.5 per image containing it 0.0 0.4 0.8 0.1 0.5 0.9 wood pallet, NMS 0.1: AP 0.274 wood pallet, NMS 0.3: AP 0.278 wood pallet, NMS 0.5: AP 0.269 wood pallet, NMS 0.7: AP 0.281 wood pallet, NMS 0.9: AP 0.252 cardboard box 11.3 per image containing it 0.0 0.4 0.8 0.1 0.5 0.9 cardboard box, NMS 0.1: AP 0.730 cardboard box, NMS 0.3: AP 0.734 cardboard box, NMS 0.5: AP 0.729 cardboard box, NMS 0.7: AP 0.721 cardboard box, NMS 0.9: AP 0.693 person 3.3 per image containing it 0.0 0.4 0.8 0.1 0.5 0.9 person, NMS 0.1: AP 0.545 person, NMS 0.3: AP 0.599 person, NMS 0.5: AP 0.598 person, NMS 0.7: AP 0.580 person, NMS 0.9: AP 0.499 helmet 3.4 per image containing it 0.0 0.4 0.8 0.1 0.5 0.9 helmet, NMS 0.1: AP 0.449 helmet, NMS 0.3: AP 0.524 helmet, NMS 0.5: AP 0.524 helmet, NMS 0.7: AP 0.516 helmet, NMS 0.9: AP 0.488 car 1.4 per image containing it 0.0 0.4 0.8 0.1 0.5 0.9 car, NMS 0.1: AP 0.601 car, NMS 0.3: AP 0.599 car, NMS 0.5: AP 0.598 car, NMS 0.7: AP 0.594 car, NMS 0.9: AP 0.351 gloves 1.1 per image containing it 0.0 0.4 0.8 0.1 0.5 0.9 gloves, NMS 0.1: AP 0.638 gloves, NMS 0.3: AP 0.637 gloves, NMS 0.5: AP 0.637 gloves, NMS 0.7: AP 0.637 gloves, NMS 0.9: AP 0.547 ladder 1.5 per image containing it 0.0 0.4 0.8 0.1 0.5 0.9 ladder, NMS 0.1: AP 0.332 ladder, NMS 0.3: AP 0.394 ladder, NMS 0.5: AP 0.442 ladder, NMS 0.7: AP 0.376 ladder, NMS 0.9: AP 0.328 x axis: NMS IoU y axis: AP at 0.5 boxes per image containing the class
    I plotted AP against the NMS threshold for seven classes chosen to span the crowding range, from wood pallet at 13.5 boxes per image where it appears down to gloves at 1.1, and I read the panels for where each class is sensitive. Person and helmet gain from 0.1 to 0.3 and then hold. Car and gloves are flat until 0.7 and then collapse, ladder peaks at 0.5, and wood pallet barely moves at any setting, which told me its problem is not NMS.
    Show the numbers
    NMS IoUwood pallet (13.5)cardboard box (11.3)person (3.3)helmet (3.4)car (1.4)gloves (1.1)ladder (1.5)
    0.10.2740.7300.5450.4490.6010.6380.332
    0.30.2780.7340.5990.5240.5990.6370.394
    0.50.2690.7290.5980.5240.5980.6370.442
    0.70.2810.7210.5800.5160.5940.6370.376
    0.90.2520.6930.4990.4880.3510.5470.328
    Best candidate IoU for each wood pallet ground truth Horizontal bars. 22 percent of pallets have no candidate overlapping them, 17.5 percent only a weak overlap, 19.5 percent a near miss between 0.3 and 0.5, and 41 percent a candidate at or above the 0.5 match threshold. 0% 10% 20% 30% share of 3,961 pallet ground truths no overlap (below 0.1) no overlap (below 0.1): 22.0% of pallets, 87022.0% 0.1 to 0.3 0.1 to 0.3: 17.5% of pallets, 69217.5% 0.3 to 0.5 (near miss) 0.3 to 0.5 (near miss): 19.5% of pallets, 77319.5% 0.5 to 0.7 0.5 to 0.7: 22.8% of pallets, 90322.8% 0.7 and up 0.7 and up: 18.3% of pallets, 72318.3%
    matched at the 0.5 thresholdbelow the match threshold
    I took every pallet ground truth in the sample, 3,961 of them, and recorded the best IoU any pallet candidate reaches before NMS to see where the pallet failures actually sit. The median I found is 0.42. Only 41 percent of pallets have a candidate at or above the 0.5 match threshold and a fifth are near misses between 0.3 and 0.5. Another 22 percent have no candidate overlapping them at all, so I do not expect any threshold to fix them and I route them to the reviewers' annotation queue.
    Show the numbers
    best candidate IoUshare of pallet ground truthscount
    no overlap (below 0.1)0.220870
    0.1 to 0.30.175692
    0.3 to 0.5 (near miss)0.195773
    0.5 to 0.70.228903
    0.7 and up0.183723

  4. Augmentation tests

    Heavy blur cost 0.17 mAP and darkening cost 0.11, while a horizontal flip changed nothing and a vertical flip cost 0.25 (an upside-down aisle is a view the robot never sees, so vertical flips stay out of retraining). Boxes per frame fell 26 to 55 percent under blur and darkness, which makes the box count a cheap online signal that the camera or the lighting has changed.

    mAP under each image transform Bar chart grouped by transform family with the untouched baseline of 0.572 marked. Blur and darkening cost the most, contrast changes almost nothing, the horizontal flip is level with the baseline and the vertical flip falls to 0.318. 0.0 0.2 0.4 0.6 mAP at match IoU 0.5, NMS 0.5 blur baseline blur none: mAP 0.5720.572 k=5 blur k=5: mAP 0.5410.541 k=9 blur k=9: mAP 0.4680.468 k=15 blur k=15: mAP 0.4020.402 brightness -80 brightness -80: mAP 0.4590.459 -40 brightness -40: mAP 0.5540.554 baseline brightness none: mAP 0.5720.572 +40 brightness +40: mAP 0.5560.556 +80 brightness +80: mAP 0.5340.534 contrast x0.6 contrast x0.6: mAP 0.5690.569 baseline contrast none: mAP 0.5720.572 x1.4 contrast x1.4: mAP 0.5620.562 flip baseline flip none: mAP 0.5720.572 horizontal flip horizontal: mAP 0.5810.581 vertical flip vertical: mAP 0.3180.318 baseline 0.572
    I ran model 2 over the 5,000-image sample after each transform and grouped the results by family against the untouched sample at 0.572, so each bar is the same images under one change to the pixels. Blur degrades in step with the kernel size. It costs 0.03 at k=5 and 0.17 at k=15, darkening by 80 costs 0.11 where brightening by 80 costs 0.04, contrast moves under 0.01, and the horizontal flip sits on the baseline at 0.581. The vertical flip is the largest drop in the set at 0.318 and that is why I keep it out of the retraining augmentations.
    Show the numbers
    variantmAPdetections at score 0.5change
    original0.57212,5040.000
    blur k=50.5419,236-0.031
    blur k=90.4687,178-0.104
    blur k=150.4025,585-0.170
    horizontal flip0.58112,448+0.009
    vertical flip0.3186,444-0.254
    brightness -800.4598,075-0.113
    brightness -400.55411,380-0.018
    brightness +400.55611,924-0.016
    brightness +800.53410,790-0.038
    contrast x0.60.56912,276-0.003
    contrast x1.40.56212,090-0.010
    The six classes that lose the most AP under each of the three strongest transforms Three panels of six bars each. Heavy blur hits van, safety vest, barcode, helmet, truck and person by 0.20 to 0.35; the vertical flip hits forklift by 0.68 and the vehicle and cone classes by 0.4 to 0.5; darkening by 80 hits traffic light, forklift and smoke by 0.19 to 0.25. blur k=15 van blur k=15, van: AP change -0.35-0.35 safety vest blur k=15, safety vest: AP change -0.31-0.31 barcode blur k=15, barcode: AP change -0.26-0.26 helmet blur k=15, helmet: AP change -0.25-0.25 truck blur k=15, truck: AP change -0.24-0.24 person blur k=15, person: AP change -0.20-0.20 vertical flip forklift vertical flip, forklift: AP change -0.68-0.68 traffic cone vertical flip, traffic cone: AP change -0.52-0.52 gloves vertical flip, gloves: AP change -0.51-0.51 van vertical flip, van: AP change -0.48-0.48 car vertical flip, car: AP change -0.43-0.43 smoke vertical flip, smoke: AP change -0.41-0.41 brightness -80 traffic light brightness -80, traffic light: AP change -0.25-0.25 forklift brightness -80, forklift: AP change -0.19-0.19 smoke brightness -80, smoke: AP change -0.19-0.19 barcode brightness -80, barcode: AP change -0.17-0.17 cardboard box brightness -80, cardboard box: AP change -0.17-0.17 license plate brightness -80, license plate: AP change -0.16-0.16 AP change against the untouched sample
    I broke the three strongest transforms down by class and kept the six classes that lose the most AP under each one, because the per-class view is where I get the design decisions from. Blur hits the protective equipment. Helmet loses 0.25 and safety vest 0.31, so I read camera focus as mattering more for helmet detection than for pallet detection. Low light takes 0.25 off traffic light and 0.19 off forklift and smoke, and the vertical flip takes 0.68 off forklift and 0.4 to 0.5 off the vehicle and cone classes, which is capacity I would not spend.
    Show the numbers
    blur k=15changevertical flipchangebrightness -80change
    van-0.35forklift-0.68traffic light-0.25
    safety vest-0.31traffic cone-0.52forklift-0.19
    barcode-0.26gloves-0.51smoke-0.19
    helmet-0.25van-0.48barcode-0.17
    truck-0.24car-0.43cardboard box-0.17
    person-0.20smoke-0.41license plate-0.16

  5. Hard negative mining weights

    I scored all 5,000 images under seven sets of loss weights and compared the 500 hardest images each one selects, then repeated the ranking with the loss divided by the number of objects in the image.

    Share of the summed loss carried by each of the four terms Horizontal bars. The objectness term carries 90 percent of the summed loss; classification 4.8 percent, no-object 4.6 percent and localization 0.5 percent, so localization would need a factor of about 200 and classification about 20 to compete. 0% 25% 50% 75% 100% localization localization: 0.5% of the summed loss, mean 0.05 per image, factor 197 to equal objectness0.5%, x197 to match objectness objectness objectness: 90.2% of the summed loss, mean 9.762 per image, factor 1 to equal objectness90.2% no-object no-object: 4.6% of the summed loss, mean 0.492 per image, factor 20 to equal objectness4.6%, x20 to match objectness classification classification: 4.8% of the summed loss, mean 0.515 per image, factor 19 to equal objectness4.8%, x19 to match objectness
    I measured the size of each loss term on the 5,000 sample images before I weighted anything, because a lambda only changes the ranking when it closes the gap between its term and the largest one. Objectness carries 90 percent of the summed loss. Localization would need a factor of about 200 and classification about 20 to compete. The no-object term is the one the lecture singles out for weighting and I found it is the only one a modest multiplier brings within range.
    Show the numbers
    termmean per imagemedian per imagemean per ground truth boxshare of summed lossfactor to equal objectness
    localization0.0500.0160.0310.005197
    objectness9.7625.6693.4490.9021
    no-object0.4920.0690.1510.04620
    classification0.5150.3070.1920.04819
    Overlap of the hardest 500 images between lambda settings Six by six heatmap of Jaccard overlap. Every pair of settings overlaps at 0.94 to 0.98 except no-object times five, which overlaps the others at 0.77 to 0.81. balanced localization x5 classification x5 no-object x5 no-object x0.1 objectness x5 balanced balanced and balanced: Jaccard 1.001.00 balanced and localization x5: Jaccard 0.980.98 balanced and classification x5: Jaccard 0.980.98 balanced and no-object x5: Jaccard 0.810.81 balanced and no-object x0.1: Jaccard 0.950.95 balanced and objectness x5: Jaccard 0.970.97 localization x5 localization x5 and balanced: Jaccard 0.980.98 localization x5 and localization x5: Jaccard 1.001.00 localization x5 and classification x5: Jaccard 0.960.96 localization x5 and no-object x5: Jaccard 0.810.81 localization x5 and no-object x0.1: Jaccard 0.940.94 localization x5 and objectness x5: Jaccard 0.950.95 classification x5 classification x5 and balanced: Jaccard 0.980.98 classification x5 and localization x5: Jaccard 0.960.96 classification x5 and classification x5: Jaccard 1.001.00 classification x5 and no-object x5: Jaccard 0.790.79 classification x5 and no-object x0.1: Jaccard 0.960.96 classification x5 and objectness x5: Jaccard 0.970.97 no-object x5 no-object x5 and balanced: Jaccard 0.810.81 no-object x5 and localization x5: Jaccard 0.810.81 no-object x5 and classification x5: Jaccard 0.790.79 no-object x5 and no-object x5: Jaccard 1.001.00 no-object x5 and no-object x0.1: Jaccard 0.770.77 no-object x5 and objectness x5: Jaccard 0.780.78 no-object x0.1 no-object x0.1 and balanced: Jaccard 0.950.95 no-object x0.1 and localization x5: Jaccard 0.940.94 no-object x0.1 and classification x5: Jaccard 0.960.96 no-object x0.1 and no-object x5: Jaccard 0.770.77 no-object x0.1 and no-object x0.1: Jaccard 1.001.00 no-object x0.1 and objectness x5: Jaccard 0.980.98 objectness x5 objectness x5 and balanced: Jaccard 0.970.97 objectness x5 and localization x5: Jaccard 0.950.95 objectness x5 and classification x5: Jaccard 0.970.97 objectness x5 and no-object x5: Jaccard 0.780.78 objectness x5 and no-object x0.1: Jaccard 0.980.98 objectness x5 and objectness x5: Jaccard 1.001.00
    I compared the 500 hardest images each lambda setting selects with every other setting by Jaccard overlap, which is the share of images the two sets have in common. Within the x0.1 to x5 range I swept, the lambdas barely matter. Every pair sits at 0.94 to 0.98 except no-object times five, which overlaps the others at 0.77 to 0.81 because x5 is the one multiplier I tried that brings its term within a factor of four of objectness.
    Show the numbers
    balancedlocalization x5classification x5no-object x5no-object x0.1objectness x5
    balanced1.000.980.980.810.950.97
    localization x50.981.000.960.810.940.95
    classification x50.980.961.000.790.960.97
    no-object x50.810.810.791.000.770.78
    no-object x0.10.950.940.960.771.000.98
    objectness x50.970.950.970.780.981.00
    What the hardest 500 images look like under each loss weighting Two panels across six lambda settings. Ranked by raw loss the hardest 500 images average about 20 boxes each and wood pallet is a third of the boxes; ranked by loss per object they average 1.2 boxes and fire is the top class at 15 percent. The whole sample averages 4.1 boxes per image. boxes per image top class, share of boxes 0 10 20 0.00 0.25 0.50 whole sample 4.1 balanced balanced, raw loss: 19.7 boxes balanced, loss per object: 1.22 boxes balanced, raw loss: 0.33 share (wood pallet) balanced, loss per object: 0.15 share (fire) localization x5 localization x5, raw loss: 19.7 boxes localization x5, loss per object: 1.23 boxes localization x5, raw loss: 0.33 share (wood pallet) localization x5, loss per object: 0.15 share (fire) classification x5 classification x5, raw loss: 19.7 boxes classification x5, loss per object: 1.22 boxes classification x5, raw loss: 0.32 share (wood pallet) classification x5, loss per object: 0.15 share (fire) no-object x5 no-object x5, raw loss: 19.7 boxes no-object x5, loss per object: 1.23 boxes no-object x5, raw loss: 0.34 share (wood pallet) no-object x5, loss per object: 0.14 share (fire) no-object x0.1 no-object x0.1, raw loss: 19.6 boxes no-object x0.1, loss per object: 1.22 boxes no-object x0.1, raw loss: 0.32 share (wood pallet) no-object x0.1, loss per object: 0.15 share (fire) objectness x5 objectness x5, raw loss: 19.7 boxes objectness x5, loss per object: 1.23 boxes objectness x5, raw loss: 0.32 share (wood pallet) objectness x5, loss per object: 0.15 share (fire) 19.7 1.2 wood pallet 0.33 fire 0.15
    raw lossloss per object
    I ranked the sample twice under each of the six loss weightings and looked at the hardest 500 images from each ranking, once by the raw summed loss and once by that loss divided by the number of labeled objects in the image. The raw ranking selects crowds. Every setting I tried returns frames with about 20 boxes each and a third of those boxes are wood pallets, against 4.1 boxes for an average image, and the lambdas only reshuffle 1 to 10 percent of that set. When I divide by the object count I get frames with 1.2 boxes led by fire and the other low-AP classes, and that set shares 25 images with the raw one.
    Show the numbers
    settingboxes per image (raw)small boxes (raw)top class (raw)boxes per image (per object)small boxes (per object)top class (per object)shared with raw top 500
    balanced19.70.15wood pallet (0.33)1.220.00fire (0.15)0.05
    localization x519.70.15wood pallet (0.33)1.230.00fire (0.15)0.06
    classification x519.70.14wood pallet (0.32)1.220.00fire (0.15)0.05
    no-object x519.70.16wood pallet (0.34)1.230.01fire (0.14)0.05
    no-object x0.119.60.15wood pallet (0.32)1.220.00fire (0.15)0.05
    objectness x519.70.15wood pallet (0.32)1.230.00fire (0.15)0.05
    whole sample4.10.06person (0.20)
04

What I found

0.583
mAP for the selected model, against 0.544 for the other
0.25
Deployed objectness threshold, set on person recall
41%
Wood pallets with a candidate box at the match threshold
25/500
Hard images shared by the raw and per-object rankings

The worst class is not a threshold problem

Wood pallets have the most labeled boxes in the dataset and one of the lowest APs (about 0.3), and no NMS setting moves them. When I measured the best candidate box for each pallet, only 41 percent had one at the 0.5 match threshold and 22 percent had nothing overlapping at all. That is a labeling and retraining problem, and the design routes it to the human review queue.

The NMS curve is lopsided

Setting the NMS threshold too low cost 0.03 mAP from merged neighbors. Setting it too high cost 0.10 from duplicate boxes on every object. Sparse classes like car and gloves ignore the setting until 0.8, while crowded ones like person and helmet need at least 0.3.

Summed loss mines crowds

Ranked by total loss the hardest 500 images average 20 objects each against 4 for the dataset (mostly pallet stacks), and the loss weights I tried keep 90 to 99 percent of that set. Dividing the loss by the object count selects an almost entirely different set (25 images in common) led by fire and the other low-AP classes. I would make that switch the miner's first configuration option.

The design, one row per decision

Every decision in the report, the measurement it rests on, the requirement it serves, and the alternative I rejected.

model
model 2
evidencemAP 0.583 against 0.544; the ranking holds in all nine cells of the sensitivity grid
servesdetect the twenty classes
rejectedmodel 1, behind on every class
objectness threshold
0.25
evidenceperson recall 0.620 against 0.638 at 0.05; 3.5 boxes per image, 0.03 unlabeled
servessafety first, planner load
rejected0.05, half the boxes unlabeled; 0.5, a quarter of the people lost
frame sampling
1 in 2
evidence13 ms per frame against the 33 ms budget
serves30 fps on CPU
rejectedevery frame, no headroom for a slower CPU
evaluation and retraining sample
stratified by rarest class
evidencerare-class deviation 0.016 against 0.032 for uniform; 183 ladder images against 86
servesfind the classes the model gets wrong
rejecteduniform draw, starves the rare classes
NMS IoU
0.5
evidencebest or within 0.005 at every score threshold, held-out included
servesone box per object
rejected0.6, same mAP and more boxes; per-class thresholds, pallets do not respond
retraining augmentation
blur, brightness, horizontal flip
evidenceblur and darkness cost 0.03 to 0.17 mAP; horizontal flip +0.009; vertical flip -0.254
servesreviewers correct and retrain
rejectedvertical flip, a view the robot never sees
miner ranking
loss per ground truth box
evidenceraw loss selects crowds at 20 boxes per image; per-object selects the low-AP classes
servesretrain on what the model has not learned
rejectedraw loss, 85 pallet-stack composites in the top 500
online metric
boxes per frame
evidencethe count fell 26 to 55 percent under blur and darkness
servespost-deployment monitoring
rejectedwaiting for labels, which arrive days later
05

Limits and notes

The dataset is built from mosaic tiles rather than real warehouse frames, so the crowding and augmentation results describe that construction as much as a live camera. The timing (about 13 ms per frame against a 33 ms budget) is one laptop CPU, not the robot's.

Course repositories stay private under the university's rules, so this page summarizes my results and does not link the code.