1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
|
---
name: photo-processing
description: "Process photo directories to detect duplicates, burst sequences, sharpness-ranked keepers, and cleanup candidates. Handles perceptual hashing, EXIF-based burst clustering, edge-sharpness scoring, Syncthing conflicts, and safe staged deletion. Use when asked to deduplicate photos, find blurry shots, clean up burst sequences, detect near-duplicates, or organize camera imports. Triggers on: photo cleanup, deduplicate images, find duplicates, burst shots, sharpness ranking, photo processing, camera import cleanup, remove blurry photos."
---
# Photo Processing
Clean up photo directories by detecting duplicates, burst sequences, and blurry/outlier frames. Provide actionable keep/delete recommendations with safe staged deletion.
## Reference Files
- [Auto-enhance](references/auto-enhance.md) — standalone ImageMagick 7 batch script (`auto-enhance-photos.sh`) for auto-orientation, auto-levels, auto-gamma, mild brightness/saturation. Use this to *improve* surviving keepers, separate from the culling workflow below.
## When to Use
- User asks to **deduplicate**, **find duplicates**, or **detect near-duplicates** in a photo folder.
- User asks to **clean up burst shots**, **shutter-speed sequences**, or **keep only sharpest** frames.
- User asks to **remove blurry photos**, **find out-of-focus shots**, or **rank by sharpness**.
- User mentions **Syncthing conflicts** in a photo directory.
- User imports a **camera card** or **Fujifilm/Canon/Sony/Nikon** folder and wants cleanup.
- Triggers: *photo cleanup*, *deduplicate images*, *find duplicates*, *burst shots*, *sharpness ranking*, *photo processing*, *camera import cleanup*, *remove blurry photos*.
## Required Tools
- `exiftool` — EXIF metadata extraction (timestamps, burst grouping)
- `python3` + `PIL/Pillow` — perceptual hashing, sharpness scoring
- `ImageFilter.FIND_EDGES` from Pillow — edge-based sharpness metric
- `fdupes` — exact byte-level duplicate detection (optional fallback)
- Standard shell tools (`ls`, `stat`, `md5sum`, `mv`, `mkdir`)
## Core Techniques
### 1. Exact Duplicate Detection
Run `fdupes -r -s -S <dir>` first. If it finds groups, those are byte-identical duplicates — safe to delete all but one without visual review.
### 2. Near-Duplicate Detection (Perceptual Hashing)
Use **average hash (aHash)** on a 16×16 grayscale resize:
```python
from PIL import Image
def ahash(path, size=16):
with Image.open(path) as im:
im = im.convert('L').resize((size, size), Image.Resampling.LANCZOS)
pixels = list(im.getdata())
avg = sum(pixels) / len(pixels)
bits = ''.join('1' if p > avg else '0' for p in pixels)
return int(bits, 2)
def hamming_distance(a, b):
x = a ^ b
dist = 0
while x:
dist += 1
x &= x - 1
return dist
```
- **hamming = 0**: identical or nearly identical
- **hamming ≤ 5**: very similar, likely burst variants
- **hamming ≤ 10**: similar composition, review recommended
- **hamming > 20**: different shots
Compare only nearby filename ranges (e.g., ±15 frames) to avoid O(n²) on large sets.
### 3. Burst Sequence Detection (EXIF Timestamp Clustering)
Use `exiftool -DateTimeOriginal -SubSecTimeOriginal` to extract capture time. Group frames where consecutive timestamps differ by ≤ 3 seconds. A burst is defined as ≥ 3 consecutive frames within that gap.
```python
from datetime import datetime
from collections import defaultdict
def parse_exif_datetime(raw):
# 2026:06:07 13:19:22
return datetime.strptime(raw, '%Y:%m:%d %H:%M:%S')
```
Also report **same-second shots** (≥ 2 frames with identical `HH:MM:SS`) — these are always burst/machine-gun frames.
### 4. Sharpness Ranking (Keep the Best, Delete the Rest)
Compute **variance of edge intensities** as a proxy for sharpness. A sharper image has stronger high-frequency detail, which shows up after `ImageFilter.FIND_EDGES`.
```python
from PIL import Image, ImageFilter
def sharpness_score(path):
with Image.open(path) as im:
im = im.convert('L').resize((1024, 1024), Image.Resampling.LANCZOS)
edges = im.filter(ImageFilter.FIND_EDGES)
pixels = list(edges.getdata())
mean = sum(pixels) / len(pixels)
var = sum((p - mean) ** 2 for p in pixels) / len(pixels)
return var
```
Within each burst:
1. Compute sharpness for every frame
2. Sort descending by sharpness
3. Recommend **keeping top 2–5** depending on burst size
4. Label remainder as **delete candidates**
**File size as secondary signal:** larger JPEGs in a burst often correlate with more detail (less motion blur, higher complexity), but sharpness score is the primary metric.
### 5. Syncthing Conflict Handling
Files matching `*.sync-conflict-*.JPG` are Syncthing duplicates. For each conflict:
- Find the non-conflict original by stripping `.sync-conflict-<id>` from the basename
- Compare perceptual hashes:
- If hamming ≈ 0: conflict is identical → delete the conflict
- If hamming is small but sizes differ: keep the larger/better one, delete the other
- If no original exists: the conflict is an orphan → move to `to_delete/` for manual review
### 6. Safe Staged Deletion Workflow
**Never delete immediately.** Always:
1. Compute all metrics and present a ranked summary to the user
2. Confirm the delete candidate list
3. `mkdir -p to_delete/`
4. `mv <candidates> to_delete/`
5. Report what was moved and total space freed
6. User can later `rm -rf to_delete/` after review, or `mv to_delete/* .` to restore
## Typical Workflow
1. **Scan directory** — count files, check for exact duplicates (`fdupes`)
2. **Hash all images** — compute aHash for perceptual similarity
3. **Extract EXIF timestamps** — group into burst sequences by time gaps ≤ 3s
4. **Compute sharpness** — rank each burst by edge variance
5. **Identify conflicts** — flag Syncthing orphans
6. **Present summary** — bursts found, keepers, delete candidates, space estimate
7. **Move to `to_delete/`** after user confirmation
## Output Format
For each burst, present:
- Burst size and time range
- Top N keepers with sharpness score + file size
- Bottom N delete candidates with sharpness score + file size
- Clear recommendation ("Keep top 3, delete the other 12")
## Caveats
- **Low-resolution thumbnails** or heavily compressed frames may have low file size but still be sharp — trust the sharpness score over size when they disagree.
- **Monochrome/B&W images** have different perceptual hash behavior than color; the edge-sharpness metric still works.
- **Portraits with shallow DOF** may have a blurry background but sharp subject. The global edge-variance metric can mis-rank these if the background dominates. For critical portrait bursts, the user should visually review the top-ranked frames rather than auto-deleting.
- **Different compositions** within a short time window (e.g. wide shot → close-up) may cluster as a single burst by timestamp. The hamming distance check catches this — if consecutive frames differ by > 20, they are different compositions and should not be auto-deleted.
## Auto-Enhancement
After culling, optionally improve the surviving keepers with the standalone ImageMagick batch script. See [references/auto-enhance.md](references/auto-enhance.md) for the full `auto-enhance-photos.sh` script, a per-flag explanation, output naming, and saturation tuning.
|
