GoofyTex: find the products in an outfit

The challenge: for every “bundle” photo, a model wearing a full outfit, return up to 15 catalogue products ranked by how likely they are to be in it.

Why look at garments?

A whole-image embedding mostly learns the scene: the street, the pose, the light. So the pipeline looks at garments, not photos. YOLO finds each piece of clothing, each crop is embedded with a fashion-tuned SigLIP (OpenCLIP, Marqo), and products are retrieved per crop by cosine similarity, then merged into one ranking.

The examples above show clothing detections, garment crops and ranked catalogue products. Select a box to inspect its crop. The product ranking belongs to the complete outfit and stays fixed as you explore the detected regions.

The difficult query is worth a look: the saved ranking leads with sunscreen products for a photo of a dressed person. Similarity can latch onto the wrong visual signal. Seeing the products next to the input makes that mistake easy to spot.

What I worked on

• Contrastive fine-tuning of the bundle and product encoders, with hard negatives mined from the model’s own mistakes every few epochs.
• The inference pipelines: per-crop retrieval with test-time augmentation, a top-1-per-garment variant so a three-piece outfit doesn’t get padded to 15 guesses, and a “phase 1” variant that restricts the index to the right catalogue section and boosts products whose category matches a zero-shot label on the crop.
• The model ensemble: combining the best runs by reciprocal rank fusion.

What I’d point out

• The catalogue already knows things the model would otherwise have to learn: section, category, gender. Using them as filters and boosts is cheap signal.
• Ranking 15 slots when an outfit has three pieces is a precision problem disguised as a recall one. The top-1-per-garment variant is the honest answer to it.