How many affiliate feed listings are actually duplicates?

Part of: For creators

Three dedupe rules, three answers. Deduplicating on the listing URL removed 3832 of 12944 rows (29.6%) — the same listing returned by several keyword queries, and true duplication.

Deduplicating on advertiser product id removed only 279 more. Deduplicating on advertiser, title and price would remove 1569 further rows, and every one of them has a different URL, so that rule deletes real products.

Black platform flip-flops with a rainbow-striped wedge sole, one secondhand listing among many identically titled ones — Forty-seven different pairs of shoes on one marketplace share the title Platform Sandals. Deduplicate on the title and forty-six real listings disappear.

Three rules, measured on the same pull

Dedupe rule	Rows it removes	Share of the pull	Is it correct?
Same listing URL	3832	29.6% of 12944 fetched	Yes. One keyword query returning a listing does not make it a different listing.
Same advertiser product id	279	3.1% of 8951 unique URLs	Yes. This is the genuine same-product-twice rate, and it is small.
Same advertiser, title and price	1569	17.5% of 8951	No. 0 of those rows share a URL with the row they duplicate; the rest are distinct products.

That third row is the finding. It looks like a sensible rule, it produces a plausible number, and it is wrong — because on peer-to-peer marketplaces a two-word title and a round price are not identifying information. 1382 of the 1569 rows it removes come from a single marketplace.

Where genuine duplication comes from

Cause	What it looks like	Scale here
One listing matching several keyword queries	The same URL returned by 'vinyl record' and 'vinyl lp album'	3832 rows
Multiple catalogue feeds per advertiser	The same product published in a B2B feed, a US feed and an additional-images feed	14 advertisers publish more than one catalogue, one of them 29
Colourway and variant rows	One product id, several URLs with different variant parameters	279 rows share a product id

Wooden vinyl record storage crate holding LPs upright — The other kind: one homeware product published across several catalogue feeds at slightly different prices. The product id is identical, so the id catches it and the title never would.

What to do with this

Deduplicate on the URL first. It is cheap, it is correct, and it removes the largest block.
Then collapse on advertiser plus product id if you want one row per product rather than one row per variant. Decide which you actually want — a size or colour variant is a different purchase.
Never deduplicate on title, and never on title plus price. On marketplaces these are not identifiers.
Count generic titles before you build anything automated on them. 13.8% of rows here cannot be told apart by title alone.

What this costs us, stated plainly

Our own pipeline collapses advertiser, cleaned title and price, which is the rule this page has just argued against. That is deliberate: it keeps published statistics conservative, and we would rather undercount a marketplace than double-count it. The consequence is that every marketplace listing count on this site is a floor, not a census — in this pull it understates one marketplace by roughly 1382 listings. Saying so is cheaper than being quietly wrong.

LinkToLooks has earned $0 from affiliate links to date and is approved by no network. Nothing here is a pitch for a feed tool; it is the arithmetic behind a decision we had to make ourselves.

Honest empty: we cannot measure how many of these listings are duplicates in the sense a shopper means — the same physical item listed twice by two sellers. Nothing in a product feed identifies a used object across sellers, and we will not guess at it from titles.

Last verified 1 September 2026: 12944 rows fetched across 44 topic queries, 9112 unique listing URLs, 8951 after advertiser exclusions, from the CJ product feed.

Some links here are affiliate links. LinkToLooks has earned $0 from them to date — no network has approved us yet — so nothing on this page is picked to hit a payout.

Feed duplication, asked plainly

How much duplication is there in an affiliate product feed?

In this pull, 12944 rows came back from the API and 3832 of them (29.6%) were the same listing returned by more than one keyword query. Among the 9112 unique URLs left, only 279 rows shared an advertiser product id with another row.

What is the right key to deduplicate on?

The destination URL, or the advertiser plus their product id. Both identify a listing. Title and price do not, and using them removes real inventory.

Why does deduplicating on title and price go wrong?

Because generic titles collide. 1239 rows in this pull (13.8%) have titles of two words or fewer, and on one marketplace 47 different pairs of shoes are all titled Platform Sandals. Several of them are also priced at the same round number.

Do advertisers publish the same product more than once?

Some do. 14 advertisers here publish under more than one catalogue name, one of them under 29. That is the mechanism behind most genuine same-product duplication, and the product id catches it.

Sources

CJ Affiliate product feeds, US advertisers, queried 1 September 2026 — every count on this page is computed from that pull, at the row level, during this run.