Embeddings and Vector Search, Step by Step: Finding Duplicate Suppliers

Part 2 of the series, after the business case and before Part 3, the full pipeline (coming soon).

We’ll find duplicate suppliers in a small list, one idea at a time. Here’s the path:

The example data

45 supplier records, written straight into the notebook:

SUPPLIERS_CSV = """\
id,supplier,country,city,category
100234,ABC Logistics AB,Sweden,Stockholm,Transportation
100239,ABC Logistics,Sweden,Stockholm,Transportation
100240,A.B.C. Logistics AB,SE,Stockholm,Transportation
100241,ABC LOGISTICS AB,Sweden,Stokholm,Transportation
100259,Müller & Söhne Maschinenbau GmbH,Germany,Stuttgart,Manufacturing
100260,Mueller und Soehne Maschinenbau GmbH,Germany,Stuttgart,Manufacturing
100264,TEST,,,
100266,N/A,N/A,N/A,N/A
100270,   ,Sweden,Stockholm,Transportation
100278,ABC Logistics AB,,,
...
"""

10 real companies typed in different ways, 13 junk entries and one personal name. The goal: find the duplicates without being fooled by the junk.

Step 1 · Embedding: text becomes numbers

In simple words: an embedding is a list of numbers that captures what a text means. Similar meaning, similar numbers.

A free model, all-MiniLM-L6-v2, turns each record into 384 numbers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

text = """
Supplier: ABC Logistics AB
Country: Sweden
City: Stockholm
Category: Transportation
"""
vector = model.encode(text, normalize_embeddings=True)

dimensions: 384
first 8 values: [ 0.051 -0.032 -0.036 -0.014  0.013  0.019  0.043  0.002]

Step 2 · Distance: how alike are two records?

In simple words: distance measures how far apart two embeddings are. 0 means identical.

ABC Logistics AB compared with	Distance
A.B.C. Logistics AB	0.06	same company
Nordic Freight Solutions	0.20	different company, same industry
Berlin Office Supplies GmbH	0.41	different company, different industry

Anything under 0.15 counts as a possible duplicate. Note how close the different transport company is: similar is not the same.

supplier-check.ipynbnot started · runs in your browser

▶[ ]

from sentence_transformers import SentenceTransformer model = SentenceTransformer("all-MiniLM-L6-v2") suppliers = load_vectors("suppliers") # 45 records, already embedded

Try a candidate supplier. Pick one, or edit the text in the cell below and press ▶ (or Shift+Enter).

TypoSwedish spellingNew company, same industryJunkName only

▶[ ]

candidate = """ Supplier: Country: City: Category: """ SKIP_JUNK = True # set junk records aside first check_duplicate(candidate, threshold=0.15, top=5)

Blank fields are embedded as nan, like in the notebook. Scores can differ from the notebook in the third decimal.

• Typo finds the Sunrise records.
• New company, same industry is flagged at 0.14, though it’s a different company.
• Name only misses the complete ABC Logistics AB record.

Step 3 · Vector database: store the numbers

In simple words: a vector database stores embeddings and quickly finds the ones closest to a new one.

We use Chroma, which runs inside the notebook:

import chromadb

db = chromadb.EphemeralClient()
collection = db.get_or_create_collection("suppliers", metadata={"hnsw:space": "cosine"})

collection.upsert(
    ids=[s["id"] for s in SUPPLIERS],
    documents=[s["text"] for s in SUPPLIERS],
    embeddings=model.encode([s["text"] for s in SUPPLIERS], normalize_embeddings=True).tolist(),
)

Step 4 · Semantic search: search by meaning

In simple words: find records that mean the same as your question, even with no words in common.

result = collection.query(query_embeddings=[embed("transportation supplier in Sweden")], n_results=3)

[100270] distance=0.183   Supplier: "   "   Sweden, Stockholm, Transportation
[100239] distance=0.238   Supplier: ABC Logistics
[100240] distance=0.245   Supplier: A.B.C. Logistics AB

The Swedish transport companies come back. But the top hit has no name at all: its country, city and category fit perfectly, and there’s no name to disagree.

Step 5 · Duplicate check: nearest match plus a cut-off

In simple words: find a record’s nearest neighbour. Under 0.15 means a possible duplicate.

A new supplier, before it’s created:

candidate = """
Supplier: ABC Logistics A.B.
Country: Sweden
City: Stockholm
Category: Transport
"""
match = collection.query(query_embeddings=[embed(candidate)], n_results=1)

Possible duplicate of [100234] ABC Logistics AB (distance=0.023)

The whole list: 29 pairs come back under 0.15. Twenty are real duplicates, and nine involve junk:

Pair	Distance
ABC Logistics ↔ " " (blank name)	0.07
Berlin Office Supplies ↔ DO NOT USE - DUPLICATE	0.09
Nordic Freight Solutions ↔ 12345	0.13

The embedding reads the whole record, so junk with the right country, city and category looks like a real supplier.

Step 6 · Rules first: clean before you match

In simple words: simple if-then checks catch obvious junk cheaply. Run them before the AI part.

def is_junk(name) -> bool:
    n = str(name).strip().lower()
    return (
        n in {"", "nan", "null", "n/a", "tbd", "test", "unknown vendor"}
        or "test" in n or "do not use" in n or n.startswith("zzz")
        or not re.search(r"[a-z]{3}", n)      # no real word: '12345', '!!!@@@###'
        or re.fullmatch(r"(.{2,4})\1+", n)    # keyboard mash: 'asdfasdf'
    )

The rules catch all 13 junk records. Checking again gives 21 pairs: 20 real, 1 wrong. Both the wrong pair and the one missed duplicate come from ABC Logistics AB with no country or city: it lands far from its twin in Stockholm.

Lesson: an embedding is only as good as the fields that are filled in. That’s why Part 3 also matches on tax ID and bank account.

Step 7 · AI as a judge: a second opinion

In simple words: show each flagged pair to a language model and ask, “Same company? Why?”

A small local model (llama3.2 in Ollama) got 13 of 20 real duplicates right and 7 wrong, with confident nonsense:

[100238] <-> [100243]  same_supplier=False  confidence=0.8
  Different legal suffixes ('AS' vs 'AS')

The model gives a suggestion and a reason. A person makes the call.

Step 8 · Mini RAG: ask in plain words

In simple words: RAG means retrieve the matching records, augment a prompt with them, and let a model generate the answer from them only.

Embeddings hold no facts, but they find the right records. That’s the retrieve step, built in Steps 1–5.

mini-rag.ipynbnot started · runs in your browser

Ask a question about the suppliers. Pick one to run all three cells, or edit the question and press ▶.

Which Swedish transport suppliers might be duplicates of each other?Do we have any suppliers in Finland?What is the bank account number for ABC Logistics AB?

▶[ ]

# 1. Retrieve: the records closest to the question question = "" records = retrieve(question, top=6)

▶[ ]

# 2. Augment: put the records and the question into one prompt prompt = build_prompt(question, records) print(prompt)

▶[ ]

# 3. Generate: a language model answers from those records only answer = ask_llm(prompt) # llama3.2 via Ollama in the notebook print(answer)

Steps 1 and 2 run live in your browser. No language model runs on this page: step 3 shows the answers saved from the notebook run.

• Finland: correct, citing both Helsinki Steel records.
• Swedish duplicates: the right four records, but it made up a country code (SW) and then doubted itself.
• Bank account: “Not in these records.” Correct, and nothing invented.

Lesson: RAG keeps the model close to your data, but details can still be wrong. Show the record ids so people can check. More in RAG Explained.

This is a demo. It won’t scale

Everything here runs on 45 records in one notebook. A real supplier master has hundreds of thousands to millions of records, spread across SAP systems, and it changes every day. This demo won’t hold up:

In this demo	Needed for millions of records
One search per record, against every record	Compare only within small groups (same country, same tax-ID prefix), in batches
Chroma in memory	A database built for millions of vectors, such as Postgres with pgvector
You press ▶	Scheduled runs that pick up only new and changed records
Results print on screen	A review queue for data stewards, and an audit log
Nothing goes back to SAP	Approved changes written back through SAP MDG

That needs a proper data pipeline: each step becomes a task, and Airflow runs them in order, retries failures, and waits for people to approve. That’s Part 3 (coming soon).

Takeaways

1. Embeddings find similar, not same. Great for a shortlist.
2. Missing or junk fields fool them. Rules go first.
3. AI suggests, people decide. The small model was wrong a third of the time, and sounded sure.
4. RAG = retrieve, augment, generate. Check the ids it cites.

Next, Part 3 (coming soon) turns these steps into an Airflow pipeline, with every change approved by a person and written back through SAP MDG.