Series · Part 2 of 5
Embeddings and Vector Search, Step by Step: Finding Duplicate Suppliers
Learn embeddings, distance, vector databases, semantic search and a mini RAG one step at a time, by finding duplicate suppliers in a small dataset.
Part 2 of the series, after the business case and before Part 3, the full pipeline (coming soon).
We’ll find duplicate suppliers in a small list, one idea at a time. Here’s the path:
The example data
45 supplier records, written straight into the notebook:
SUPPLIERS_CSV = """\
id,supplier,country,city,category
100234,ABC Logistics AB,Sweden,Stockholm,Transportation
100239,ABC Logistics,Sweden,Stockholm,Transportation
100240,A.B.C. Logistics AB,SE,Stockholm,Transportation
100241,ABC LOGISTICS AB,Sweden,Stokholm,Transportation
100259,Müller & Söhne Maschinenbau GmbH,Germany,Stuttgart,Manufacturing
100260,Mueller und Soehne Maschinenbau GmbH,Germany,Stuttgart,Manufacturing
100264,TEST,,,
100266,N/A,N/A,N/A,N/A
100270, ,Sweden,Stockholm,Transportation
100278,ABC Logistics AB,,,
...
"""
10 real companies typed in different ways, 13 junk entries and one personal name. The goal: find the duplicates without being fooled by the junk.
Step 1 · Embedding: text becomes numbers
In simple words: an embedding is a list of numbers that captures what a text means. Similar meaning, similar numbers.
A free model, all-MiniLM-L6-v2, turns each record into 384 numbers:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
text = """
Supplier: ABC Logistics AB
Country: Sweden
City: Stockholm
Category: Transportation
"""
vector = model.encode(text, normalize_embeddings=True)
dimensions: 384
first 8 values: [ 0.051 -0.032 -0.036 -0.014 0.013 0.019 0.043 0.002]
Step 2 · Distance: how alike are two records?
In simple words: distance measures how far apart two embeddings are. 0 means identical.
ABC Logistics AB compared with | Distance | |
|---|---|---|
A.B.C. Logistics AB | 0.06 | same company |
Nordic Freight Solutions | 0.20 | different company, same industry |
Berlin Office Supplies GmbH | 0.41 | different company, different industry |
Anything under 0.15 counts as a possible duplicate. Note how close the different transport company is: similar is not the same.
Try a candidate supplier. Pick one, or edit the text in the cell below and press ▶ (or Shift+Enter).
Blank fields are embedded as nan, like in the notebook. Scores can differ from the notebook in the third decimal.
- Typo finds the Sunrise records.
- New company, same industry is flagged at 0.14, though it’s a different company.
- Name only misses the complete
ABC Logistics ABrecord.
Step 3 · Vector database: store the numbers
In simple words: a vector database stores embeddings and quickly finds the ones closest to a new one.
We use Chroma, which runs inside the notebook:
import chromadb
db = chromadb.EphemeralClient()
collection = db.get_or_create_collection("suppliers", metadata={"hnsw:space": "cosine"})
collection.upsert(
ids=[s["id"] for s in SUPPLIERS],
documents=[s["text"] for s in SUPPLIERS],
embeddings=model.encode([s["text"] for s in SUPPLIERS], normalize_embeddings=True).tolist(),
)
Step 4 · Semantic search: search by meaning
In simple words: find records that mean the same as your question, even with no words in common.
result = collection.query(query_embeddings=[embed("transportation supplier in Sweden")], n_results=3)
[100270] distance=0.183 Supplier: " " Sweden, Stockholm, Transportation
[100239] distance=0.238 Supplier: ABC Logistics
[100240] distance=0.245 Supplier: A.B.C. Logistics AB
The Swedish transport companies come back. But the top hit has no name at all: its country, city and category fit perfectly, and there’s no name to disagree.
Step 5 · Duplicate check: nearest match plus a cut-off
In simple words: find a record’s nearest neighbour. Under 0.15 means a possible duplicate.
A new supplier, before it’s created:
candidate = """
Supplier: ABC Logistics A.B.
Country: Sweden
City: Stockholm
Category: Transport
"""
match = collection.query(query_embeddings=[embed(candidate)], n_results=1)
Possible duplicate of [100234] ABC Logistics AB (distance=0.023)
The whole list: 29 pairs come back under 0.15. Twenty are real duplicates, and nine involve junk:
| Pair | Distance |
|---|---|
ABC Logistics ↔ " " (blank name) | 0.07 |
Berlin Office Supplies ↔ DO NOT USE - DUPLICATE | 0.09 |
Nordic Freight Solutions ↔ 12345 | 0.13 |
The embedding reads the whole record, so junk with the right country, city and category looks like a real supplier.
Step 6 · Rules first: clean before you match
In simple words: simple if-then checks catch obvious junk cheaply. Run them before the AI part.
def is_junk(name) -> bool:
n = str(name).strip().lower()
return (
n in {"", "nan", "null", "n/a", "tbd", "test", "unknown vendor"}
or "test" in n or "do not use" in n or n.startswith("zzz")
or not re.search(r"[a-z]{3}", n) # no real word: '12345', '!!!@@@###'
or re.fullmatch(r"(.{2,4})\1+", n) # keyboard mash: 'asdfasdf'
)
The rules catch all 13 junk records. Checking again gives 21 pairs: 20 real, 1 wrong. Both the wrong pair and the one missed duplicate come from ABC Logistics AB with no country or city: it lands far from its twin in Stockholm.
Lesson: an embedding is only as good as the fields that are filled in. That’s why Part 3 also matches on tax ID and bank account.
Step 7 · AI as a judge: a second opinion
In simple words: show each flagged pair to a language model and ask, “Same company? Why?”
A small local model (llama3.2 in Ollama) got 13 of 20 real duplicates right and 7 wrong, with confident nonsense:
[100238] <-> [100243] same_supplier=False confidence=0.8
Different legal suffixes ('AS' vs 'AS')
The model gives a suggestion and a reason. A person makes the call.
Step 8 · Mini RAG: ask in plain words
In simple words: RAG means retrieve the matching records, augment a prompt with them, and let a model generate the answer from them only.
Embeddings hold no facts, but they find the right records. That’s the retrieve step, built in Steps 1–5.
Ask a question about the suppliers. Pick one to run all three cells, or edit the question and press ▶.
Steps 1 and 2 run live in your browser. No language model runs on this page: step 3 shows the answers saved from the notebook run.
- Finland: correct, citing both Helsinki Steel records.
- Swedish duplicates: the right four records, but it made up a country code (
SW) and then doubted itself. - Bank account: “Not in these records.” Correct, and nothing invented.
Lesson: RAG keeps the model close to your data, but details can still be wrong. Show the record ids so people can check. More in RAG Explained.
This is a demo. It won’t scale
Everything here runs on 45 records in one notebook. A real supplier master has hundreds of thousands to millions of records, spread across SAP systems, and it changes every day. This demo won’t hold up:
| In this demo | Needed for millions of records |
|---|---|
| One search per record, against every record | Compare only within small groups (same country, same tax-ID prefix), in batches |
| Chroma in memory | A database built for millions of vectors, such as Postgres with pgvector |
| You press ▶ | Scheduled runs that pick up only new and changed records |
| Results print on screen | A review queue for data stewards, and an audit log |
| Nothing goes back to SAP | Approved changes written back through SAP MDG |
That needs a proper data pipeline: each step becomes a task, and Airflow runs them in order, retries failures, and waits for people to approve. That’s Part 3 (coming soon).
Takeaways
- Embeddings find similar, not same. Great for a shortlist.
- Missing or junk fields fool them. Rules go first.
- AI suggests, people decide. The small model was wrong a third of the time, and sounded sure.
- RAG = retrieve, augment, generate. Check the ids it cites.
Next, Part 3 (coming soon) turns these steps into an Airflow pipeline, with every change approved by a person and written back through SAP MDG.
Enterprise Master Data Cleaning · 2 of 5 published
- 1 Enterprise Master Data Cleaning: The Business Case for SAP Supplier Data
- 2 Embeddings and Vector Search, Step by Step: Finding Duplicate Suppliers
- 3 Cleaning SAP Supplier Master Data with Airflow and Human-in-the-Loop soon
- 4 Hands-On: Building the SAP Supplier Cleanup Pipeline in Airflow soon
- 5 Cleaning SAP Supplier Master Data with SAP BTP, MDG and Business AI soon
Related posts
Part 1 — Building an Agent Platform
Every Team Built an AI Agent. Nobody Knew Who Used Which Model, or What It Cost.
One gateway in front of every agent: role-based access, human approvals, and every LLM call logged per user, agent and model.
ReadPart 2 — Building an Agent Platform
An Enterprise Agent Platform on My Laptop, in 8 Containers
The agent platform on a laptop, screen by screen: start the stack, deploy a Langflow and a LangChain agent, wire up models, chat, approve.
ReadGet new posts in your inbox
No spam. No digest. Just a note when I publish something new.
Discussion
On this page