We're interested in retrieval of relatively short passages (one to a few sentences) using conversational queries that may be found in a chat. To this end we've developed an evaluation based on datasets QReCC, StrategyQA, and MS MARCO. (Fine-tuning is done with different datasets.) Results in the following table show top-3 and top-10 retrieval accuracy for each dataset.
| Model | qrecc @3 | qrecc @10 | strategyqa @3 | strategyqa @10 | msmarco @3 | msmarco @10 |
|---|---|---|---|---|---|---|
| openai:text-embedding-ada-002 | 67.09 | 76.80 | 68.00 | 82.40 | 73.10 | 80.14 |
| st:sentence-transformers/multi-qa-MiniLM-L6-cos-v1 | 69.98 | 77.57 | 73.90 | 87.75 | 70.31 | 77.71 |
| st:sentence-transformers/all-distilroberta-v1 | 65.01 | 76.15 | 66.35 | 82.50 | 68.59 | 78.34 |
| st:sentence-transformers/sentence-t5-large | 68.40 | 78.28 | 72.55 | 86.60 | 71.30 | 80.51 |
| st:sentence-transformers/all-mpnet-base-v2 | 70.69 | 80.19 | 74.50 | 87.65 | 75.00 | 81.77 |
| st:sentence-transformers/multi-qa-mpnet-base-cos-v1 | 74.95 | 82.42 | 79.75 | 91.25 | 75.00 | 82.85 |
| flag:BAAI/bge-base-en-v1.5 | 74.18 | 82.70 | 78.55 | 89.40 | ||
| flag:BAAI/bge-large-en-v1.5 | 77.51 | 84.33 | 78.15 | 90.00 | ||
| em-MiniLM-p1-01 (ours) | 72.43 | 79.26 | 75.50 | 89.00 | 71.75 | 79.60 |
| em-MiniLM-p3-01 (ours) | 72.87 | 80.02 | 78.00 | 89.75 | 73.38 | 79.96 |
| em-distilroberta-p1-01 (ours) | 77.67 | 83.84 | 83.25 | 94.15 | 79.78 | 84.39 |
| em-distilroberta-p3-01 (ours) | 78.33 | 84.66 | 86.55 | 95.40 | 79.51 | 85.29 |
| em-distilroberta-p5-01 (ours) | 78.88 | 84.44 | 87.40 | 95.70 | 79.24 | 84.84 |
Note: The msmarco dataset has apparently changed since these tests (except for FlagEmbedding tests) were conducted.
Model em-distilroberta-p1-01 is the default embedding model used
by this library. While em-distilroberta-p3-01 and em-distilroberta-p5-01 have better
retrieval accuracy, note that they require storing 3 and 5 embeddings
per chunk, respectively.
The following are 10 queries sampled from the dataset constructed out of StrategyQA, along with the expected passage, and the 3 top matching passages retrieved using 2 different embedding models. There is a high margin of error when selecting a sample of size 10, but we picked one that is representative of the overall results.
| Query | Expected Passage | em-distilroberta-p5-01 | openai:text-embedding-ada-002 |
|---|---|---|---|
| Stephen: British mathematician and computer scientist Rachel: Did Alan Turing suffer the same fate as Abraham Lincoln? Stephen: | Alan Turing committed suicide via cyanide poisoning. | 1) Scholars have determined Noah's Ark to be 75 feet wide. Alan Turing committed suicide via cyanide poisoning. Jeff Goldnlum starred in the 1986 movie The Fly. 2) Keelhauling was a severe punishment whereby the condemned man was dragged beneath the ship’s keel on a rope. Nikola Tesla was born in the Austrian Empire Lips and skin turn blue in response to a lack of oxygen. 3) The main character of "Alice's Adventures in Wonderland" is Alice, a young girl. Arnold Schwarzenegger deadlifted 710 pounds in a competition. | 1) James Watson is a geneticist, who believes in his own work. The Greek alphabet is still commonly used Rick and Morty is available in blu-ray format. 2) Scholars have determined Noah's Ark to be 75 feet wide. Alan Turing committed suicide via cyanide poisoning. Jeff Goldnlum starred in the 1986 movie The Fly. 3) Weather phenomena refers to types of weather caused conditions such as cyclones, storms, and tsunamis. Elton John was knighted by the Queen of England. Alice's Adventures in Wonderland was published in 1865 Subway hired Jared Fogle as a spokesman for their sandwich shops. |
| power Rachel: Is greed the most prevalent of the Seven Deadly Sins? Stephen: | Greed is a longing for wealth and power. | 1) Bing is a search engine, which is a digital object. The seven deadly sins are: pride, greed, wrath, envy, lust, gluttony, and sloth. 2) Adherents to Catholicism subscribe to the notion of the '7 deadly sins'. Pea pods on average have 5 to 6 peas inside. Family Guy and American Dad are both Fox Animated Sitcoms animated by Seth MacFarlane. 3) Greed is a longing for wealth and power. Lieutenant the second junior-most or in some cases the junior-most commissioned officer in the armed forces, fire services, police, and other organizations of many nations. | 1) Greed is a longing for wealth and power. Lieutenant the second junior-most or in some cases the junior-most commissioned officer in the armed forces, fire services, police, and other organizations of many nations. 2) Bing is a search engine, which is a digital object. The seven deadly sins are: pride, greed, wrath, envy, lust, gluttony, and sloth. 3) Water polo is a sport played by teams of seven competitors The biblical Sarah lived to the age of 127. Glenn Beck is a right wing commentator known for strong opinions and serious tone. |
Rachel: Would it be difficult for Kami Rita to climb Mount Emei? Stephen: | Kami Rita has climbed Mount Everest 24 times. | 1) The MLB World Series is held annually in a stadium belonging to one of its teams Kami Rita has climbed Mount Everest 24 times. Electronics company Casio was founded in 1946. 2) The Golden Gate Bridge is one of the most popular suicide spots in the USA. Mount Emei is a 70 ton mountain located in China. 3) Tenzing Norgay was a mountaineer that climbed Mount Everest in 1953. J.K Rowling's top sellers are her Harry Potter series. Goblin sharks are carnivores that subsist on other fish, cephalopods and crustaceans A | 1) Mount Emei is a 70 ton mountain located in China. A central tenet of the Catholic Church is a one-to-one match between man and woman. 2) The MLB World Series is held annually in a stadium belonging to one of its teams Kami Rita has climbed Mount Everest 24 times. Electronics company Casio was founded in 1946. 3) Tenzing Norgay was a mountaineer that climbed Mount Everest in 1953. J.K Rowling's top sellers are her Harry Potter series. Goblin sharks are carnivores that subsist on other fish, cephalopods and crustaceans A |
| armed forces Rachel: Would Gomer Pyle salute a lieutenant? Stephen: | Gomer Pyle was a character on a television sitcom | 1) The Empire Award for Best Newcomer was awarded for an actor in their debut role. Many people do not notice depression in their friends or loved ones. Gomer Pyle was a character on a television sitcom China accounts for 4. 2) Lieutenant the second junior-most or in some cases the junior-most commissioned officer in the armed forces, fire services, police, and other organizations of many nations. 3) is a 13 member kpop group Nicole Kidman supports the Nashville Predators and has been photographed almost nightly throughout the season. A soldier is a member of the armed forces. The Yale University student body consists of 12,385 people according to a 2015 poll. | 1) Jane Austen was the second youngest of 8 children. The Confederate States Army was clad in cadet gray uniforms. 2) Cast iron skillets can scratch or crack flat top stoves. In 1965 the president announced an intention to increase the amount of troops to 125,000 Glutamic acid is an amino acid and neurotransmitter The Holy Land is sacred to Judaism, Islam and Christianity The lunar surface 3) Many people do not notice depression in their friends or loved ones. Gomer Pyle was a character on a television sitcom China accounts for 4.6 trillion or 12. |
| of stereoisomers Rachel: Is it bad to have lactic acid in your body? Stephen: | The body naturally produces and uses lactic acid to convert glucose into energy | 1) Coffee can only be grown in subtropical and equatorial climates The body naturally produces and uses lactic acid to convert glucose into energy Solvent refers to the assets of a project being greater than the liabilities. 2) The Mona Lisa is in the Louvre. Tricarboxylic acid as an acid that manifests itself in fruits as citric acid. 3) The rings are made mostly of dust and particles. Myocardial infarction is a problem in the heart. Chlorine prevents algae from growing in pools Grand master is the highest title a chess player can get. | 1) Tricarboxylic acid as an acid that manifests itself in fruits as citric acid. The S in C-SPAN refers to Satellite. 2) Coffee can only be grown in subtropical and equatorial climates The body naturally produces and uses lactic acid to convert glucose into energy Solvent refers to the assets of a project being greater than the liabilities. 3) Bodybuilders want to build muscle and keep fat low When milk becomes acidic, the water and fats separate from each other. The next presidential elections will take place in November of 2020 Apartheid lasted from 1948 until the early 1990s. |
| Stephen: form of alternative medicine Rachel: Are some chiropractic manipulations dangerous? Stephen: | Manipulations of the neck can lead to complications such as stroke or paralysis. | 1) The Order of the British Empire is awarded to people that have made significant contributions to the United Kingdom The American Flag is colored red, white, and blue. Manipulations of the neck can lead to complications such as stroke or paralysis. 2) Dementia refers to various disorders of the brain. Surgery is used to correct medical problems or make physical alterations to the body Conan the Barbarian is a comic book character. 3) Macaques grow up to 28 inches in length The fairy tale character Snow White was friends with seven dwarfs. Acupuncture doesn't usually feel painful for most people. Lee Sedol is a former South Korean professional Go player of 9 dan rank. | 1) Nikola Tesla built a facility called the Wardenclyffe Tower in Shoreham, New York Doctors of Homeopathy are practitioners of "alternative medicine" Lord Voldemort is a powerful wizard from the Harry Potter Series. 2) Walt Disney died in 1966 Snakebites are dangerous because they inject venom into blood streams. Pacifists are a group opposed to violence and war. "Marco Polo" is a popular game among children and adults played while swimming. 3) Macaques grow up to 28 inches in length The fairy tale character Snow White was friends with seven dwarfs. Acupuncture doesn't usually feel painful for most people. Lee Sedol is a former South Korean professional Go player of 9 dan rank. |
| British–American film director, screenwriter, and producer Rachel: Could Christopher Nolan's movies finance Cyprus's entire GDP? Stephen: | The films of Christopher Nolan have grossed around 4.7 billion at the box office. | 1) The Easter Bunny is a symbol of the Christian holiday of Easter The films of Christopher Nolan have grossed around 4.7 billion at the box office. The Constitution of the Philippines is a document ratified in 1987 MP3 is a file compression format for audio recordings Friday the 13th is known as 2) variety of Mexican and Tex-Mex foods that include tacos, burritos, quesadillas, and nachos. Iceland had a nominal GDP of $27 billion as of a 2018 estimate. As of 2020, The Rush Limbaugh Show has been on the airwaves since 1988. 3) The Coen brothers direct films primarily using English The B-52 bomber plane cost 60 million dollars in 2018. David Lynch is a director that created the television show Twin Peaks. | 1) The Easter Bunny is a symbol of the Christian holiday of Easter The films of Christopher Nolan have grossed around 4.7 billion at the box office. The Constitution of the Philippines is a document ratified in 1987 MP3 is a file compression format for audio recordings Friday the 13th is known as 2) the right to vote in presidential elections. Microsoft Excel is a computer program Adam Sandler is Jewish. Christopher Nolan rose to fame in large part because of his trilogy of Batman movies released from 2005 to 2012 Green is a color made by mixing blue 3) The Coen brothers direct films primarily using English The B-52 bomber plane cost 60 million dollars in 2018. David Lynch is a director that created the television show Twin Peaks. |
| Stephen: American actor, comedian, screenwriter, and producer Rachel: Would the average American family find Adam Sandler's home to be too small? Stephen: | The average American family has about 3 people in it. | 1) Immersion Baptism is the practice of submerging people underwater for a religious ritual. The average American family has about 3 people in it. The Social Democratic Party of Germany was founded in 1863. 2) the right to vote in presidential elections. Microsoft Excel is a computer program Adam Sandler is Jewish. Christopher Nolan rose to fame in large part because of his trilogy of Batman movies released from 2005 to 2012 Green is a color made by mixing blue 3) Silicon is a key material for the production of semiconductor chips. Adam Sandler and Cole Sprouse starred together in "Big Daddy". Lil Wayne has four children. | 1) Adherents to Catholicism subscribe to the notion of the '7 deadly sins'. Pea pods on average have 5 to 6 peas inside. Family Guy and American Dad are both Fox Animated Sitcoms animated by Seth MacFarlane. 2) the right to vote in presidential elections. Microsoft Excel is a computer program Adam Sandler is Jewish. Christopher Nolan rose to fame in large part because of his trilogy of Batman movies released from 2005 to 2012 Green is a color made by mixing blue 3) Saddam Hussein died on December 30th, 2006. Steve Martin is allergic to shellfish. Mickey Mouse is a Disney character that has starred in numerous movies and TV specials. |
| Stephen: Medical condition where poor blood flow to the brain causes cell death Rachel: Did Dale Jr.'s father crash his car due to a stroke? Stephen: | Dale Earnhardt Jr. is his late father's namesake. | 1) The Fibonaacci number is a sequence of numbers that adds a number to the one before it and goes on forever. Dale Earnhardt Jr. is his late father's namesake. 2) A caracal is a small wild cat Dale Jr. and his father Dale Sr. last raced together at the Daytona 500 in 2001. 3) Cerebral palsy is a disease that results from damage to a young person's brain. The bottom of the gulf is one of the world's largest ship cemeteries. | 1) in 1908 Multiple Sclerosis is a progressive condition affecting the brain and spinal chord. Saddam Hussein died on December 30th, 2006. 2) Parsifal was loosely based on a poem about Percival Hypothermia typically occurs from exposure to extreme cold. Traffic collisions sometimes result in extremely expensive physical damage. Goofy is an anthropomorphic dog character. 3) Cerebral palsy is a disorder caused by damage to fetal or infant brains. Bitcoin was launched as a currency in 2009. |
| Stephen: Academic tertiary education, such as from colleges and universities Rachel: Did Emma Stone pursue a higher education? Stephen: | Higher education, also called post-secondary education, third-level or tertiary education, is an optional final stage of formal learning that occurs after completion of secondary education. | 1) Shivambu is another term for 'Urine Therapy', an alternative belief about healing with urine. Higher education, also called post-secondary education, third-level or tertiary education, is an optional final stage of formal learning that occurs after completion of secondary education. 2) Superbad features two main characters that are nerds on a quest for love, and ends with them being victorious. Johns Hopkins University had an endowment of $6. 3) The qualifications to be a tax collector in the US inlude a bachelor's degree in accounting. Naomi Watts starred in King Kong (2005 film). Gay men can have any of the various sex organs that humans have. | 1) Higher education, also called post-secondary education, third-level or tertiary education, is an optional final stage of formal learning that occurs after completion of secondary education. 2) Alfa Romeo makes cars. Some of the things geologists study include gemstones, minerals, and stone The Toronto Star has a classifieds section QWERTY keyboards are an alphabet key layout that were first used on typrwriters 3) College degrees require at least 2 years of study to obtain. LeBron James is 6 feet 9 inches tall. |
At this time, we haven't found a reranking cross-encoder that can consistently boost the retrieval accuracy of our best embedding models. That said, you can use accurate embedding models as query-passage matching models, alongside a low-resource embedding model. This combination's accuracy nearly matches that of the best embedding models from this library.
The following tests were performed by configuring a memory to use em-MiniLM-p1-01
as the embedding model and em:em-distilroberta-p5-01 as the matching/reranking model
with different values of the reranking_k_factor setting.
| reranking_k_factor | qrecc @3 | qrecc @10 | strategyqa @3 | strategyqa @10 | msmarco @3 | msmarco @10 |
|---|---|---|---|---|---|---|
| 1 | 76.97 | 79.75 | 84.65 | 89.05 | 76.62 | 79.33 |
| 2 | 77.95 | 81.39 | 86.55 | 92.85 | 77.89 | 81.41 |
| 3 | 78.33 | 82.53 | 87.10 | 93.90 | 78.61 | 82.85 |
| 5 | 78.60 | 83.19 | 87.35 | 94.60 | 78.97 | 83.48 |
| 8 | 78.93 | 84.44 | 87.25 | 95.15 | 79.33 | 83.84 |
| 10 | 78.93 | 84.50 | 87.45 | 95.55 | 79.15 | 84.21 |
Model em-MiniLM-p1-01 stores a single embedding of size 384 per chunk, while model
em-distilroberta-p5-01 produces 5 storage embeddings per chunk, of size 768. The number of storage embeddings
has little impact when the embedding model is used as a query-passage matching model.
It does matter, however, in terms of chunk storage capacity when embeddings
are attached to chunks. In the case of the two models in question, the difference
in footprint is 10-fold.
Using a matching model does come with a performance overhead in every query, but in the context of an LLM-based chat agent implementation, the overhead should be unnoticeable.