I clarify tips on how to construct an app that generates a number of selection questions (MCQs) on any user-defined topic. The app is extracting Wikipedia articles which can be associated to the consumer’s request and makes use of RAG to question a chat mannequin to generate the questions.
I’ll show how the app works, clarify how Wikipedia articles are retrieved, and present how these are used to invoke a chat mannequin. Subsequent, I clarify the important thing parts of this app in additional element. The code of the app is accessible here.
App Demo
The gif above exhibits the consumer coming into the training context, the generated MCQ and the suggestions after the consumer submitted a solution.
On the first display the consumer describes the context of the MCQs that must be generated. After urgent “Submit Context” the app searches for Wikipedia articles which content material matches the consumer question.
The app splits every Wikipedia web page into sections and scores them based mostly on how carefully they match the consumer question. These scores are used to pattern the context of the subsequent query which is displayed within the subsequent display with 4 selections to reply. The consumer can choose a selection and submit it by “Submit Reply”. Additionally it is doable to skip this query by way of “Subsequent Query”. On this case it’s thought-about that the query didn’t meet the consumer’s expectation. Will probably be prevented to make use of the context of this query for the technology of following questions. To finish the session the consumer can select “Finish MCQ”.
The subsequent display after the consumer submitted a solution exhibits if the reply was appropriate and gives an extra rationalization. Following, the consumer can both get a brand new query by way of “Subsequent Query” or finish the session with “Finish MCQ”.
The tip session display exhibits what number of questions had been appropriately and wrongly answered. Moreover, it additionally accommodates the variety of questions the consumer rejected by way of “Subsequent Query”. If the consumer selects “Begin New Session” the beginning display shall be displayed the place a brand new context for the subsequent session could be supplied.
Idea
The purpose of this app is to supply prime quality and up-to-date questions on any user-defined subject. Thereby consumer suggestions is taken into account to make sure that the generated questions are assembly the consumer’s expectations.
To retrieve high-quality and up-to-date context, Wikipedia articles are chosen with respect to the consumer’s question. Every article is break up into sections whereas each part is scored based mostly on its similarity with the consumer question. If the consumer rejects a query the respective part rating shall be downgraded to scale back the probability of sampling this part once more.
This course of could be separated into two workflows:
- Context Retrieval
- Query Era
That are described under.
Context Retrieval
The workflow how the context of the MCQs is derived from Wikipedia based mostly on the consumer question is proven under.
The consumer inserts the question that describes the context of the MCQs initially display. An instance of the consumer question might be: “Ask me something about stars and planets”.
To effectively seek for Wikipedia articles this question is transformed into key phrases. The key phrases of the question above are: “Stars”, “Planets”, “Astronomy”, “Photo voltaic System”, and “Galaxy”.
For every key phrase a Wikipedia search is executed of which the highest three pages are chosen. Not every of those 15 pages are a very good match to the question supplied by the consumer. To take away irrelevant pages on the earliest doable stage the vector similarity of the embedded consumer question and web page excerpt is calculated. Pages which similarity is under a threshold are filtered out. In our instance 3 of 15 pages had been eliminated.
The remaining pages are learn and divided into sections. As not the complete web page content material could also be associated to the consumer question, splitting the pages into sections permits to pick elements of the web page that match particularly properly to the consumer question. Therefore, for every part the vector similarity towards the consumer question is calculated and sections with low similarity are filtered out. The remaining 12 pages contained 305 sections of which 244 had been saved after filtering.
The final step of the retrieval workflow is to assign a rating to every part with respect to the vector similarity. This rating will later be used to pattern sections for the query technology.
Query Era
The workflow to generate a brand new MCQ is proven under:
Step one is to pattern one part with respect to the part scores. The textual content of this part is inserted along with the consumer question right into a immediate to invoke a chat mannequin. The chat mannequin returns a json formatted response that accommodates the query, reply selections, and an evidence of the proper reply. In case the context supplied is just not appropriate to generate a MCQ that addresses the consumer question the chat mannequin is instructed to return a key phrase to determine that the query technology was not profitable.
If the query technology was profitable, the questions and the reply selections are exhibited to the consumer. As soon as the consumer submits a solution it’s evaluated if the reply was appropriate, and the reason of the proper reply is proven. To generate a brand new query the identical workflow is repeated.
In case the query technology was not profitable, or the consumer rejected the query by clicking on “Subsequent Query” the rating of the part that was chosen to generate the immediate is downgraded. Therefore, it’s much less possible that this part shall be chosen once more.
Key Parts
Subsequent, I’ll clarify some key parts of the workflows in additional element.
Extracting Wiki Articles
Wikipedia articles are extracted in two steps: First a search is run to search out appropriate pages. After filtering the search outcomes, the pages separated by sections are learn.
Search requests are despatched to this URL. Moreover, a header containing the requestor’s contact data and a parameter dictionary with the search question and the variety of pages to be returned. The output is in json format that may be transformed to a dictionary. The code under exhibits tips on how to run the request:
headers = {'Person-Agent': os.getenv('WIKI_USER_AGENT')}
parameters = {'q': search_query, 'restrict': number_of_results}
response = requests.get(WIKI_SEARCH_URL, headers=headers, params=parameters)
page_info = response.json()['pages']After filtering the search outcomes based mostly on the pages’ excerpts the textual content of the remaining pages is imported utilizing wikipediaapi:
import wikipediaapi
def get_wiki_page_sections_as_dict(page_title, sections_exclude=SECTIONS_EXCLUDE):
wiki_wiki = wikipediaapi.Wikipedia(user_agent=os.getenv('WIKI_USER_AGENT'), language='en')
web page = wiki_wiki.web page(page_title)
if not web page.exists():
return None
def sections_to_dict(sections, parent_titles=[]):
outcome = {'Abstract': web page.abstract}
for part in sections:
if part.title in sections_exclude: proceed
section_title = ": ".be a part of(parent_titles + [section.title])
if part.textual content:
outcome[section_title] = part.textual content
outcome.replace(sections_to_dict(part.sections, parent_titles + [section.title]))
return outcome
return sections_to_dict(web page.sections)To entry Wikipedia articles, the app makes use of wikipediaapi.Wikipedia, which requires a user-agent string for identification. It returns a WikipediaPage object which accommodates a abstract of the web page, web page sections with the title and the textual content of every part. Sections are hierarchically organized which means every part is one other WikipediaPage object with one other checklist of sections which can be the subsections of the respective part. The operate above reads all sections of a web page and returns a dictionary that maps a concatenation of all part and subsection titles to the respective textual content.
Context Scoring
Sections that match higher to the consumer question ought to get a better likelihood of being chosen. That is achieved by assigning a rating to every part which is used as weight for sampling the sections. This rating is calculated as follows:
[s_{section}=w_{rejection}s_{rejection}+(1-w_{rejection})s_{sim}]
Every part receives a rating based mostly on two elements: how usually it has been rejected, and the way carefully its content material matches the consumer question. These scores are mixed right into a weighted sum. The part rejection rating consists of two parts: the variety of how usually the part’s web page has been rejected over the best variety of web page rejections and the variety of this part’s rejections over the best variety of part rejections:
[s_{rejection}=1-frac{1}{2}left( frac{n_{page(s)}}{max_{page}n_{page}} + frac{n_s}{max_{s}n_s} right)]
Immediate Engineering
Immediate engineering is a vital facet of the Studying App’s performance. This app is utilizing two prompts to:
- Get key phrases for the wikipedia web page search
- Generate MCQs for sampled context
The template of the key phrase technology immediate is proven under:
KEYWORDS_TEMPLATE = """
You are an assistant to generate key phrases to seek for Wikipedia articles that include content material the consumer needs to be taught.
For a given consumer question return at most {n_keywords} key phrases. Be certain each key phrase is an efficient match to the consumer question.
Slightly present fewer key phrases than key phrases which can be much less related.
Directions:
- Return the key phrases separated by commas
- Don't return the rest
"""This technique message is concatenated with a human message containing the consumer question to invoke the Llm mannequin.
The parameter n_keywords set the utmost variety of key phrases to be generated. The directions make sure that the response could be simply transformed to a listing of key phrases. Regardless of these directions, the LLM usually returns the utmost variety of key phrases, together with some much less related ones.
The MCQ immediate accommodates the sampled part and invokes the chat mannequin to reply with a query, reply selections, and an evidence of the proper reply in a machine-readable format.
MCQ_TEMPLATE = """
You're a studying app that generates multiple-choice questions based mostly on academic content material. The consumer supplied the
following request to outline the training content material:
"{user_query}"
Based mostly on the consumer request, following context was retrieved:
"{context}"
Generate a multiple-choice query immediately based mostly on the supplied context. The proper reply should be explicitly said
within the context and may at all times be the primary possibility within the selections checklist. Moreover, present an evidence for why
the proper reply is appropriate.
Variety of reply selections: {n_choices}
{previous_questions}{rejected_questions}
The JSON output ought to observe this construction (for variety of selections = 4):
{{"query": "Your generated query based mostly on the context", "selections": ["Correct answer (this must be the first choice)","Distractor 1","Distractor 2","Distractor 3"], "rationalization": "A short rationalization of why the proper reply is appropriate."}}
Directions:
- Generate one multiple-choice query strictly based mostly on the context.
- Present precisely {n_choices} reply selections, making certain the primary one is the proper reply.
- Embrace a concise rationalization of why the proper reply is appropriate.
- Don't return the rest than the json output.
- The supplied rationalization mustn't assume the consumer is conscious of the context. Keep away from formulations like "As said within the textual content...".
- The response should be machine readable and never include line breaks.
- Test whether it is doable to generate a query based mostly on the supplied context that's aligned with the consumer request. If it isn't doable set the generated query to "{fail_keyword}".
"""
The inserted parameters are:
user_query: textual content of consumer questioncontext: textual content of sampled partn_choices: variety of reply selectionsprevious_questions: instruction to not repeat earlier questions with checklist of all earlier questionsrejected_questions: instruction to keep away from questions of comparable nature or context with checklist of rejected questionsfail_keyword: key phrase that signifies that query couldn’t be generated
Together with earlier questions reduces the prospect that the chat mannequin repeats questions. Moreover, by offering rejected questions, the consumer’s suggestions is taken into account when producing new questions. The instance ought to make sure that the generated output is within the appropriate format in order that it may be simply transformed to a dictionary. Setting the proper reply as the primary selection avoids requiring an extra output that signifies the proper reply. When displaying the alternatives to the consumer the order of selections is shuffled. The final instruction defines what output must be supplied in case it isn’t doable to generate a query matching the consumer question. Utilizing a standardized key phrase makes it straightforward to determine when the query technology has failed.
Streamlit App
The app is constructed utilizing Streamlit, an open-source app framework in Python. Streamlit has many capabilities that enable so as to add web page components with just one line of code. Like for instance the component through which the consumer can write the question is created by way of:
context_text = st.text_area("Enter the context for MCQ questions:")the place context_text accommodates the string, the consumer has written. Buttons are created with st.button or st.radio the place the returned variable accommodates the knowledge if the button has been pressed or what worth has been chosen.
The web page is generated top-down by a script that defines every component sequentially. Each time the consumer is interacting with the web page, e.g. by clicking on a button the script could be re-run with st.rerun(). When re-running the script, it is very important carry over data from the earlier run. That is finished by st.session_state which might include any objects. For instance, the MCQ generator occasion is assigned to session states as:
st.session_state.mcq_generator = MCQGenerator()in order that when the context retrieval workflow has been executed, the discovered context is accessible to generate a MCQ on the subsequent web page.
Enhancements
There are a lot of choices to boost this app. Past Wikipedia, customers might additionally add their very own PDFs to generate questions from customized supplies—comparable to lecture slides or textbooks. This may allow the consumer to generate questions on any context, for instance it might be used to arrange for exams by importing course supplies.
One other facet that might be improved is to optimize the context choice to reduce the variety of rejected questions by the consumer. As a substitute of updating scores, additionally a ML mannequin might be educated to foretell how possible it’s {that a} query shall be rejected with respect to options like similarity to accepted and rejected questions. Each time one other query is rejected this mannequin might be retrained.
Additionally, the generated query might be saved in order that when a consumer needs to repeat the training train these questions might be used once more. An algorithm might be utilized to pick beforehand wrongly answered questions extra incessantly to give attention to bettering the learner’s weaknesses.
Abstract
This text showcases how retrieval-augmented technology (RAG) can be utilized to construct an interactive studying app that generates high-quality, context-specific multiple-choice questions from Wikipedia articles. By combining keyword-based search, semantic filtering, immediate engineering, and a feedback-driven scoring system, the app dynamically adapts to consumer preferences and studying objectives. Leveraging instruments like Streamlit allows fast prototyping and deployment, making this an accessible framework for educators, college students, and builders alike. With additional enhancements—comparable to customized doc uploads, adaptive query sequencing, and machine learning-based rejection prediction—the app holds robust potential as a flexible platform for customized studying and self-assessment.
Additional Studying
To be taught extra about RAGs I can advocate these articles from Shaw Talebi and Avishek Biswas. Harrison Hoffman wrote two glorious tutorials on embeddings and vector databases and building an LLM RAG Chatbot. The best way to handle states in streamlit could be present in Baertschi’s article.
If not said in any other case, all pictures had been created by the writer.
