Chat Soufflé

Loading…

<html><h5>Custom assessment <i class="fas fa-caret-right"></i></h5></html> !! Part 1: select and evaluate your MC question <html> <p>In this track, you’ll focus on using ChatGPT to improve an existing multiple choice question. This could be a question you have developed or encountered in the past. You can select any question as long as it is not AI generated. </p> <p>Before you involve ChatGPT in revising your selected question, we want to get your opinion on your assessment item. That will give you a baseline from which to see how much ChatGPT helped you.</p> <li>Evaluate your multiple choice question based on the criteria for effective distractors (review criteria below if needed). (Don’t enter it into ChatGPT yet.) </ol> <h3>Is the assessment unacceptable/weak/adequate in your opinion?</h3> <form id="custom-track-1" class="pure-form" data-parent="custom-track"> <label for="custom-track-1-opt-1" class="pure-checkbox"> <input type="radio" name="custom-track-1-choice" id="custom-track-1-opt-1" value="human-first-attempt-at-custom-distractor-was-unacceptable" onclick="checkOptionSelected('custom-track-1')" />Unacceptable </label> <label for="custom-track-1-opt-2" class="pure-checkbox"> <input type="radio" name="custom-track-1-choice" id="custom-track-1-opt-2" value="human-first-attempt-at-custom-distractor-was-weak" onclick="checkOptionSelected('custom-track-1')" />Weak </label> <label for="custom-track-1-opt-3" class="pure-checkbox"> <input type="radio" name="custom-track-1-choice" id="custom-track-1-opt-3" value="human-first-attempt-at-custom-distractor-was-adequate" onclick="checkOptionSelected('custom-track-1')" />Adequate </label> <button type="button" onclick="sendFormXAPIStatementNoVoid('custom-track-1'); checkOptionSelected('custom-track-1'); customButtonHandler('custom assessment track 2')" id="custom-track-1-submit" class="pure-button pure-button-primary" disabled>Submit your choice</button> </form> <br /> <div class="dropdown-container"> <div class="dropdown-header" onclick="toggleContent()"> <h3><span class="light-gray-text">Review criteria</span></h3> <span class="arrow">▾</span> </div> <div id="custom-track-1-content-dropdown" class="dropdown-text-content" style="display: none;"> <p>Use the criteria below to evaluate ChatGPT’s assessment questions.</p> <div class="tip-box"> <div class="tip-icon"> <i class="fas fa-pen"></i> </div> <div class="tip-content"> <h3 class="tip-title">Evaluation criteria</h3> <ul> <li>Write options that are congruent in length.</li> <li>Write options that use parallel grammatical structure (e.g. if answer(s) start with a verb, so should the distractors).</li> <li>Avoid “True/False,” “All of the Above,” or “Both A and B Only” (or other such combinations).</li> <li>Use plausible but incorrect distractors.</li> <li>Use only truly correct answers.</li> <li>Avoid negative phrasing.</li> </ul> </div> </div> <p>Review assessment writing best practices at these external resources before moving on:</p> <ul> <li><a id="vanderbilt-external-link-review-custom-1" class="external-link" href="https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/" target="blank">Writing Good Multiple Choice Test Questions - Vanderbilt University Center for Teaching</a></li> <li><a id="conneticut-external-link-review-custom-1" class="external-link" href="https://kb.ecampus.uconn.edu/2020/09/30/writing-effective-multiple-choice-questions-2/" target="blank">Writing Effective Multiple Choice Questions - University of Connecticut Knowledge Base</a></li> </ul> </div> </html> <script> function toggleContent() { const content = document.getElementById('custom-track-1-content-dropdown'); const arrow = document.querySelector('.arrow'); if (content.style.display === 'none') { content.style.display = 'block'; arrow.classList.add('arrow-up'); } else { content.style.display = 'none'; arrow.classList.remove('arrow-up'); } } </script> <<run setup.trackStartsTrack2++>> <<run setup.currentIterationIdTrack2 = "custom_assessment_track_loop_" + (setup.trackStartsTrack2)>> <<run setup.iterationActivities[setup.currentIterationIdTrack2] = []>>

<html><h5>Custom assessment <i class="fas fa-caret-right"></i></h5></html> !! Criteria for effective distractors <html> <h3>Use the criteria below to evaluate ChatGPT’s assessment questions.</h3> <ul> <li>Write options that are congruent in length.</li> <li>Write options that use parallel grammatical structure (e.g. if answer(s) start with a verb, so should the distractors).</li> <li>Avoid “True/False,” “All of the Above,” or “Both A and B Only” (or other such combinations).</li> <li>Use plausible but incorrect distractors.</li> <li>Use only truly correct answers.</li> <li>Avoid negative phrasing.</li> </ul> <div class="tip-box"> <div class="tip-icon"> <i class="fas fa-pen"></i> </div> <div class="tip-content"> <h3 class="tip-title">Optional</h3> <p>Review assessment writing best practices at these external resources before moving on:</p> <ul> <li><a id="vanderbilt-external-link-2" class="external-link" href="https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/" target="blank">Writing Good Multiple Choice Test Questions - Vanderbilt University Center for Teaching</a></li> <li><a id="conneticut-external-link-2" class="external-link" href="https://kb.ecampus.uconn.edu/2020/09/30/writing-effective-multiple-choice-questions-2/" target="blank">Writing Effective Multiple Choice Questions - University of Connecticut Knowledge Base</a></li> </ul> </div> </div> <div> <hr> <a id="provided-criteria" class="anchor-nav" onclick="customLinkHandler('custom assessment track 2')">On to Part 2!<i class="fas fa-arrow-right"></i></a> </div> </html>

<html><h5>Custom assessment <i class="fas fa-caret-right"></i></h5></html> !! Refine part 2 <html> <p>Alright, now let’s use ChatGPT to revise the question you just evaluated. It helps to give some context when asking ChatGPT to refine an existing multiple choice assessment question. In her article <a id="mega-prompts-external-link-provided-2a" class="external-link" href="https://drphilippahardman.substack.com/p/introducing-chatgpt-edu-mega-prompts?utm_source=profile&utm_medium=reader2" target="blank">“Introducing: ChatGPT Edu-Mega-Prompts”</a> Dr Philippa Hardman suggests specifying a role, a task, and an output in your prompt to ChatGPT.</p> <ol start="1"> <li>Give ChatGPT some context about the training. Edit this sample prompt to match your situation and enter it into ChatGPT.</li> <ul> <li><div class="prompt-text-area"> <div class="prompt-text-header"> <h3><span class="light-gray-text">Role</span></h3> <button class="copyButton" title="Copy to Clipboard"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-clipboard"> <path d="M8 4H6c-1.1 0-2 .9-2 2v12c0 1.1.9 2 2 2h12c1.1 0 2-.9 2-2V10l-4-4h-6zM8 4h6v2H8V4zM8 6v2h6V6H8z"/> </svg> Copy </button> <span class="copiedMessage">Copied!</span> </div> <div class="prompt-text-content"> <pre class="monospaced-text">Role: You are an expert in database design and adult learning. You are proficient at teaching database admins how to learn new software tools and at assessing their understanding.</pre> </div> </div><br > </li> <li><div class="prompt-text-area"> <div class="prompt-text-header"> <h3><span class="light-gray-text">Task</span></h3> <button class="copyButton" title="Copy to Clipboard"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-clipboard"> <path d="M8 4H6c-1.1 0-2 .9-2 2v12c0 1.1.9 2 2 2h12c1.1 0 2-.9 2-2V10l-4-4h-6zM8 4h6v2H8V4zM8 6v2h6V6H8z"/> </svg> Copy </button> <span class="copiedMessage">Copied!</span> </div> <div class="prompt-text-content"> <pre class="monospaced-text">Task: Refine existing multiple choice assessment questions I provide so that they are more effective at accurately assessing the learner’s knowledge and understanding.</pre> </div> </div> </li><br > <li><div class="prompt-text-area"> <div class="prompt-text-header"> <h3><span class="light-gray-text">Output</span></h3> <button class="copyButton" title="Copy to Clipboard"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-clipboard"> <path d="M8 4H6c-1.1 0-2 .9-2 2v12c0 1.1.9 2 2 2h12c1.1 0 2-.9 2-2V10l-4-4h-6zM8 4h6v2H8V4zM8 6v2h6V6H8z"/> </svg> Copy </button> <span class="copiedMessage">Copied!</span> </div> <div class="prompt-text-content"> <pre class="monospaced-text">Output: Multiple choice assessment question with four answer options. Indicate which answer option is correct. Do not use “all of the above” or “none of the above” as answer options.</pre> </div> </div> </li><br > </ul> <li>Evaluate your multiple choice question based on the criteria for effective distractors (review criteria below if needed). (Don’t enter it into ChatGPT yet.) </ol> <p>Evaluate the revised version based on the criteria for effective distractors (review criteria below if needed). Is the assessment now worse/equal/better in your opinion?</p> <br > <form id="custom-track-2" class="pure-form" data-parent="custom-track"> <label for="custom-track-2-opt-1" class="pure-checkbox"> <input type="radio" name="custom-track-2-choice" id="custom-track-2-opt-1" value="chatgpt-assessment-is-worse" onclick="checkOptionSelected('custom-track-2')" />Worse </label> <label for="custom-track-2-opt-2" class="pure-checkbox"> <input type="radio" name="custom-track-2-choice" id="custom-track-2-opt-2" value="chatgpt-assessment-is-equal" onclick="checkOptionSelected('custom-track-2')" />Equal </label> <label for="custom-track-2-opt-3" class="pure-checkbox"> <input type="radio" name="custom-track-2-choice" id="custom-track-2-opt-3" value="chatgpt-assessment-is-better" onclick="checkOptionSelected('custom-track-2')" />Better </label> <button type="button" onclick="sendFormXAPIStatementNoVoid('custom-track-2'); checkOptionSelected('custom-track-2'); customButtonHandler('custom assessment track 2 freeform')" id="custom-track-2-submit" class="pure-button pure-button-primary" disabled>Submit your choice</button> </form> <br /> <div class="dropdown-container"> <div class="dropdown-header" onclick="toggleContent()"> <h3><span class="light-gray-text">Review criteria</span></h3> <span class="arrow">▾</span> </div> <div id="custom-track-2-content-dropdown" class="dropdown-text-content" style="display: none;"> <p>Use the criteria below to evaluate ChatGPT’s assessment questions.</p> <div class="tip-box"> <div class="tip-icon"> <i class="fas fa-pen"></i> </div> <div class="tip-content"> <h3 class="tip-title">Evaluation criteria</h3> <ul> <li>Write options that are congruent in length.</li> <li>Write options that use parallel grammatical structure (e.g. if answer(s) start with a verb, so should the distractors).</li> <li>Avoid “True/False,” “All of the Above,” or “Both A and B Only” (or other such combinations).</li> <li>Use plausible but incorrect distractors.</li> <li>Use only truly correct answers.</li> <li>Avoid negative phrasing.</li> </ul> </div> </div> <p>Review assessment writing best practices at these external resources before moving on:</p> <ul> <li><a id="vanderbilt-external-link-review-custom-2" class="external-link" href="https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/" target="blank">Writing Good Multiple Choice Test Questions - Vanderbilt University Center for Teaching</a></li> <li><a id="conneticut-external-link-review-custom-2" class="external-link" href="https://kb.ecampus.uconn.edu/2020/09/30/writing-effective-multiple-choice-questions-2/" target="blank">Writing Effective Multiple Choice Questions - University of Connecticut Knowledge Base</a></li> </ul> </div> </html> <script> setTimeout(function() { const copyButtons = document.querySelectorAll(".copyButton"); copyButtons.forEach(button => { button.addEventListener("click", function() { const text = this.parentElement.nextElementSibling.innerText; navigator.clipboard.writeText(text) .then(() => { console.log("Text copied to clipboard"); const copiedMessage = this.nextElementSibling; setTimeout(() => { copiedMessage.classList.remove("show"); }, 1000); this.innerText = "Copied"; this.classList.add("copied"); }) .catch(err => { console.error("Error copying text: ", err); }); }); }); }, 100); </script> <script> function toggleContent() { const content = document.getElementById('custom-track-2-content-dropdown'); const arrow = document.querySelector('.arrow'); if (content.style.display === 'none') { content.style.display = 'block'; arrow.classList.add('arrow-up'); } else { content.style.display = 'none'; arrow.classList.remove('arrow-up'); } } </script>

<html><h5>Custom assessment <i class="fas fa-caret-right"></i></h5></html> !! Reflect on part 2 <html> <p>Reflect: why do you think the new version changed the way it did or mostly stayed the same?</p> <div class="textarea-Wrapper" data-parent="custom-track"> <textarea class="custom-textarea" id="custom-track-1-freeform" type="text" value="" oninput="checkTextareaNotEmpty('custom-track-1-freeform', 'send-custom-track-1-freeform', '/track-1-textarea')"></textarea> </div> <button class="pure-button pure-button-primary" id="send-custom-track-1-freeform" onclick="sendTextareaXAPIStatement('custom-track-1-freeform', 'send-custom-track-1-freeform', 'custom-track-2-textarea'); customButtonHandler('custom assessment track 3')" disabled>Send your thoughts</button> <hr> <div> <a id="skip-custom-track-2-freeform" class="anchor-nav" onclick="customLinkHandler('custom assessment track 3')">Or, skip <i class="fas fa-arrow-right"></i></a> </div> </html>

<html><h5>Custom assessment <i class="fas fa-caret-right"></i></h5></html> !! Refine part 3 <html> <p>Now give ChatGPT specific instructions one at a time to tweak the question. For example:</p> <ol start="1"> <li>Revise to make the length of each answer option more similar.</li> <li>Edit the answer option “[copy text here]” to be more plausible.</li> </ol> <p>Give as many instructions as you like. When you’ve tried everything you want to, evaluate the best revised version it produced based on the criteria for effective distractors (review criteria if needed). Is the assessment now worse/equal/better in your opinion? </p> <form id="custom-track-3" class="pure-form" data-parent="custom-track"> <label for="custom-track-3-opt-1" class="pure-checkbox"> <input type="radio" name="custom-track-3-choice" id="custom-track-3-opt-1" value="after-trying-everything-chatgpt-assessment-is-worse" onclick="checkOptionSelected('custom-track-3')" />Worse </label> <label for="custom-track-3-opt-2" class="pure-checkbox"> <input type="radio" name="custom-track-3-choice" id="custom-track-3-opt-2" value="after-trying-everything-chatgpt-assessment-is-equal" onclick="checkOptionSelected('custom-track-3')" />Equal </label> <label for="custom-track-3-opt-3" class="pure-checkbox"> <input type="radio" name="custom-track-3-choice" id="custom-track-3-opt-3" value="after-trying-everything-chatgpt-assessment-is-better" onclick="checkOptionSelected('custom-track-3')" />Better </label> <button type="button" onclick="sendFormXAPIStatementNoVoid('custom-track-3'); checkOptionSelected('custom-track-3'); customButtonHandler('custom assessment track 3 freeform')" id="custom-track-3-submit" class="pure-button pure-button-primary" disabled>Submit your choice</button> </form> <br /> <div class="dropdown-container"> <div class="dropdown-header" onclick="toggleContent()"> <h3><span class="light-gray-text">Review criteria</span></h3> <span class="arrow">▾</span> </div> <div id="custom-track-3-content-dropdown" class="dropdown-text-content" style="display: none;"> <p>Use the criteria below to evaluate ChatGPT’s assessment questions.</p> <div class="tip-box"> <div class="tip-icon"> <i class="fas fa-pen"></i> </div> <div class="tip-content"> <h3 class="tip-title">Evaluation criteria</h3> <ul> <li>Write options that are congruent in length.</li> <li>Write options that use parallel grammatical structure (e.g. if answer(s) start with a verb, so should the distractors).</li> <li>Avoid “True/False,” “All of the Above,” or “Both A and B Only” (or other such combinations).</li> <li>Use plausible but incorrect distractors.</li> <li>Use only truly correct answers.</li> <li>Avoid negative phrasing.</li> </ul> </div> </div> <p>Review assessment writing best practices at these external resources before moving on:</p> <ul> <li><a id="vanderbilt-external-link-review-custom-3" class="external-link" href="https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/" target="blank">Writing Good Multiple Choice Test Questions - Vanderbilt University Center for Teaching</a></li> <li><a id="conneticut-external-link-review-custom-3" class="external-link" href="https://kb.ecampus.uconn.edu/2020/09/30/writing-effective-multiple-choice-questions-2/" target="blank">Writing Effective Multiple Choice Questions - University of Connecticut Knowledge Base</a></li> </ul> </div> </html> <script> function toggleContent() { const content = document.getElementById('custom-track-3-content-dropdown'); const arrow = document.querySelector('.arrow'); if (content.style.display === 'none') { content.style.display = 'block'; arrow.classList.add('arrow-up'); } else { content.style.display = 'none'; arrow.classList.remove('arrow-up'); } } </script>

<html><h5>Custom assessment <i class="fas fa-caret-right"></i></h5></html> !! Reflect on Part 3 <html> <h3>Why do you think the new version changed the way it did or mostly stayed the same?</h3> <div class="textarea-Wrapper" data-parent="custom-track"> <textarea class="custom-textarea" id="custom-track-3-freeform" type="text" value="" oninput="checkTextareaNotEmpty('custom-track-3-freeform', 'send-custom-track-3-freeform', '/track-1-textarea')"></textarea> </div> <button class="pure-button pure-button-primary" id="send-custom-track-3-freeform" onclick="sendTextareaXAPIStatement('custom-track-3-freeform', 'send-custom-track-3-freeform', 'custom-track-3-textarea'); customButtonHandler('custom assessment track 4')" disabled>Send your thoughts</button> <hr> <div> <a id="skip-custom-track-3-freeform" class="anchor-nav" onclick="customLinkHandler('custom assessment track 4')">Or, skip <i class="fas fa-arrow-right"></i></a> </div> </html>

<html><h5>Custom assessment <i class="fas fa-caret-right"></i></h5></html> !! Refine part 4 <html> <p>Now, <b><i>without using ChatGPT</i></b>, take the best elements from the different iterations of the multiple choice question, combining them into the best possible assessment question. Avoid adding new material other than correcting for grammar and fluidity.</p> <p>Evaluate this human-edited version based on the criteria for effective distractors (review criteria below if needed). Is the new version is worse/equal/better than the original? </p> <br > <form id="custom-track-4" class="pure-form" data-parent="custom-track"> <label for="custom-track-4-opt-1" class="pure-checkbox"> <input type="radio" name="custom-track-4-choice" id="custom-track-4-opt-1" value="human-edited-synthesis-is-worse" onclick="checkOptionSelected('custom-track-4')" />Worse </label> <label for="custom-track-4-opt-2" class="pure-checkbox"> <input type="radio" name="custom-track-4-choice" id="custom-track-4-opt-2" value="human-edited-synthesis-is-equal" onclick="checkOptionSelected('custom-track-4')" />Equal </label> <label for="custom-track-4-opt-3" class="pure-checkbox"> <input type="radio" name="custom-track-4-choice" id="custom-track-4-opt-3" value="human-edited-synthesis-is-better" onclick="checkOptionSelected('custom-track-4')" />Better </label> <button type="button" onclick="sendFormXAPIStatementNoVoid('custom-track-4'); checkOptionSelected('custom-track-4'); customButtonHandler('custom track 5')" id="custom-track-4-submit" class="pure-button pure-button-primary" disabled>Submit your choice</button> </form> <br /> <div class="dropdown-container"> <div class="dropdown-header" onclick="toggleContent()"> <h3><span class="light-gray-text">Review criteria</span></h3> <span class="arrow">▾</span> </div> <div id="custom-track-4-content-dropdown" class="dropdown-text-content" style="display: none;"> <p>Use the criteria below to evaluate ChatGPT’s assessment questions.</p> <div class="tip-box"> <div class="tip-icon"> <i class="fas fa-pen"></i> </div> <div class="tip-content"> <h3 class="tip-title">Evaluation criteria</h3> <ul> <li>Write options that are congruent in length.</li> <li>Write options that use parallel grammatical structure (e.g. if answer(s) start with a verb, so should the distractors).</li> <li>Avoid “True/False,” “All of the Above,” or “Both A and B Only” (or other such combinations).</li> <li>Use plausible but incorrect distractors.</li> <li>Use only truly correct answers.</li> <li>Avoid negative phrasing.</li> </ul> </div> </div> <p>Review assessment writing best practices at these external resources before moving on:</p> <ul> <li><a id="vanderbilt-external-link-review-custom-1" class="external-link" href="https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/" target="blank">Writing Good Multiple Choice Test Questions - Vanderbilt University Center for Teaching</a></li> <li><a id="conneticut-external-link-review-custom-1" class="external-link" href="https://kb.ecampus.uconn.edu/2020/09/30/writing-effective-multiple-choice-questions-2/" target="blank">Writing Effective Multiple Choice Questions - University of Connecticut Knowledge Base</a></li> </ul> </div> </html> <script> function toggleContent() { const content = document.getElementById('custom-track-4-content-dropdown'); const arrow = document.querySelector('.arrow'); if (content.style.display === 'none') { content.style.display = 'block'; arrow.classList.add('arrow-up'); } else { content.style.display = 'none'; arrow.classList.remove('arrow-up'); } } </script>

<html><h5>Custom assessment <i class="fas fa-caret-right"></i></h5></html> !! Conclusion <html> <h3>Congrats on completing a track! You can go back to different points in this module:</h3> <hr> <div> <a id="start-custom-assessment-track-over-from-custom-track" class="anchor-nav complete-track-2" onclick="customLinkHandler('custom assessment track 1'); handleXAPIClickTrack2(event);">I want to run this track again<i class="fas fa-arrow-right"></i></a> </div> <div> <a id="start-provided-assessment-from-custom-track" class="anchor-nav complete-track-2" onclick="customLinkHandler('provided assessment track 1'); handleXAPIClickTrack2(event);">I want to do the provided assessment track<i class="fas fa-arrow-right"></i></a> </div> <div> <a id="finish-move-to-exit-from-from-custom-track" class="anchor-nav complete-track-2" onclick="customLinkHandler('post survey 1'); handleXAPIClickTrack2(event);">Or, go on to complete this module by taking our survey.<i class="fas fa-arrow-right"></i></a> </div> <p><i>You’ll also get a chance to share more chatGPT experiments with other learners in a designated slack channel.</i></p> </html>

<h4 class="progress-tracker-title">Your progress</h4><div id="progress-tracker"></div>

<html><h5>Introduction <i class="fas fa-caret-right"></i></h5></html> !! Set up ChatGPT <html> <ol start="1"> <li>Go to <a id="openai-external-link" class="external-link" href="https://chat.openai.com/chat" target="blank">chat.OpenAi.com</a> and register for an account with an email address, or a Google or Microsoft account. The paid upgrade is not required for this course.</li> <li>If you haven’t created an account, click on <b>Sign Up</b>. Otherwise, log in with your OpenAI credentials.</li> <li>Once you’ve logged into your ChatGPT account, read through the terms and click on <b>Next</b>. Click on <b>Done</b> when you reach the last one.</li> <li>That’s it. Now you know how to access ChatGPT. If you need more detailed instructions, go to <a id="zdnet-external-link" class="external-link" href="https://www.zdnet.com/article/how-to-use-chatgpt/" target="_blank">ZDNET</a>.</li> </ol> <p><i>Warning! During peak times, you may be unable to create or access ChatGPT. If you’re unable to create an account, you can try later or get a paid GPT 4 version. If you’re unable to access your ChatGPT account, you can refresh the page or sign up to receive an email when it's available again.</i></p> <p>Lastly, you may want to install <a id="shareGPT-external-link" class="external-link" href="https://sharegpt.com/" target="blank">ShareGPT</a>, an open-source Chrome Extension, to share your ChatGPT conversations easily.</p> <hr> <div> <a id="self-assess-early-exit" class="anchor-nav " onclick="customLinkHandler('test drive chatgpt')">Test drive ChatGPT <i class="fas fa-arrow-right"></i></a> </div> </html>

<html><h5>Introduction <i class="fas fa-caret-right"></i></h5></html> !! Test Drive <html> <p>Before we get into assessment best practices, let’s talk about how to tell ChatGPT what you want (called “prompt engineering”).</p> <p>Try entering each of these prompts in sequence just to see how ChatGPT’s responses differ as you provide more specific instruction.</P. <div class="prompt-text-area"> <div class="prompt-text-header"> <h3><span class="light-gray-text">Prompt 1</span></h3> <button class="copyButton" title="Copy to Clipboard"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-clipboard"> <path d="M8 4H6c-1.1 0-2 .9-2 2v12c0 1.1.9 2 2 2h12c1.1 0 2-.9 2-2V10l-4-4h-6zM8 4h6v2H8V4zM8 6v2h6V6H8z"/> </svg> Copy </button> <span class="copiedMessage">Copied!</span> </div> <div class="prompt-text-content"> <pre class="monospaced-text">What should I have for dinner?</pre> </div> </div> <br /> <div class="prompt-text-area"> <div class="prompt-text-header"> <h3><span class="light-gray-text">Prompt 2</span></h3> <button class="copyButton" title="Copy to Clipboard"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-clipboard"> <path d="M8 4H6c-1.1 0-2 .9-2 2v12c0 1.1.9 2 2 2h12c1.1 0 2-.9 2-2V10l-4-4h-6zM8 4h6v2H8V4zM8 6v2h6V6H8z"/> </svg> Copy </button> <span class="copiedMessage">Copied!</span> </div> <div class="prompt-text-content"> <pre class="monospaced-text">What should I have for dinner if I want a unique dining experience in San Francisco, CA?</pre> </div> </div> <br /> <div class="prompt-text-area"> <div class="prompt-text-header"> <h3><span class="light-gray-text">Prompt 3</span></h3> <button class="copyButton" title="Copy to Clipboard"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-clipboard"> <path d="M8 4H6c-1.1 0-2 .9-2 2v12c0 1.1.9 2 2 2h12c1.1 0 2-.9 2-2V10l-4-4h-6zM8 4h6v2H8V4zM8 6v2h6V6H8z"/> </svg> Copy </button> <span class="copiedMessage">Copied!</span> </div> <div class="prompt-text-content"> <pre class="monospaced-text">What should I have for dinner if I want a unique dining experience in San Francisco, CA? I'm vegetarian, and I'm on a budget?</pre> </div> </div> <hr> <div> <a id="self-assess-early-exit" class="anchor-nav " onclick="customLinkHandler('writing assessments')">Write effective distractors <i class="fas fa-arrow-right"></i></a> </div> </html> <script> setTimeout(function() { const copyButtons = document.querySelectorAll(".copyButton"); copyButtons.forEach(button => { button.addEventListener("click", function() { const text = this.parentElement.nextElementSibling.innerText; navigator.clipboard.writeText(text) .then(() => { console.log("Text copied to clipboard"); const copiedMessage = this.nextElementSibling; setTimeout(() => { copiedMessage.classList.remove("show"); }, 1000); this.innerText = "Copied"; this.classList.add("copied"); }) .catch(err => { console.error("Error copying text: ", err); }); }); }); }, 100); </script>

<html><h5>Introduction <i class="fas fa-caret-right"></i></h5></html> !! Write effective distractors <html> <p>Assessment is a huge topic. In this module, we’ll narrow our focus to writing effective distractors for multiple choice questions.</p> <p>Below are some guidelines for ensuring quality distractors. Use the criteria below to evaluate ChatGPT’s assessment questions.</p> <div class="tip-box"> <div class="tip-icon"> <i class="fas fa-pen"></i> </div> <div class="tip-content"> <h3 class="tip-title">Evaluation criteria</h3> <ul> <li>Write options that are congruent in length.</li> <li>Write options that use parallel grammatical structure (e.g. if answer(s) start with a verb, so should the distractors).</li> <li>Avoid “True/False,” “All of the Above,” or “Both A and B Only” (or other such combinations).</li> <li>Use plausible but incorrect distractors.</li> <li>Use only truly correct answers.</li> <li>Avoid negative phrasing.</li> </ul> </div> </div> <p><i>Optional:</i> review assessment writing best practices at these external resources before moving on:</p> <ul> <li><a id="vanderbilt-external-link" class="external-link" href="https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/" target="blank">Writing Good Multiple Choice Test Questions - Vanderbilt University Center for Teaching</a></li> <li><a id="conneticut-external-link" class="external-link" href="https://kb.ecampus.uconn.edu/2020/09/30/writing-effective-multiple-choice-questions-2/" target="blank">Writing Effective Multiple Choice Questions - University of Connecticut Knowledge Base</a></li> </ul> <hr> <div> <a id="self-assess-early-exit" class="anchor-nav " onclick="customLinkHandler('improve an assessment')">Improve an assessment<i class="fas fa-arrow-right"></i></a> </div> </html>

<html><h5>Introduction <i class="fas fa-caret-right"></i></h5></html> !! Improve an assessment with ChatGPT <html> <p>Now you’re ready to start using ChatGPT to generate and improve a multiple choice assessment!</p> <p>We’ll guide you through four rounds of refining an assessment.</p> <p>For this activity, you can use either your own assessment or one we provide. (You can always come back and choose a different track.) We recommend trying the first option if you are new to ChatGPT.</p> <hr> <div> <a id="chose-provided-assessment-track" class="anchor-nav" onclick="customLinkHandler('provided assessment track 1')">Use our provided assessment [track 1]<i class="fas fa-arrow-right"></i></a> </div> <div> <a id="chose-custom-assessment-track" class="anchor-nav" onclick="customLinkHandler('custom assessment track 1')">Use one of your own assessments [track 2]<i class="fas fa-arrow-right"></i></a> </div> </html>

<html><h5>Survey <i class="fas fa-caret-right"></i></h5></html> !! Post-survey (1 of 3) <html> <p>Thanks for participating! Please take our 2-question survey about your experience.</p> <h3>How likely are you to use ChatGPT to develop multiple choice assessments as a result of this activity?</h3> <form id="survey-1" class="pure-form" data-parent="post-survey"> <label for="survey-1-opt-1" class="pure-checkbox"> <input type="radio" name="survey-1-choice" id="survey-1-opt-1" value="more-likely-to-user-chatgpt" onclick="checkOptionSelected('survey-1')" />More Likely </label> <label for="survey-1-opt-2" class="pure-checkbox"> <input type="radio" name="survey-1-choice" id="survey-1-opt-2" value="equally-likely-to-user-chatgpt" onclick="checkOptionSelected('survey-1')" />Equally Likely </label> <label for="survey-1-opt-3" class="pure-checkbox"> <input type="radio" name="survey-1-choice" id="survey-1-opt-3" value="less-likely-to-user-chatgpt" onclick="checkOptionSelected('survey-1')" />Less Likely </label> <button type="button" onclick="sendFormXAPIStatementNoVoid('survey-1'); checkOptionSelected('survey-1'); customButtonHandler('post survey 2')" id="survey-1-submit" class="pure-button pure-button-primary" disabled>Submit your choice</button> </form> </html>

<html><h5>Survey <i class="fas fa-caret-right"></i></h5></html> !! Post-survey (2 of 3) <html> <h3>As a result of this activity, I’m confident in ChatGPT’s ability to compose effective assessments with prompt engineering.</h3> <form id="survey-2" class="pure-form" data-parent="post-survey"> <label for="survey-2-opt-1" class="pure-checkbox"> <input type="radio" name="survey-2-choice" id="survey-2-opt-1" value="much-more-cofident-in-chat-gpt" onclick="checkOptionSelected('survey-2')" />Much more confident </label> <label for="survey-2-opt-2" class="pure-checkbox"> <input type="radio" name="survey-2-choice" id="survey-2-opt-2" value="slightly-more-cofident-in-chat-gpt" onclick="checkOptionSelected('survey-2')" />Slightly more confident </label> <label for="survey-2-opt-3" class="pure-checkbox"> <input type="radio" name="survey-2-choice" id="survey-2-opt-3" value="equally-more-cofident-in-chat-gpt" onclick="checkOptionSelected('survey-2')" />Equally confident </label> <label for="survey-2-opt-4" class="pure-checkbox"> <input type="radio" name="survey-2-choice" id="survey-2-opt-4" value="slightly-less-cofident-in-chat-gpt" onclick="checkOptionSelected('survey-2')" />Slightly less confident </label> <label for="survey-2-opt-5" class="pure-checkbox"> <input type="radio" name="survey-2-choice" id="survey-2-opt-5" value="much-less-cofident-in-chat-gpt" onclick="checkOptionSelected('survey-2')" />Much less confident </label> <button type="button" onclick="sendFormXAPIStatementNoVoid('survey-2'); checkOptionSelected('survey-2'); customButtonHandler('post survey 3 freeform')" id="survey-2-submit" class="pure-button pure-button-primary" disabled>Submit your choice</button> </form> </html>

<html><h5>Survey <i class="fas fa-caret-right"></i></h5></html> !! Post-survey (3 of 3) <html> <h3>We’ve kept you anonymous so far. This is a chance to leave us feedback about the course… and a way to get in touch!</h3> <p>If you leave your email, we may follow up with you for clarifications.</p> <div> <a id="chose-to-fill-out-typeform" class="anchor-nav anchor-nav-external" href="https://yljg97xdvs4.typeform.com/to/ZPXZEfoC" target="blank">Contact us on Typeform <i class="fas fa-arrow-right"></i></a> </div> <div> <a id="chose-to skip-typeform" class="anchor-nav" onclick="customLinkHandler('thank you')">Finish the course <i class="fas fa-arrow-right"></i></a> </div> </html>

<html><h5>Refining a provided prompt <i class="fas fa-caret-right"></i></h5></html> !! Part 1 <html> <h3>Copy the prompt below that asks ChatGPT to refine the multiple choice question.</h3> <div class="prompt-text-area"> <div class="prompt-text-header"> <h3><span class="light-gray-text">Prompt Text</span></h3> <button class="copyButton" title="Copy to Clipboard"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-clipboard"> <path d="M8 4H6c-1.1 0-2 .9-2 2v12c0 1.1.9 2 2 2h12c1.1 0 2-.9 2-2V10l-4-4h-6zM8 4h6v2H8V4zM8 6v2h6V6H8z"/> </svg> Copy </button> <span class="copiedMessage">Copied!</span> </div> <div class="prompt-text-content"> <pre class="monospaced-text">You are a teacher of cooking class. You teach middle school students the basics of healthy and safe cooking. The age of your students is 12-16. Your task: Using the information above, generate a multiple choice assessment which helps the learners described above to correctly understand core foundational knowledge relating to cooking healthy meals.</pre> </div> </div><br /> <h3>Evaluate it based on the criteria for effective distractors (review criteria below if needed). Is the assessment unacceptable/adequate/excellent in your opinion?</h3> <hr> <h3>Which of the following best describes your experience using ChatGPT?</h3> <form id="provided-track-1" class="pure-form" data-parent="provided-example-track"> <label for="provided-track-1-opt-1" class="pure-checkbox"> <input type="radio" name="provided-track-1-choice" id="provided-track-1-opt-1" value="unacceptable-using-chatGPT-with-first-copy-of-prompt" onclick="checkOptionSelected('provided-track-1')" />Unacceptable </label> <label for="provided-track-1-opt-2" class="pure-checkbox"> <input type="radio" name="provided-track-1-choice" id="provided-track-1-opt-2" value="adequate-experience-using-chatGPT-with-first-copy-of-prompt" onclick="checkOptionSelected('provided-track-1')" />Adequate </label> <label for="provided-track-1-opt-3" class="pure-checkbox"> <input type="radio" name="provided-track-1-choice" id="provided-track-1-opt-3" value="excellent-experience-using-chatGPT-with-first-copy-of-prompt" onclick="checkOptionSelected('provided-track-1')" />Excellent </label> <button type="button" onclick="sendFormXAPIStatementNoVoid('provided-track-1'); checkOptionSelected('provided-track-1'); customButtonHandler('provided assessment track 1 freeform')" id="provided-track-1-submit" class="pure-button pure-button-primary" disabled>Submit your choice</button> </form> <br /> <div class="dropdown-container"> <div class="dropdown-header" onclick="toggleContent()"> <h3><span class="light-gray-text">Review the criteria</span></h3> <span class="arrow">▾</span> </div> <div id="provided-track-1-content-dropdown" class="dropdown-text-content" style="display: none;"> <p>Use the criteria below to evaluate ChatGPT’s assessment questions.</p> <div class="tip-box"> <div class="tip-icon"> <i class="fas fa-pen"></i> </div> <div class="tip-content"> <h3 class="tip-title">Evaluation criteria</h3> <ul> <li>Write options that are congruent in length.</li> <li>Write options that use parallel grammatical structure (e.g. if answer(s) start with a verb, so should the distractors).</li> <li>Avoid “True/False,” “All of the Above,” or “Both A and B Only” (or other such combinations).</li> <li>Use plausible but incorrect distractors.</li> <li>Use only truly correct answers.</li> <li>Avoid negative phrasing.</li> </ul> </div> </div> <p>Review assessment writing best practices at these external resources before moving on:</p> <ul> <li><a id="vanderbilt-external-link-review-1" class="external-link" href="https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/" target="blank">Writing Good Multiple Choice Test Questions - Vanderbilt University Center for Teaching</a></li> <li><a id="conneticut-external-link-review-1" class="external-link" href="https://kb.ecampus.uconn.edu/2020/09/30/writing-effective-multiple-choice-questions-2/" target="blank">Writing Effective Multiple Choice Questions - University of Connecticut Knowledge Base</a></li> </ul> </div> </div> </html> <script> setTimeout(function() { const copyButtons = document.querySelectorAll(".copyButton"); copyButtons.forEach(button => { button.addEventListener("click", function() { const text = this.parentElement.nextElementSibling.innerText; navigator.clipboard.writeText(text) .then(() => { console.log("Text copied to clipboard"); const copiedMessage = this.nextElementSibling; setTimeout(() => { copiedMessage.classList.remove("show"); }, 1000); this.innerText = "Copied"; this.classList.add("copied"); }) .catch(err => { console.error("Error copying text: ", err); }); }); }); }, 100); </script> <script> function toggleContent() { const content = document.getElementById('provided-track-1-content-dropdown'); const arrow = document.querySelector('.arrow'); if (content.style.display === 'none') { content.style.display = 'block'; arrow.classList.add('arrow-up'); } else { content.style.display = 'none'; arrow.classList.remove('arrow-up'); } } </script> <<run setup.trackStarts++>> <<run setup.currentIterationId = "provided_assessment_track_loop_" + (setup.trackStarts)>> <<run setup.iterationActivities[setup.currentIterationId] = []>>

<html><h5>Refining a provided prompt <i class="fas fa-caret-right"></i></h5></html> !! Reflect on part 1 <html> <h3>Share any other thoughts you have on the quality of the answer.</p> <div class="textarea-Wrapper" data-parent="provided-example-track"> <textarea class="custom-textarea" id="provided-track-1-freeform" type="text" value="" oninput="checkTextareaNotEmpty('provided-track-1-freeform', 'send-provided-track-1-freeform', '/track-1-textarea')"></textarea> </div> <button class="pure-button pure-button-primary" id="send-provided-track-1-freeform" onclick="sendTextareaXAPIStatement('provided-track-1-freeform', 'send-provided-track-1-freeform', 'provided-track-1-textarea'); customButtonHandler('criteria for effective distractors')" disabled>Send your thoughts</button> <hr> <div> <a id="skip-provided-track-1-freeform" class="anchor-nav" onclick="customLinkHandler('criteria for effective distractors')">Or, skip <i class="fas fa-arrow-right"></i></a> </div> </html>

<html><h5>Refining a provided prompt <i class="fas fa-caret-right"></i></h5></html> !! Criteria for effective distractors <html> <h3>Use the criteria below to evaluate ChatGPT’s assessment questions.</h3> <ul> <li>Write options that are congruent in length.</li> <li>Write options that use parallel grammatical structure (e.g. if answer(s) start with a verb, so should the distractors).</li> <li>Avoid “True/False,” “All of the Above,” or “Both A and B Only” (or other such combinations).</li> <li>Use plausible but incorrect distractors.</li> <li>Use only truly correct answers.</li> <li>Avoid negative phrasing.</li> </ul> <div class="tip-box"> <div class="tip-icon"> <i class="fas fa-pen"></i> </div> <div class="tip-content"> <h3 class="tip-title">Optional</h3> <p>Review assessment writing best practices at these external resources before moving on:</p> <ul> <li><a id="vanderbilt-external-link-2" class="external-link" href="https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/" target="blank">Writing Good Multiple Choice Test Questions - Vanderbilt University Center for Teaching</a></li> <li><a id="conneticut-external-link-2" class="external-link" href="https://kb.ecampus.uconn.edu/2020/09/30/writing-effective-multiple-choice-questions-2/" target="blank">Writing Effective Multiple Choice Questions - University of Connecticut Knowledge Base</a></li> </ul> </div> </div> <div> <hr> <a id="provided-criteria" class="anchor-nav" onclick="customLinkHandler('provided assessment track 2')">On to Part 2 <i class="fas fa-arrow-right"></i></a> </div> </html>

<html><h5>Refining a provided prompt <i class="fas fa-caret-right"></i></h5></html> !! Refine part 2 <html> <p>Now write a follow up prompt refining what you’re asking ChatGPT to do. Mention a criterion for effective distractors or other best practices as needed to help ChatGPT produce better content.</p> <p>Evaluate it based on the criteria for effective distractors (review criteria below if needed). Is the new version is worse/equal/better than the original?</p> <form id="provided-track-2" class="pure-form" data-parent="provided-example-track"> <label for="provided-track-2-opt-1" class="pure-checkbox"> <input type="radio" name="provided-track-2-choice" id="provided-track-2-opt-1" value="worse-followup-providing-criterion" onclick="checkOptionSelected('provided-track-2')" />Worse </label> <label for="provided-track-2-opt-2" class="pure-checkbox"> <input type="radio" name="provided-track-2-choice" id="provided-track-2-opt-2" value="equal-followup-providing-criterion" onclick="checkOptionSelected('provided-track-2')" />Equal </label> <label for="provided-track-2-opt-3" class="pure-checkbox"> <input type="radio" name="provided-track-2-choice" id="provided-track-2-opt-3" value="better-followup-providing-criterion" onclick="checkOptionSelected('provided-track-2')" />Better </label> <button type="button" onclick="sendFormXAPIStatement('provided-track-2'); checkOptionSelected('provided-track-2'); customButtonHandler('provided assessment track 2 freeform')" id="provided-track-2-submit" class="pure-button pure-button-primary" disabled>Submit your choice</button> </form><br > <div class="dropdown-container"> <div class="dropdown-header" onclick="toggleContent()"> <h3><span class="light-gray-text">Review the criteria</span></h3> <span class="arrow">▾</span> </div> <div id="provided-track-2-a-content-dropdown" class="dropdown-text-content" style="display: none;"> <p>Use the criteria below to evaluate ChatGPT’s assessment questions.</p> <div class="tip-box"> <div class="tip-icon"> <i class="fas fa-pen"></i> </div> <div class="tip-content"> <h3 class="tip-title">Evaluation criteria</h3> <ul> <li>Write options that are congruent in length.</li> <li>Write options that use parallel grammatical structure (e.g. if answer(s) start with a verb, so should the distractors).</li> <li>Avoid “True/False,” “All of the Above,” or “Both A and B Only” (or other such combinations).</li> <li>Use plausible but incorrect distractors.</li> <li>Use only truly correct answers.</li> <li>Avoid negative phrasing.</li> </ul> </div> </div> <p>Review assessment writing best practices at these external resources before moving on:</p> <ul> <li><a id="vanderbilt-external-link-review-1" class="external-link" href="https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/" target="blank">Writing Good Multiple Choice Test Questions - Vanderbilt University Center for Teaching</a></li> <li><a id="conneticut-external-link-review-1" class="external-link" href="https://kb.ecampus.uconn.edu/2020/09/30/writing-effective-multiple-choice-questions-2/" target="blank">Writing Effective Multiple Choice Questions - University of Connecticut Knowledge Base</a></li> </ul> </div> </div> </html> <script> setTimeout(function() { const copyButtons = document.querySelectorAll(".copyButton"); copyButtons.forEach(button => { button.addEventListener("click", function() { const text = this.parentElement.nextElementSibling.innerText; navigator.clipboard.writeText(text) .then(() => { console.log("Text copied to clipboard"); const copiedMessage = this.nextElementSibling; setTimeout(() => { copiedMessage.classList.remove("show"); }, 1000); this.innerText = "Copied"; this.classList.add("copied"); }) .catch(err => { console.error("Error copying text: ", err); }); }); }); }, 100); </script> <script> function toggleContent() { const content = document.getElementById('provided-track-2-a-content-dropdown'); const arrow = document.querySelector('.arrow'); if (content.style.display === 'none') { content.style.display = 'block'; arrow.classList.add('arrow-up'); } else { content.style.display = 'none'; arrow.classList.remove('arrow-up'); } } </script>

<html><h5>Refining a provided prompt <i class="fas fa-caret-right"></i></h5></html> !! Reflect on part 2 <html> <h3>Why do you think the new version changed the way it did or mostly stayed the same?</p> <div class="textarea-Wrapper" data-parent="provided-example-track"> <textarea class="custom-textarea" id="provided-track-2-freeform" type="text" value="" oninput="checkTextareaNotEmpty('provided-track-2-freeform', 'send-provided-track-2-freeform', '/track-1-textarea')"></textarea> </div> <button class="pure-button pure-button-primary" id="send-provided-track-2-freeform" onclick="sendTextareaXAPIStatement('provided-track-2-freeform', 'send-provided-track-2-freeform', 'provided-track-2-textarea'); customButtonHandler('provided assessment track 3')" disabled>Send your thoughts</button> <hr> <div> <a id="skip-provided-track-2-freeform" class="anchor-nav" onclick="customLinkHandler('provided assessment track 3')">Or, skip <i class="fas fa-arrow-right"></i></a> </div> </html>

<html><h5>Refining a provided prompt <i class="fas fa-caret-right"></i></h5></html> !! Refine part 3 <html> <p>Now test a prompt that applies prompt engineering techniques explained in the article <a id="mega-prompts-external-link-procided-3" class="external-link" href="https://drphilippahardman.substack.com/p/introducing-chatgpt-edu-mega-prompts?utm_source=profile&utm_medium=reader2" target="blank">“Introducing: ChatGPT Edu-Mega-Prompts”</a> by Dr Philippa Hardman.</p> <p>The prompt specifies AI’s role, describes the subject the and learner, identifies the instructional strategy, provides context and an example. It clearly outline the task and the desired outputs.</p> <p>Copy the entire prompt and paste it to ChatGPT to see what assessment it will produce.</p> <div class="prompt-text-area"> <div class="prompt-text-header"> <h3><span class="light-gray-text">Prompt Text</span></h3> <button class="copyButton" title="Copy to Clipboard"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="feather feather-clipboard"> <path d="M8 4H6c-1.1 0-2 .9-2 2v12c0 1.1.9 2 2 2h12c1.1 0 2-.9 2-2V10l-4-4h-6zM8 4h6v2H8V4zM8 6v2h6V6H8z"/> </svg> Copy </button> <span class="copiedMessage">Copied!</span> </div> <div class="prompt-text-content"> <pre class="monospaced-text">You are a teacher of cooking class . You teach middle school students the basics of healthy and safe cooking. The age of your students is 12-16. Without the ability to recall & process accurate foundational information, the learning process is futile. You want to support your learners to recall and process accurate foundational information without using quizzes which are proven to only enable short term recall rather than long term memorisation. Instead, you want to use two instructional strategies proven to increased the effective memorisation of foundational knowledge. The first strategy you will use is known as Undoing. Topic: Undoing Instructional Strategy Research shows that learners learn better & more quickly when the learning experience intentionally challenges commonly mistaken foundational ideas and assumptions and encourages new ways of thinking. The highest impact learning experiences are designed to identify and correct foundational misconceptions before memorisation occurs. Undoing activities are particularly helpful when used at the start of the learning process by identifying and then correcting gaps in knowledge, misconceptions and misplaced reasoning. Undoing activities often ask learners to predict the outcome of a scenario. An example of a well formed undoing activity goes as follows: Sharna is watching a candle burn. Once it’s burned down, she wonders what happened to the wax. She has 4 ideas - which do you agree with most, and why? A. XXXX - common misconception B. XXXX - a common misconception C. XXXX - the correct answer D. XXXX - a common misconception Your task: Using the information above, generate an undoing activity which helps the learners described above to correctly understand core foundational knowledge relating to cooking healthy meals. Write feedback for each potential response. The feedback should be succinct and explain the difference between the student's response and the correct answer. It should replace the misconception with an alternative conception, explaining what the alternative, correct way of thinking about the scenario is and how it differs from the student's misconception. Finally, suggest a resource for the defined learner type to look at which will help them to learn the correct version of the foundational knowledge.</pre> </div> </div> <p>Evaluate it based on the criteria for effective distractors (review criteria below if needed).</p> <h3>Is the new version is worse/equal/better than the original? Which of the following best describes your experience using ChatGPT?</h3> <form id="provided-track-3" class="pure-form" data-parent="provided-example-track"> <label for="provided-track-3-opt-1" class="pure-checkbox"> <input type="radio" name="provided-track-3-choice" id="provided-track-3-opt-1" value="worse-experience-with-mega-prompts" onclick="checkOptionSelected('provided-track-3')" />Worse </label> <label for="provided-track-3-opt-2" class="pure-checkbox"> <input type="radio" name="provided-track-3-choice" id="provided-track-3-opt-2" value="equal-experience-with-mega-prompts" onclick="checkOptionSelected('provided-track-3')" />Equal </label> <label for="provided-track-3-opt-3" class="pure-checkbox"> <input type="radio" name="provided-track-3-choice" id="provided-track-3-opt-3" value="better-experience-with-mega-prompts" onclick="checkOptionSelected('provided-track-3')" />Better </label> <button type="button" onclick="sendFormXAPIStatement('provided-track-3'); checkOptionSelected('provided-track-3'); customButtonHandler('provided assessment track 3 freeform')" id="provided-track-3-submit" class="pure-button pure-button-primary" disabled>Submit your choice</button> </form><br > <div class="dropdown-container"> <div class="dropdown-header" onclick="toggleContent()"> <h3><span class="light-gray-text">Review the criteria</span></h3> <span class="arrow">▾</span> </div> <div id="provided-track-3-a-content-dropdown" class="dropdown-text-content" style="display: none;"> <p>Use the criteria below to evaluate ChatGPT’s assessment questions.</p> <div class="tip-box"> <div class="tip-icon"> <i class="fas fa-pen"></i> </div> <div class="tip-content"> <h3 class="tip-title">Evaluation criteria</h3> <ul> <li>Write options that are congruent in length.</li> <li>Write options that use parallel grammatical structure (e.g. if answer(s) start with a verb, so should the distractors).</li> <li>Avoid “True/False,” “All of the Above,” or “Both A and B Only” (or other such combinations).</li> <li>Use plausible but incorrect distractors.</li> <li>Use only truly correct answers.</li> <li>Avoid negative phrasing.</li> </ul> </div> </div> <p>Review assessment writing best practices at these external resources before moving on:</p> <ul> <li><a id="vanderbilt-external-link-review-1" class="external-link" href="https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/" target="blank">Writing Good Multiple Choice Test Questions - Vanderbilt University Center for Teaching</a></li> <li><a id="conneticut-external-link-review-1" class="external-link" href="https://kb.ecampus.uconn.edu/2020/09/30/writing-effective-multiple-choice-questions-2/" target="blank">Writing Effective Multiple Choice Questions - University of Connecticut Knowledge Base</a></li> </ul> </div> </div> </html> <script> setTimeout(function() { const copyButtons = document.querySelectorAll(".copyButton"); copyButtons.forEach(button => { button.addEventListener("click", function() { const text = this.parentElement.nextElementSibling.innerText; navigator.clipboard.writeText(text) .then(() => { console.log("Text copied to clipboard"); const copiedMessage = this.nextElementSibling; setTimeout(() => { copiedMessage.classList.remove("show"); }, 1000); this.innerText = "Copied"; this.classList.add("copied"); }) .catch(err => { console.error("Error copying text: ", err); }); }); }); }, 100); </script> <script> function toggleContent() { const content = document.getElementById('provided-track-3-a-content-dropdown'); const arrow = document.querySelector('.arrow'); if (content.style.display === 'none') { content.style.display = 'block'; arrow.classList.add('arrow-up'); } else { content.style.display = 'none'; arrow.classList.remove('arrow-up'); } } </script>

<html><h5>Refining a provided prompt <i class="fas fa-caret-right"></i></h5></html> !! Reflect on part 3 <html> <h3>Why do you think the new version changed the way it did or mostly stayed the same?</h3> <div class="textarea-Wrapper" data-parent="provided-example-track"> <textarea class="custom-textarea" id="provided-track-3-freeform" type="text" value="" oninput="checkTextareaNotEmpty('provided-track-3-freeform', 'send-provided-track-3-freeform', '/track-1-textarea')"></textarea> </div> <button class="pure-button pure-button-primary" id="send-provided-track-3-freeform" onclick="sendTextareaXAPIStatement('provided-track-3-freeform', 'send-provided-track-3-freeform', 'provided-track-3-textarea'); customButtonHandler('provided assessment track 4')" disabled>Send your thoughts</button> <hr> <div> <a id="skip-provided-track-3-freeform" class="anchor-nav" onclick="customLinkHandler('provided assessment track 4')">Or, skip <i class="fas fa-arrow-right"></i></a> </div> </html>

<html><h5>Refining a provided prompt <i class="fas fa-caret-right"></i></h5></html> !! Refine Part 4 <html> <p>Now, <b><i>without using ChatGPT</i></b>, take the best elements from the different iterations of the multiple choice question, combining them into the best possible assessment question. Avoid adding new material other than correcting for grammar and fluidity.</p> <p>Evaluate this human-edited version based on the criteria for effective distractors (review effective criteria below if needed).</p> <h3>Is the new version worse/equal/better than the original? </h3> <form id="provided-track-4" class="pure-form" data-parent="provided-example-track"> <label for="provided-track-4-opt-1" class="pure-checkbox"> <input type="radio" name="provided-track-4-choice" id="provided-track-4-opt-1" value="worse-prompt-after-human-edited" onclick="checkOptionSelected('provided-track-4')" />Worse </label> <label for="provided-track-4-opt-2" class="pure-checkbox"> <input type="radio" name="provided-track-4-choice" id="provided-track-4-opt-2" value="equal-prompt-after-human-edited" onclick="checkOptionSelected('provided-track-4')" />Equal </label> <label for="provided-track-4-opt-3" class="pure-checkbox"> <input type="radio" name="provided-track-4-choice" id="provided-track-4-opt-3" value="better-prompt-after-human-edited" onclick="checkOptionSelected('provided-track-4')" />Better </label> <button type="button" onclick="sendFormXAPIStatement('provided-track-4'); checkOptionSelected('provided-track-4'); customButtonHandler('provided assessment track 4 freeform')" id="provided-track-4-submit" class="pure-button pure-button-primary" disabled>Submit your choice</button> </form><br /> <div class="dropdown-container"> <div class="dropdown-header" onclick="toggleContent()"> <h3><span class="light-gray-text">Review the criteria</span></h3> <span class="arrow">▾</span> </div> <div id="provided-track-4-a-content-dropdown" class="dropdown-text-content" style="display: none;"> <p>Use the criteria below to evaluate ChatGPT’s assessment questions.</p> <div class="tip-box"> <div class="tip-icon"> <i class="fas fa-pen"></i> </div> <div class="tip-content"> <h3 class="tip-title">Evaluation criteria</h3> <ul> <li>Write options that are congruent in length.</li> <li>Write options that use parallel grammatical structure (e.g. if answer(s) start with a verb, so should the distractors).</li> <li>Avoid “True/False,” “All of the Above,” or “Both A and B Only” (or other such combinations).</li> <li>Use plausible but incorrect distractors.</li> <li>Use only truly correct answers.</li> <li>Avoid negative phrasing.</li> </ul> </div> </div> <p>Review assessment writing best practices at these external resources before moving on:</p> <ul> <li><a id="vanderbilt-external-link-review-1" class="external-link" href="https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/" target="blank">Writing Good Multiple Choice Test Questions - Vanderbilt University Center for Teaching</a></li> <li><a id="conneticut-external-link-review-1" class="external-link" href="https://kb.ecampus.uconn.edu/2020/09/30/writing-effective-multiple-choice-questions-2/" target="blank">Writing Effective Multiple Choice Questions - University of Connecticut Knowledge Base</a></li> </ul> </div> </div> </html> <script> function toggleContent() { const content = document.getElementById('provided-track-4-a-content-dropdown'); const arrow = document.querySelector('.arrow'); if (content.style.display === 'none') { content.style.display = 'block'; arrow.classList.add('arrow-up'); } else { content.style.display = 'none'; arrow.classList.remove('arrow-up'); } } </script>

<html><h5>Refining a provided prompt <i class="fas fa-caret-right"></i></h5></html> !! Reflect on part 4 <html> <h3>Why do you think the new version changed the way it did or mostly stayed the same?</h3> <div class="textarea-Wrapper" data-parent="provided-example-track"> <textarea class="custom-textarea" id="provided-track-4-freeform" type="text" value="" oninput="checkTextareaNotEmpty('provided-track-4-freeform', 'send-provided-track-4-freeform', '/track-1-textarea')"></textarea> </div> <button class="pure-button pure-button-primary" id="send-provided-track-4-freeform" onclick="sendTextareaXAPIStatement('provided-track-4-freeform', 'send-provided-track-4-freeform', 'provided-track-4-textarea'); customButtonHandler('provided assessment track exit')" disabled>Send your thoughts</button> <hr> <div> <a id="skip-provided-track-4-freeform" class="anchor-nav" onclick="customLinkHandler('provided assessment track exit')">Or, skip <i class="fas fa-arrow-right"></i></a> </div> </html>

<html><h5>Refining a provided prompt <i class="fas fa-caret-right"></i></h5></html> !! You’ve completed this track <html> <div> <a id="start-provided-assessment-over-from-provided-track" class="anchor-nav complete-track-1" onclick="customLinkHandler('provided assessment track 1'); handleXAPIClick(event);">I want to do the track again for practice<i class="fas fa-arrow-right"></i></a> </div> <div> <a id="start-custom-assessment-track-from-provided-track" class="anchor-nav complete-track-1" onclick="customLinkHandler('custom assessment track 1'); handleXAPIClick(event);">I want to try writing my own assessments<i class="fas fa-arrow-right"></i></a> </div> <div> <a id="finish-move-to-survey-from-assessment-track-from-provided-track" class="anchor-nav complete-track-1" onclick="customLinkHandler('post survey 1'); handleXAPIClick(event);">Or, go on to complete this module by taking our survey<i class="fas fa-arrow-right"></i></a> </div> </html>

<html><h5>Self assessment <i class="fas fa-caret-right"></i></h5></html> !! Survey questions (1 of 4) <html> <p>Please help us get to know you by answering the following three survey questions. Your responses will only be shared in aggregate (combined with others&rsquor; responses).</p> <h3>Which of the following best describes your experience with developing multiple choice assessments?</h3> <form id="survey-1" class="pure-form" data-parent="self-assessment-track"> <label data-lpignore="true" for="survey-1-opt-1" class="pure-checkbox"> <input type="radio" name="survey-1-choice" id="survey-1-opt-1" value="no-experience-with-assessments" onclick="checkOptionSelected('survey-1')" />No experience: I have never written a multiple choice assessment. </label> <label data-lpignore="true" for="survey-1-opt-2" class="pure-checkbox"> <input type="radio" name="survey-1-choice" id="survey-1-opt-2" value="beginner-experience-with-assessments" onclick="checkOptionSelected('survey-1')" />Beginner: I have performed this task occasionally. </label> <label data-lpignore="true" for="survey-1-opt-3" class="pure-checkbox"> <input type="radio" name="survey-1-choice" id="survey-1-opt-3" value="intermediate-experience-with-assessments" onclick="checkOptionSelected('survey-1')" />Intermediate: This task has been a central part of my work. I have performed it myself routinely without any supervision. </label> <label data-lpignore="true" for="survey-1-opt-4" class="pure-checkbox" data-parent="self-assessment-track"> <input type="radio" name="survey-1-choice" id="survey-1-opt-4" value="advanced-experience-with-assessments" onclick="checkOptionSelected('survey-1')" />Advanced: I have trained others in performance of this task and/or others have consulted me as an expert for assistance in performing this task. </label> <button type="button" onclick="sendFormXAPIStatement('survey-1'); checkOptionSelected('survey-1'); handleButtonSurvey1to2(); customButtonHandler('self assessment 2')" id="survey-1-submit" class="pure-button pure-button-primary" disabled>Submit your choice</button> </form> </html>

<html><h5>Self assessment <i class="fas fa-caret-right"></i></h5></html> !! Survey questions (2 of 4) <html> <h3>Which of the following best describes your experience using ChatGPT?</h3> <form id="survey-2" class="pure-form" data-parent="self-assessment-track"> <label for="survey-2-opt-1" class="pure-checkbox"> <input type="radio" name="survey-2-choice" id="survey-2-opt-1" value="no-experience-with-chatgpt" onclick="checkOptionSelected('survey-2')" />No experience: I have never used ChatGPT. </label> <label for="survey-2-opt-2" class="pure-checkbox"> <input type="radio" name="survey-2-choice" id="survey-2-opt-2" value="beginner-experience-with-chatgpt" onclick="checkOptionSelected('survey-2')" />Beginner: I have used chatGPT occasionally. </label> <label for="survey-2-opt-3" class="pure-checkbox"> <input type="radio" name="survey-2-choice" id="survey-2-opt-3" value="intermediate-experience-with-chatgpt" onclick="checkOptionSelected('survey-2')" />Intermediate: I have used ChatGPT to create output that I used in my job. </label> <label for="survey-2-opt-4" class="pure-checkbox"> <input type="radio" name="survey-2-choice" id="survey-2-opt-4" value="advanced-experience-with-chatgpt" onclick="checkOptionSelected('survey-2')" />Advanced: I use ChatGPT almost daily in my job. </label> <button type="button" onclick="sendFormXAPIStatement('survey-2'); checkOptionSelected('survey-2'); handleButtonSurvey2to3(); customButtonHandler('self assessment 3')" id="survey-2-submit" class="pure-button pure-button-primary" disabled>Submit your choice</button> </form> </html>

<html><h5>Self assessment <i class="fas fa-caret-right"></i></h5></html> !! Survey questions (3 of 4) <html> <h3>Overall, how do you feel about the increased use of artificial intelligence in computer programs in daily life?</h3> <form id="survey-3" class="pure-form" data-parent="self-assessment-track"> <label for="survey-3-opt-1" class="pure-checkbox" data-parent="self-assessment-track"> <input type="radio" name="survey-3-choice" id="survey-3-opt-1" value="more-concerned-than-excited-about-ai" onclick="checkOptionSelected('survey-3')" />More concerned than excited </label> <label for="survey-3-opt-2" class="pure-checkbox"> <input type="radio" name="survey-3-choice" id="survey-3-opt-2" value="equally-concerned-and-excited-about-ai" onclick="checkOptionSelected('survey-3')" />Equally concerned and excited </label> <label for="survey-3-opt-3" class="pure-checkbox"> <input type="radio" name="survey-3-choice" id="survey-3-opt-3" value="more-excited-than-concerned-about-ai" onclick="checkOptionSelected('survey-3')" />More excited than concerned </label> <button type="button" onclick="sendFormXAPIStatement('survey-3'); checkOptionSelected('survey-3'); handleButtonSurvey3toText(); customButtonHandler('self assessment general feedback')" id="survey-3-submit" class="pure-button pure-button-primary" disabled>Submit your choice</button> </form> </html>

<html><h5>Self assessment <i class="fas fa-caret-right"></i></h5></html> !! Open Feedback <html> <h3>Let us know what you think about ChatGPT, in your own words.</h3> <div class="textarea-Wrapper" data-parent="self-assessment-track"> <textarea class="custom-textarea" id="self-assessment-general-feedback" type="text" value="" data-activity-id="self-assessment-general-feedback-activity" oninput="checkTextareaNotEmpty('self-assessment-general-feedback', 'send-self-assessment-general-feedback')"></textarea> </div> <button class="pure-button pure-button-primary" id="send-self-assessment-general-feedback" onclick="sendTextareaXAPIStatement('self-assessment-general-feedback', 'send-self-assessment-general-feedback', 'self-assessment-general-feedback-activity'); customButtonHandler('complete assessment')" disabled>Send your thoughts</button> <hr> <div> <a id="skip-self-assessment-general-feedback" class="anchor-nav" onclick="customLinkHandler('complete assessment')">Or, skip <i class="fas fa-arrow-right"></i></a> </div> </html>

<html><h5>Self assessment <i class="fas fa-caret-right"></i></h5></html> !! Set up and test ChatGPT <html> <p>Before you start using ChatGPT to refine assessments, you need to get access and give it a test drive.</p> <p>If you’ve already got access, give it a test drive (guided prompt engineering).</p> <hr> <div> <a id="set-up-chat-gpt" class="anchor-nav " onclick="customLinkHandler('set up chatgpt')">Set up ChatGPT<i class="fas fa-arrow-right"></i></a> </div> <div> <a id="test-drive-chat-gpt-skip-setup" class="anchor-nav " onclick="customLinkHandler('test drive chatgpt')">Test drive ChatGPT<i class="fas fa-arrow-right"></i></a> </div> </html>

!! Welcome! <html> <p>Welcome to this course on using ChatGPT to generate and improve multiple choice assessments. </p> <p>This is geared for instructional designers who already have basic skills in assessment development. (Don’t worry if you don’t have experience with writing assessments – you’ll still be able to learn about it.)</p> <p>There is an xAPI component here too, of course. Throughout the experience, we invite you to open up your browser’s console to see what statements are being sent to the LRS. Keep in mind this is a prototype; things may not work perfectly all the time!</p> <p>In this course, you’ll get the opportunity to practice:<p> <ul> <li>Using ChatGPT as a tool to generate multiple choice assessments</li> <li>Applying prompt engineering techniques to iterate on developing multiple choice assessments</li> <li>Evaluating the quality of distractors in multiple choice assessments generated by AI</li> </ul> <button onclick="handleButtonStartSurvey1(); customButtonHandler('self assessment 1')" id="buttonStartSurvey1" class="pure-button pure-button-primary">Start your self assessment</button> </html>

!! Thank you <html> <p>Thanks for joining us in exploring the limits and potential of using ChatGPT to develop multiple choice questions and improve distractors. We hope you enjoyed the chance to:</p> <ul> <li>Review criteria for writing effective distractors in multiple choice questions</li> <li>Use ChatGPT as a tool to generate multiple choice assessments</li> <li>Apply prompt engineering techniques to iterate on multiple choice assessments</li> <li>Evaluate the quality of distractors in multiple choice assessments generated by AI</li> </ul> <h3>What’s next?</h3> <p>Share about your experience with others in the Slack channel <a id="slack-channel-link" class="external-link" href="https://xapi-cohort.slack.com/archives/C0520V1AE22" target="blank">#assessments-with-chatgpt</a>. Get <a id="shareGPT-external-link-from-thank-you" class="external-link" href="https://sharegpt.com/" target="blank">ShareGPT</a> to easily share your conversations. <hr> <div> <a id="start-provided-assessment-track-from-thankyou" class="anchor-nav complete-track-1" onclick="customLinkHandler('custom assessment track 1'); handleXAPIClick(event);">I want to try working with the provided assessment agian<i class="fas fa-arrow-right"></i></a> </div> <div> <a id="start-custom-assessment-track-from-thankyou" class="anchor-nav complete-track-1" onclick="customLinkHandler('custom assessment track 1'); handleXAPIClick(event);">I want to try writing my own assessments again<i class="fas fa-arrow-right"></i></a> </div><br /> <div class="prompt-text-area"> <div class="prompt-text-header"> <h3><span class="light-gray-text">Thank you</span></h3> </div> <div class="prompt-text-content"> <pre id="json-container" class="json-output"></pre> </div> </div> </html> <<script>> setup.jsonExample = { "actor": { "mbox": "team.chatsouffle@gmail.com", "name": "Chat Soufflé", "objectType": "Team" }, "verb": { "id": "https://doughahn.github.io/chat-souffle/verbs/collaborating", "display": { "en-US": "collaborating" } }, "object": { "id": "https://doughahn.github.io/chat-souffle/ingredients", "definition": { "name": { "en-US": "Anya, Doug, Evan, Kelly, and Sam" }, "description": { "en-US": "Making recipes with xAPI, Twine, & Veracity" } }, "objectType": "A Learning Guild Team" }, "result": { "score": { "raw": "No one knows yet" }, "response": "Thank you to everyone who got this far" } }; setup.typingSpeed = 35; setup.typeJSON = function(jsonString, index = 0) { const jsonContainer = document.getElementById("json-container"); if (index < jsonString.length) { let char = jsonString[index]; let span = document.createElement("span"); if (char === '"') { const keyMatch = jsonString.substring(index).match(/^"([^"]+)":/); if (keyMatch) { span.classList.add("json-key"); char = keyMatch[0]; index += keyMatch[0].length - 1; } else { span.classList.add("json-string"); const strMatch = jsonString.substring(index).match(/"((?:[^"\\]|\\.)*)"/); if (strMatch) { char = strMatch[0]; index += strMatch[0].length - 1; } } } else if (char.match(/\d/)) { span.classList.add("json-number"); const numMatch = jsonString.substring(index).match(/-?\d+(\.\d+)?([eE][+-]?\d+)?/); if (numMatch) { char = numMatch[0]; index += numMatch[0].length - 1; } } else if (char.match(/true|false|null/)) { span.classList.add("json-boolean"); const boolMatch = jsonString.substring(index).match(/true|false|null/); if (boolMatch) { char = boolMatch[0]; index += boolMatch[0].length - 1; } } span.textContent = char; jsonContainer.appendChild(span); setTimeout(() => setup.typeJSON(jsonString, index + 1), setup.typingSpeed); } }; postdisplay["type-json"] = function() { setTimeout(() => { setup.typeJSON(JSON.stringify(setup.jsonExample, null, 2)); }, 1000); delete postdisplay["type-json"]; }; <</script>>

Welcome!

Welcome to this course on using ChatGPT to generate and improve multiple choice assessments.

This is geared for instructional designers who already have basic skills in assessment development. (Don’t worry if you don’t have experience with writing assessments – you’ll still be able to learn about it.)

There is an xAPI component here too, of course. Throughout the experience, we invite you to open up your browser’s console to see what statements are being sent to the LRS. Keep in mind this is a prototype; things may not work perfectly all the time!

In this course, you’ll get the opportunity to practice:

Using ChatGPT as a tool to generate multiple choice assessments
Applying prompt engineering techniques to iterate on developing multiple choice assessments
Evaluating the quality of distractors in multiple choice assessments generated by AI

Your progress

Self assessment

Setup & review criteria

Track 1: Use a provided assessment

Track 2: Use a custom assessment

Complete the course