Using Data to Inform Instruction

Not everything that can be counted counts, and not everything that counts can be counted.

—Albert Einstein

The word data can seem intimidating at first, with its suggestion of the tedious collection and organizations of lots of numbers, but the truth is that data in the online classroom comprises the phenomenon that we come into contact with daily. With the use of the learning management system (LMS), it can be even easier to collect data and represent the data in a variety of ways.

The types of data that are most widely available in a LMS can be classified into three broad categories: demographic data, usage data, and achievement data.

• Demographic data. Information about students such as enrollment status, selected majors and courses of study, cumulative GPA, prior degree completion, etc. If this data is not available in the LMS, it is most likely housed in an institution’s online Student Information System (SIS).
• Usage data. Also known sometimes as "click-by-click" data, this information reveals how students navigate through and use the features of an online course (e.g., login/logout times; dwell-time on activities such as a discussion forum; utilization of embedded supports such as videos, text-to-speech, etc.).
• Achievement data. Information about student performance such as quiz and test scores, course and assignment grades, etc.

With this data, you can apply basic statistical methods to analyze and interpret the data, using measures of central tendency and variability, as well as item analysis.

• Measures of Central Tendency. Examples of central tendency are the mean (average value in a data set), median (number falling in the middle, when a data set is arranged in ascending order), and mode (most frequently occurring data value).
• Measures of Variability. An important use of statistics is to measure variability or the spread of data. First, the distribution of a data set is a listing or function showing all the possible values (or intervals) of the data and how often they occur. Two measures of variability are the standard deviation and the range. The standard deviation measures the spread of data from the mean or the average score. The range is simply the difference between the largest and smallest data values.
• Item Analysis. Item analysis is a process which examines student responses to individual test or quiz items (questions) in order to assess the quality of those items and of the test as a whole. Aside from standard deviation (defined above), you can find the discrimination index, which indicates how well a question differentiates between high and low performers. It can range from -100% to 100%, with high values indicating a “good” question, and low values indicating a “bad” question.
  • In addition, you can determine the point biserial correlation coefficient which relates individuals’ quiz scores to whether or not they got a question correct. It ranges from -1.00 to 1.00, with high values indicating a “good” question, and low values indicating a “bad” question.

Analyzing & Interpreting Data

In the online learning environment, data and data reports provide important insight into student behaviors, needs, and progress. In fact, some of the data we can harness from these engagement reports will signal concerns about a student in a way that would not be possible in the physical classroom. In addition, understanding what to look for related to student engagement can allow faculty to intervene sooner to provide needed support to a student.

Students often appreciate it when their professor frames a missing assignment outreach with:

When analyzing course data, remember that correlation does not necessarily imply causation. Just because two variables appear to relate to one another does not mean that one causes the other. For instance, if you see a drop in student engagement during a certain week, it doesn't necessarily mean that the learning materials didn't engage students. It could mean that students had deadlines to meet in other courses, or perhaps another assessment you have on the horizon demanded more of their attention than you anticipated. Use whatever context you can to look for correlations between your course data and the conclusions that you draw from it.

Keep in mind also that not all student needs are academic-related. Many are not. Students may be dealing with any number of issues outside of their online activity. It's important that we assume the best when we see concerns about student engagement or performance and approach students from a place of concern and understanding.

In all cases, when providing outreach to students, it is important to approach your communication with a supportive tone. Providing a culture of care is particularly important when students are managing multiple priorities and stress. 

Example:

Dear Emma, I do hope everything is okay with you. I noticed you missed last week's assignment; is there anything I can be doing to help you turn that in, less the late penalties?

Example:

Dear Ryan, I liked the overall tone of this assignment and its intent, and closer alignment to the assignment directions (e.g., using the ACE framework) would have strengthened it and maximized your points.

Spotting Engagement Behaviors

Here are some engagement behaviors to look for as you analyze and interpret data:

• Incomplete or missing work. Students who are not submitting work regularly need quick intervention to help you understand the root of the issue, whether technical, academic, or personal.
• Consistent late work. If a student is consistently submitting late work, reach out to the student to troubleshoot. Support the student’s executive functioning skills by sending resources on time management or other related topics.
• Attendance. Keep tabs on which students have not logged in recently. It’s important that students are maintaining their attendance in the course. Reach out to the student, and be sure to explain attendance policies at the start of the term.
• Lack of consistent content access. Throughout the arc of your course, students should access and progress through your content at a reasonable pace. Students who are not accessing content at the rate of others may need to be reminded of the utility of materials to their performance and/or may signal a need to update resources in your course.
• Inconsistent engagement. If something changes with a student’s behavior (such as if they were very active, and that then changes), reach out to see if there is an issue. Sudden changes in behavior might be a sign that a student has had a change in circumstances.
• Lack of responsiveness. If a student is not responding to your communication, begin by varying your approach. Some students do not check email regularly, and prefer other methods of communication. Open a case in Retain if you are unable to connect with the student.

Data from D2L

D2L Brightspace has excellent built-in statistical tools that we can use as instructors to provide us with a snapshot of where students are at any given time. Below is a list of the tools in D2L and the data that is available.

• Classlist. Your Classlist is found in the "Other Tools" menu visible right under the name of your course. Here, you can see the list of your students and can access information about them by clicking on their profile or picture. Importantly, you can see when they last accessed the course on this screen. (You can watch the Classlist Tools for Instructors tutorial on the D2L Brightspace YouTube channel for more information.)
• Class Engagement. The Class Engagement function can be found in the right hand navigation bar. Here, you can see the grade distribution of the class, each student's performance, when they last logged on, and what activities they have participated in. (You can read more about the Class Engagement interface on the Brightspace website.)
• Class Progress. The Class Progress function is also found in the "Other Tools" menu visible right under the name of your course. Here, you can see how much content your students are accessing, and also charts about their logins and grades (you can see drop-off in logins or grades in this view). Red, Yellow, and Green markers visually indicate risk levels of each student as well. (You can watch the Class Progress tutorial on the D2L Brightspace YouTube channel for more information.)
• Gradebook. The Gradebook tool is one that faculty are most familiar with and a quick glance after grading each week will tell you if a student has not turned in an assignment and color-coding indicates risk and lateness (the latter only if due dates are entered into the system) by panning through the graded items. (You can watch the Grades for Instructors tutorial on the D2L Brightspace YouTube channel for more information.)
• Assignments, Discussions, Quizzes, and Rubrics. You can view statistics about students' submission and attempts, including an item analysis to assess the quality of quiz items and of the test as a whole.

Data can help you answer questions, such as, "When and which resources did students view prior to submitting an assignment?", "Which discussion generated the most traffic---that is, has the most student views?", "Which resources are used most frequently that are important to students' success on a quiz?", and "What are the patterns of performance on an assignment or quiz as related to how often students logged on to the course?"

More on the User Progress Tool

The User Progress tool can help you track student progress in a course by measuring their completion of nine different progress indicators. You can use User Progress to track your students' overall progress and prepare progress reports, while students can use User Progress to keep track of all of their course-specific assignments and feedback.

Reflective Practice

Locating and reviewing data is only one step in the process of data-driven instruction. Faculty must also engage in reflective practice to make sense of the data, examine trends, dispel myths, and set goals.

Engaging in reflective practice means that you intentionally set aside time to consider what your objectives were for a particular class, module, or lesson, what approaches you used to meet those objectives, how you measured those objectives, and what worked well that you’ll continue to expand on or what you might adjust. All of these steps are made much easier in the online environment when instructors can leverage data as evidence throughout the reflective process.

Reflective practice challenges us to use the data that is richly available in the LMS to make decisions. For example, the Class Engagement function available in D2L can help faculty see a more accurate trend of how many students are engaged and in what. If student engagement dips, it might be an indicator that there needs to be more scaffolding or resources provided.

Likewise, using the Class Progress function gives insight into how the class is progressing, and whether students are on track or not. By reviewing and reflecting on these data, faculty have the advantage of making real-time changes that can drastically impact student learning and success.

Finally, by staying abreast of the color-coding within the Gradebook function, faculty can confirm (or not!) their assumptions about student learning through assessment. Again, the benefit being that this important reflection can help faculty not only make data-informed decisions about how to adjust their methods, resources, or outreach, but also work on goal setting, as they engage in a process of personal continuous improvement.

There are five main principles that will make sure you get the most out of your reflections: reacting, recording, reviewing, reworking, and re-assessing. These are sometimes referred to as the five "Rs". When analyzing and interpreting data, react (decide what you need to focus on), record (write down what you think you need to do), review (what does the data really tell you), rework (what can you change or adapt), and reassess (take time to think about how successful you were with the new strategy).

Continuous Course Improvement

Whether you designed and developed a course yourself or teaching a course with content from a course master, taking time, especially near the end of the term, to reflect on efficacy of a course is a critical aspect of teaching practice. Data can be very useful in identifying opportunities to improve a course for future terms, whether you developed the course on your own or you used content from a course master.

First, identify opportunities to enhance the learning in the course. Here are a few questions to ask yourself: 

• Does achievement data show that students may need additional guidance to complete an assignment or contribute to a discussion?
• Does your analyses of students' performance on quizzes suggest that the questions are appropriately difficult and effectively differentiate among students on the basis of how well they knew the material being assessed?
• Does usage data show that students did not access certain resources that are important to the development of knowledge, skills, and dispositions included in the learning outcomes? Does additional information need to be included to drive students to using the resource?  

Next, think about how the course could be improved. As you reacted to the data throughout the term, what strategies did you try and felt successfully helped students overcome challenges? What changes did you see in the data that let you feel the new strategy was successful? Which could be used again and incorporated into the design of the course?

Finally, share your recommendations with your program leader. They would appreciate any insights you have to improve the course to enhance student achievement, engagement, and satisfaction.