User Tools

Site Tools


paper-summaries
A Bayes Net Toolkit for Student Modeling in Intelligent Tutoring Systems

Kai-min Chang, J. B. J. M. A. A. C. (2006). A Bayes Net Toolkit for Student Modeling in Intelligent Tutoring Systems, 1–10.

  • Mainly introduce their toolBNT-SM that takes a data set and XML spec of a Bayes net model and then executes code that trains and tests the model
  • While the package is not interesting, the first sections describing bayes nets and their version of models is useful
    • student performance affected by student knowledge and tutor intervention
    • student knowledge affected by tutor intervention
  • More general points that I like and need reminding of
    • cannot read the students mind, therefore must infer knowledge state from observable events
    • student knowledge not easily inferred from observable events
      • guessing
      • slipping
    • student knowledge changes over time
Student assessment using Bayesian nets

Martin, J., & VanLehn, K. (1995). Student assessment using Bayesian nets. International Journal of Human Computer Studies, 42(6), 575–592.

  • From 1995, so quite old
  • Knowledge unit is rules with different types
    • we decided that our knowledge units or nodes would not have types so don’t think this is a very useful paper
  • Mentions a third possible difficulty in inferring student knowledge: if there are many ways to produce the same answer the answer is not enough to credit the student with any specific knowledge
Informing the design of a course data visualisator: an empirical study

R. Mazza and V. Dimitrova. Informing the design of a course data visualisator: an empirical study (ICNEE 2003)

  • A bit old and information is more about for course management systems (CMS)
  • They did a survey asking teachers that use CMS’s what kind of analytics they would want
  • more interesting points:
    • many instructors wanted something to help them discover students progressing slowly
    • many instructors wanted to know students access and time spent behavior with the system
  • They built a system based on this survey CourseVis
Automated Predictive Assessment from Unstructured Student Writing

Ming, Norma C., and Vivienne Ming. “Automated Predictive Assessment from Unstructured Student Writing.” DATA ANALYTICS 2012, The First International Conference on Data Analytics. 2012.

  • Has some useful looking motivator references
  • General idea
    • Analysis
      • Look at “semantic content of student-generated products”
      • probabilistic latent semantic analysis (pLSA)
        • assume topic and n-gram likelihoods are Gaussian
        • use topics inferred from student posts, specifically the inferred coefficients of the latent factors, as predictor variables
      • hierarchical latent dirichlet allocation (hLDA)
        • multinomial model of word occurrence and from that infer hierarchy of topics
        • from this get both topic and specificity
      • Training data
        • weekly topic coefficients
    • Data
      • forum posts
        • students required to respond to two discussion questions per week
      • “Document “generated” by mixing topics and then selecting words from those topic mixtures”
        • What does this mean?
      • Tweaks
        • removed students not finish course
  • Results
    • hLDA is the best predictor
    • as get more data get better predictions
    • using the hierarchy depths from hLDA found that higher grades are correlated with using topics at a deeper depth
  • Interesting points
    • each algorithm also comes up with what we would basically call nodes, though how they are connected is not how we would define an edge
  • Questions Raised
    • Could this data be used as a formative assessment to provide more content relevant feedback?
      • I imagine so, but if the response to a low prediction from this thing is to write more forum posts, not quite sure that is the correct one
    • I’d be curious to see how “severe” the prediction error was. As in what is the histogram of how off the prediction was
Educational Data Mining: A Review of the State of the Art

Romero, C., & Ventura, S. (n.d.). Educational Data Mining: A Review of the State of the Art. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 40(6), 601–618. doi:10.1109/TSMCC.2010.2053532

A method for finding prerequisites within a curriculum

Vuong, Annalies, Tristan Nixon, and Brendon Towle. “A method for finding prerequisites within a curriculum.” Proc. of the 4th International Conference on Educational Data Mining. 2011.

  • I think this paper is definitely useful
  • General idea
    • Data
      • From Carnegie Learning’s Cognitive Tutor where material is divided into 4 main curricula, each curricula is made up of units, and each unit is made up of sections
      • Teachers were allowed to rearrange and create curricula with the units
      • Teachers could not change the sections within a unit
    • Analysis
      • Looked at the order students went through a unit (i.e. Unit A and Unit B)
        • Sample A - Those that had and did at least one problem in Unit A and Unit B
        • Sample B - Those that did not have Unit A and attempted at least one problem in Unit B
      • Define success as mastering the first section of Unit B
      • Looked at success rate of mastering Unit B with and without Unit A
      • Test
        • Binomial test with alpha = 0.01
        • look for significant difference in average success rate for Sample A compared to the success rate of Sample B
  • Interesting Results
    • Only 56% agreement between expert and data of what are and are not prerequisites
    • Disagreement where expert claim prereq but data doesn’t see it are commonly earlier in the curriculum
      • their explanation is that a large number of students already proficient on the preliminary material that more practice is unnecessary, so data makes it appear like it is not a prereq
  • Potential Data Tweaks that may be a bad idea
    • Only considered if a student succeeded in the first section of Unit B, assuming that their success in later sections is due to within unit experience and not previous units
  • Questions raised
    • It could be that Unit A and Unit B share some common skill set and the practice of that skill set makes it appear that Unit A is a prereq of Unit B
    • Are there other metrics for student performance that would show similar results?
Adaptive item-based learning environments based on the item response theory: possibilities and challenges

Wauters, K., Desmet, P., & Van den Noortgate, W. (2010). Adaptive item-based learning environments based on the item response theory: possibilities and challenges. Journal of Computer Assisted Learning, 26(6), 549–562. doi:10.1111/j.1365-2729.2010.00368.x

  • Dan claiming this one
  • Goes over a bunch of the different issues associated with different data sets and item response theory
  • Makes an interesting distinction between adaptive hypermedia (AH) and intelligent tutor systems (ITS) - AH are a lot of content while ITS are some content with a focus on supporting problem solving
  • Paper focused on one kind of ITS: adaptive curriculum sequencing with item based ITSs
  • Data set challenges
    • missing values because students can choose what to do
      • NOT AN ISSUE
      • I think more of our data set is from quizzes that students have to take, though I can see this becoming more of an issue in the future but possibly prevented because with such a large student population we could reasonably believe that we’ll have enough data for each question
    • skipped items
      • NOT AN ISSUE
      • I think for the same reason as previous
  • Algorithm challenges
    • item difficult estimation
      • chicken and egg problem, portion of people that attempted and succeeded on an item is dependent on who saw it which is dependent on the algorithm that presented it to them
        • COULD BE AN ISSUE
        • I think we’d have to tread carefully on this one
      • therefore the proportions are only comparable over items if the group of persons is also comparable
      • literature evidence that say test specialists can and cannot reasonably estimate difficulty (but not useful for us if we want to automate everything)
    • ability estimation
      • cold start problem - no available information or very small data set in the beginning
        • MIGHT BE AN ISSUE
        • We will know what courses the student took before and/or could require a pretest, which might be a good thing regardless for MOOC courses to help predict if the student will even make it to the end
      • problem of evolving ability level - the students ability changes over time
        • NOT AN ISSUE
        • While for IRT this could be a problem, we actually want this and will take it into account in our model
    • item selection algorithm
      • if goal is to find out what a student knows the optimal items to give them are ones where they have a 50% chance of success, but this is demotivating and can increase test anxiety
        • MIGHT BE AN ISSUE
        • If we plan to dynamically provide them information like in IRT this might be an issue
        • However since our goal is learning not assessment we don’t have to worry about trying to get an accurate assessment of their abilities, so we should focus on keeping the students motivated which probably increases the threshold of the chance of success on the items we give them
        • the paper suggests giving students also a choice in difficulty level, which might not be a bad idea
  • Good general points
    • Different ways to adapt
      • form - how present the content
      • content - what present to the user while they are going through some steps
      • curriculum sequence - what order present the questions
    • Different items can take into account
      • course/item features - course difficulty level and topic
      • person features - learner’s knowledge level, motivation, cognitive load, interests, preferences
      • context features - time, place, and device during learning experience
Getting to Know your Student in Distance Learning Contexts

Zinn, Claus and Scheuer, Oliver - Getting to Know your Student in Distance Learning Contexts

  • Most of the population they surveyed is different than a MOOC population, so I think this might be only useful as a starting point of our own survey on trying to find out what people are interested in
  • Has good citation on why teachers should know more about their students:
    • “Research comparing effective and ineffective teachers cites the existence and use of a systematic procedure for keeping and interpreting data on student performance as a notable difference between these groups.”
      • K. Cotton. Monitoring student learning in the classroom. Northwest Regional Educational Laboratory, U.S. Department of Education, 1988. School Improvement Research Series (SIRS).
  • Possible future papers to find:
    • R. Mazza and V. Dimitrova. Informing the design of a course data visualisator: an empirical study (ICNEE 2003)
paper-summaries.txt · Last modified: 2018/02/28 17:02 (external edit)