# Survey Methods

Introduction We used two survey instruments to measure student outcomes from inquiry-based learning in undergraduate mathematics and to compare these outcomes between various student groups, in particular, between IBL and non-IBL students. The attitudinal survey was designed to detect the quality of and changes in students’ mathematical beliefs, affect, learning goals, and mathematical problem-solving strategies.

The learning gains survey (SALG-M) measured students’ experiences of class activities and their cognitive, affective and social gains from a college mathematics class. The surveys addressed the following questions • • • • • What learning gains do students report from an IBL mathematics class? How do students experience IBL class activities? How do students’ class experiences account for their gains? What kind of beliefs, affect, goals and strategies do IBL students report at the start of a mathematics course? How do these approaches change during a college mathematics course?

How do these changes relate to or explain students’ learning gains? For each of these outcomes—learning gains, experiences, attitudinal measures, and changes—how do the outcomes for IBL students differ from those of non-IBL students, and among IBL student sub-groups? The survey instruments provided us with large student data sets from four campuses, gathered during the two academic years 2008-2010. They offered us a comprehensive picture of students’ approaches to learning college mathematics as well as of their experiences and gains from IBL classes.

Moreover, the survey data could be used to analyze differences in reported learning approaches, classroom experiences and learning outcomes among various student groups. In addition to structured questions, students also could write about their experiences and gains in the open-ended survey questions. Both the open-ended survey answers and student interview data were used to validate, confirm, and fill in the picture of student outcomes obtained from the structured survey responses. A3. 2 Study sample The data were gathered on all four campuses in a variety of undergraduate courses.

These included courses entitled: • • • • (Honors) Analysis 1-3, (Honors) Calculus 1-3, Cryptology Discrete mathematics, Cite as: Assessment & Evaluation Center for Inquiry-Based Learning in Mathematics (2011). (Report to the IBL Mathematics Project) Boulder, CO: University of Colorado, Ethnography & Evaluation Research. Appendix A3: Survey Methods • • • • • • • • • Explorations in mathematics, Exploratory calculus, Group theory, Introduction to proofs, Introduction to real analysis, Multivariate calculus 1-2, Number theory, Probability, Real analysis 1. A3-2

They covered the full range of introductory to advanced mathematics courses. Mathematics courses specifically developed for elementary and middle school or secondary school pre-service teachers represented another type of course in the sample. This kind of survey data was obtained from two campuses. Additional smaller data sets came from a geometry course designed (but not required) for prospective high school mathematics teachers at one campus. In all, we collected surveys from 82 college mathematics sections, of which 65 were IBL sections and 17 non-IBL sections.

Data obtained with our surveys consisted of an attitudinal presurvey, a learning gains post-survey, and a combined post-survey including both the attitudinal and the learning gains questions. We received pre-surveys from 1245 students, learning gains post-surveys from 200 students, and combined post-surveys from1165 students. Combining the pre-survey data with the post-survey data produced us information from 800 individually matched surveys. These surveys included responses from 412 IBL math track students (i. e. students who studied mathematics as their major or minor subject), 156 non-IBL math track students, 208 IBL pre-service teachers, and 25 non-IBL pre-service teachers. Tables A3. 1-A3. 4 display features of our sample based on the personal information from the presurvey responses. A3. 2. 1 Survey Sample by Gender Students reported their gender both in the pre- and post-survey. Even though these were not always the same students, the percentages of women and men were rather consistent in the two surveys. About 60% of all the students were men. This varied along with student groups.

Typically, nearly 70% of the math-track students were men, whereas most of the IBL pre-service teachers (84% pre; 86% post) were women. Appendix A3: Survey Methods Table A3. 1: Survey Respondents by Gender and Course Type IBL math-track Gender Women Men TOTAL Women Men TOTAL Count 194 383 577 169 354 523 % 33. 6 66. 4 100% 32. 3 67. 7 100% Pre-Survey 104 194 342 92 231 323 30. 4 69. 6 100% 28. 5 71. 5 100% 190 37 227 190 32 222 83. 7 16. 3 100% 85. 6 14. 4 100% 12 13 25 17 15 32 48. 0 52. 0 100% 53. 1 46. 9 100% Non-IBL math-track Count % IBL pre-service Count % Non-IBL pre-service Count %

A3-3 Total Count 500 671 1171 468 632 1100 % 42. 7 57. 3 100% 42. 5 57. 5 100% Learning Gains Survey A3. 2. 2 Survey Sample by Academic Major We classified students by their reported main major, prioritizing their most mathematically oriented major. Accordingly, all students with a major in mathematics or applied mathematics were classified into one category, even if they had a second, non-mathematics major. Science majors included students with a major in physics, chemistry or another science, but not in mathematics or applied mathematics.

Engineering and computer science majors formed another category, as did students with a major in economics. All students who reported any a non-science major were classified into one group. Table A3. 2: Survey Respondents by Academic Major Main academic major Math or applied math Science Engineering or computer science Economics Other non-science TOTAL IBL math-track Count 337 82 63 28 49 559 % 60. 3 14. 7 11. 3 5. 0 8. 8 100% Non-IBL math-track Count 163 53 57 47 10 330 % 49. 4 16. 1 17. 3 14. 2 3. 0 100% IBL pre-service Count 86 18 4 3 111 222 % 38. 7 8. 1 1. 1. 4 50. 0 100% Non-IBL pre-service Count 19 1 2 0 3 25 % 76. 0 4. 0 8. 0 0. 0 12. 0 100% Total Count 605 154 126 78 173 1136 % 53. 3 13. 6 11. 1 6. 9 15. 2 100% More than half the students reported a mathematics or applied mathematics major. Students who had a non-science major mostly represented IBL pre-service teachers. Science majors formed the next biggest student group. Engineering or computer science majors (11. 1%) and economics majors were the two other majors represented in the sample. In addition to mathematics or Appendix A3: Survey Methods A3-4 pplied mathematics students, students from other STEM fields were also well represented in the sample. IBL math-track students were pursuing a math major slightly more often (60. 3%) than non-IBL math-track students (49. 4%). The proportions of science majors was similar, but more of the non-IBL math-track students were economics majors or engineers. Fully half of the IBL preservice teachers were non-science (e. g. , education) majors. These represented mostly elementary or middle school pre-service teachers. But the sample also included many secondary pre-service teachers with a math major.

A3. 2. 3 Ethnicity and Race We classified students by race into three different categories. All the students who considered themselves white and not a representative of any other race were denoted White. The category Asian consists of all the students who considered themselves only Asian, or Asian and some other race. If the students did report some other race besides White or Asian, they were classified as multiracial students. Ethnicity was a separate item; here students could choose between Hispanic or Latino, or Not Hispanic or Latino.

The distributions of respondents by ethnicity and race are shown in Table A3. 3. Table A3. 3: Survey Respondents by Ethnicity and Race IBL Math Track Count Ethnicity Hispanic or Latino Not Hispanic or Latino 47 520 567 Race Asian Multiracial White TOTAL 138 24 378 540 25. 6 4. 4 70. 0 100% 118 12 181 311 37. 9 3. 9 58. 2 100% 22 19 159 200 11. 0 9. 5 79. 5 100% 6 1 12 19 31. 6 5. 3 63. 2 100% 284 56 730 1070 26. 5 5. 2 68. 2 100% 8. 3 91. 7 100% 37 296 333 11. 1 88. 9 100% 27 195 222 12. 2 87. 8 100% 10 14 24 41. 7 58. 3 100% 121 1025 1146 10. 89. 4 100% % Non-IBL Math Track Count % IBL Pre-Service Count % Non-IBL PreService Count % Total Count % Less variety appeared in students’ ethnicity and race. Most of the students were white and not Hispanic or Latino. About a quarter of the students were Asian (26. 5%), but the sample included only a few students from other races (5. 2%). The sample represents a distribution that is typical for mathematics students in the large research universities that our study targeted. Appendix A3: Survey Methods A3. 2. 4 Academic Status A3-5

The pre-survey provided us with information about students’ academic status at the beginning of their mathematics course. Table A3. 4 shows the distribution of respondents’ academic background by course type. Table A3. 4: Survey Respondents by Academic Status First year Student Group IBL math-track Non-IBL math-track IBL pre-service teachers Non-IBL pre-service teachers TOTAL Count 206 147 3 0 356 % 35. 8 43. 2 1. 3 0. 0 30. 5 Sophomore or Junior Count 183 117 99 10 409 % 31. 8 34. 4 43. 8 40. 0 35. 0 Senior or more Count 187 76 124 15 402 % 32. 5 22. 4 54. 9 60. 0 34. Total Count 576 340 226 25 1167 % 100 100 100 100 100% Our sample included students across all stages of their college studies. However, nearly one-third (30. 5%) of all the students were first-year students. This applied especially to IBL (35. 8%) and even more to non-IBL (43. 2%) math-track students. The pre-service teachers in the sample were further along in their studies. More than half (54. 9%) were seniors or even more advanced students but only three of them were first-year students. The same trend applied to the small group of non-IBL pre-service teachers. A3. 3 Survey instruments

The final survey instruments consisted of an attitudinal pre-survey, a learning gains post-survey, and a combined post-survey including both the attitudinal questions and the learning gains questions. Both the pre- and post surveys gathered personal information about students’ gender, race and ethnicity, class year, academic majors, grade-point average, and plans to pursue teaching certification. We asked students to set themselves an identifier at the end of each survey. These identifiers were used to match the pre-survey responses with the post-survey responses individually.

In order to check the survey items and the structures, both the attitudinal and the learning gains survey were tested with two small samples of college mathematics students. Descriptive statistics and principal component analysis were used with these preliminary data sets to check the reliability of the questions and theoretical constructs in the surveys. Based on these analysis, we left out ill-behaving questions and shortened the attitudinal survey. In order to shorten the combined post-survey, we also left out some overlapping questions from the initial learning gains survey.

The final surveys are presented as Exhibit E3. 1 and E3. 2. Appendix A3: Survey Methods A3. 3. 1 Attitudinal Survey A3-6 We wanted to study the nature of students’ mathematical beliefs, affect, learning goals, and mathematical problem-solving strategies, and changes in these during a college mathematics course. We designed a structured survey to measure undergraduate students’ mathematical beliefs, affect, learning goals and strategies of problem solving, to be administered at the beginning and end of a college mathematics course.

The seven sections measured students’ interest in and enjoyment of mathematics, preferred goals in studying mathematics, and their frequency of use of various problem-solving actions when doing mathematics, and their beliefs about learning mathematics, problem solving, and proofs. A3. 3. 1. 1 Theoretical basis of the attitudinal survey The sub-sections and items were constructed on the basis of theory and previous research on mathematical beliefs, affect, learning goals and strategies of learning and problem solving. Mathematics education research n beliefs has introduced concepts such as beliefs about the nature of mathematics, about learning mathematics, about problem-solving, and beliefs about the self as a mathematics learner (Malmivuori, 2001; McLeod, 1992). All these categories of beliefs appear to have important implications for how students approach the study of mathematics and act in mathematics learning situations at various age and schooling levels. They may either significantly hinder or help student learning, performance and problem solving (Leder, Pehkonen, & Torner, 2002; Schoenfeld, 1992).

Moreover, they influence the development of negative or positive attitudes toward mathematics that have longer-term impacts on students’ choices of studying mathematics. We wanted to check the quality of and changes in these types of important mathematical beliefs. In addition, we chose to study certain types of beliefs that were particularly important for studying college mathematics and that might display possible differences between students in traditional and IBL mathematics courses. For example, mathematical proving represents an important area of beliefs in college mathematics.

How students see the nature of proofs significantly affects their success in college mathematics (Knuth, 2002; Selden & Selden, 2007; Sowder & Harel, 2003). Moreover, previous research has identified some differences in these beliefs between students who took traditional or student-centered IBL mathematics classes students (Ju & Kwon, 2007; Yoo & Smith, 2007). Based on these criteria, testing and revisions of the attitudinal survey, we measured students’ beliefs about: • • • • learning of mathematics (instructor-driven, group work, exchange of ideas) mathematical problem-solving (practice vs. easoning) mathematical proving (proving as a constructive activity or as confirming truths; Yoo & Smith, 2007), beliefs about the self (confidence in their own math ability, in teaching mathematics) Appendix A3: Survey Methods A3-7 Studies on affect have a long tradition in mathematics education research. Attitudes about mathematics, confidence, motivation and anxiety are the most-studied factors, found to essentially strengthen or diminish students’ illingness and ability to learn mathematics (Frost, Hyde, & Fennema, 1994; Goldin, 2000; Malmivuori, 2001, 2007; McLeod, 1992). Interest in and enjoyment of mathematics learning represent central features of affect and students’ motivation. Interest is suggested to facilitate deep rather than surface-level processing, and the use of more efficient learning strategies (Entwistle, 1988; Schiefele, 1991). In turn, students who enjoy learning tend to exert more effort and persist longer when they are challenged (Stipek, 2002).

Both interest and enjoyment indicate students’ strong positive relationship to mathematics and willingness to spend time and effort in studying mathematics. This relationship is also importantly weakened or strengthened by students’ confidence in their own ability to do and learn mathematics. This applies to female students in particular (Fennema, Seegers & Boekaerts, 1996). But confidence as related to enhanced self-efficacy is found to essentially promote all students’ engagement and cognitive performance (Bandura, 1993; Malmivuori, 2001; Zimmerman, 2000).

Recent education psychological literatures suggest that the types of learning goals pursued by students also profoundly impact the quality of their learning. They direct students’ level of achievement, self-regulation and problem-solving strategies (Pintrich, 2000). Closely related to personal interest, we studied students’ learning goals, categorizing them broadly as intrinsic vs. external. For example, students who pursue high grades, seek particular degrees, and display high competence and self-concept express external or performance goals that are related to superficial learning.

In contrast, intrinsic or mastery-focused goals such as a focus on one’s own effort, pursuit of knowledge, and desire to understand the learned material are seen to result in independence, responsibility and deeper learning (Ames & Archer, 1988). In contrast to externally motivated students, intrinsically motivated students show higher interest, excitement, and confidence that enhance their performance, persistence, and creativity (Ryan & Deci, 2000).

IBL teaching practices often involve group work and collaboration that both require and develop communication skills (Gillies, 2007; Duch, Groh, & Allen, 2001). Indeed, recommendations for undergraduate programs in mathematics include development of analytical thinking, critical reasoning and problem-solving but also communication skills (Pollatsek et al. , 2004). In the attitudinal survey, we wanted to study students’ preferences for communicating about mathematics and any change in this during their IBL course. Items on students’ goal of communicating about mathematics measured this preference.

In addition to mathematical confidence, our attitudinal survey studied students’ affect and motivation in the form of: • • • personal interest in mathematics, willingness to pursue a math major (or minor), plans to study more math in the future, Appendix A3: Survey Methods • • • • interest in teaching, intrinsic and extrinsic learning goals, goal for communicating about mathematics, enjoyment of learning mathematics A3-8 Current understanding of mathematical knowledge as a constructive activity (Steffe & Thompson, 2000; Tall, 1991) focuses on skillful problem-solving (Schoenfeld, 1992).

Competent mathematicians use strategies to make sense of new problem contexts or to make progress toward the solution of problems when they do not have ready access to solution methods for them (Schoenfeld, 2004). The nature of the problem-solving strategies they choose influence students’ approaches to and success at challenging mathematical problems. For example, unlike novices, expert problem-solvers use high-level planning and qualitative analysis before attacking a problem. They also demonstrate facility in choosing appropriate strategies in various situations (Kroll & Miller, 1993; Schoenfeld, 1985).

This contrasts with a lack of strategies, mechanical use of concepts, and rote memorization of previous similar problems. IBL approaches provide opportunities for students to engage in knowledge creation and argumentation (Rasmussen & Kwon, 2007). Such activities are generally suggested to promote problem-solving skills, independent thinking and intellectual growth (Buch & Wolff, 2000; Duch, Gron & Allen, 2001). Competent problem solvers can communicate the results of their mathematical work effectively, both orally and in writing (Schoenfeld, 2004).

Planning and selfmonitoring the solving process also help to ensure skillful problem-solving (Schoenfeld, 1985). Moreover, mathematics students must develop persistence in the face of difficulties, tolerance for ambiguity, and willingness to try multiple approaches, and they must learn to apply the necessary amount of rigorous and judgmental reasoning (Hanna, 1991; Pollatsek et al. , 2004). These skills require them to develop self-reflective and self-regulatory strategies (Burn, Appleby & Maher, 1998; De Corte, Verschaffel & Eynde, 2000).

We wanted to check what kind of learning and problem-solving strategies students report and how these change during IBL and traditional math courses. Our attitudinal survey studied students’ use of: • • • independent (or individual), collaborative, or self-regulatory strategies. Items related to these strategies were intended to explore the extent to which students counted on their own thinking and creativity when solving math problems and proofs, shared their thinking and strategies with other students, and actively reflected on and regulated (planned or checked) their own thinking and actions while solving math problems.

Appendix A3: Survey Methods A3. 3. 2 Example items from the pre/post attitudinal survey instrument A3-9 We divided the attitudinal survey into eight subsections, each consisting of 7-15 structured items. The answers varied on a 7-point Likert-scale between negative and positive responses. The subsections of the survey measured students’ mathematical beliefs, motivation, learning goals, enjoyment, confidence, and strategies for leaning and problem-solving. All the attitudinal presurvey questions and items are presented in Exhibit E3. 1. We studied students’ • • • Personal interest in studying mathematics: How likely is it that you will… e. . , “Bring up mathematical ideas in a non-mathematical conversation? ” Enjoyment of doing and discovering mathematics: How much do you enjoy… e. g. , “Discovering a new mathematical idea? ” Goals in studying mathematics: Below are some goals that students may have in studying mathematics. How important is each goal for you? e. g. , “Memorizing the sets of facts important for doing mathematics. ” (extrinsic goal) “Learning to construct convincing mathematical arguments. ” (intrinsic goal) Beliefs about the self: confidence: How confident are you that you can… e. g. , “Apply a variety of perspectives in solving problems? (math ability) “Teach mathematics to high school students? ” (teaching mathematics) Beliefs about learning mathematics: I learn mathematics best when… e. g. , “The instructor lectures. ” (instructor-driven) “I work on problems in a small group. ” (group work) “I explain ideas to other students. ” (exchange of ideas) Beliefs about problem-solving: In order to solve a challenging math problem, I need… e. g. , “To have lots of practice in solving similar problems. ” (practice) “To use rigorous reasoning. ” (reasoning) Beliefs about proofs: The following statements reflect some students’ views about mathematical proof.

How much do you agree or disagree with each statement? e. g. , “Proof is a tool for understanding mathematical ideas. ” (constructive) “The main purpose of proof is to confirm the truth of a mathematical result that is already known to be true. ” (confirming) Problem-solving strategies: When you do math, how often do you take each action listed below? e. g. , “Find your own ways of thinking and understanding. ” (independent) “Brainstorm with other students. ” (collaborative) “Plan a solving strategy before attacking a problem. ” (self-regulatory) • • • • • Appendix A3: Survey Methods A3. 3. Demographic and background information A3-10 The attitudinal survey also asked for demographic and background information about students’ previous achievement, personal information, and expectation for the grade of the target course. The questions dealt with: • achievement history: the highest level of high school mathematics taken, any AP Calculus test taken and scores received, number of college math courses taken, estimated overall GPA; academic background: class year, college major, pursue for a teaching certification; personal background: gender, ethnicity, race; expected grade for the course. • • •

At the end of both the pre- and post-survey, students were asked to assign themselves an identifier. This enabled us to match between pre-survey and post-survey responses by individual student (see Exhibit E3. 1). A3. 4 Learning Gains Survey The learning gains post-survey was based on the SALG instruments (SALG, 2008) developed to enable faculty and program evaluators to gather formative and summative data on classroom practices. The questions address students’ self-reported experiences of mathematics class practices and their cognitive, social and affective learning gains due to their participation in a college mathematics course.

Students provide both quantitative ratings and written responses about the course focus, learning activities, content and materials. The learning gains instrument is grounded in its authors’ (Seymour, Wiese, Hunter & Daffinrud, 2000) findings that: • students can make realistic appraisals of their gains from aspects of class pedagogy and of the pedagogical approach employed, and • this feedback allows faculty to identify course elements that support student learning and those that need improvement if specific learning needs are to be met.

The SALG instrument is easily modified to meet the needs of individual faculty in different disciplines and it has been found to be a powerful and useful tool for instructors in student feedback and course development. When first developed, data about the use of the survey showed that eighty-five percent of the instructors reported that the SALG provided qualitatively different and more useful student feedback than traditional student course evaluations. Instructors also made modifications to course design (60%) and class activities (lecture, discussion, hands-on activities) followed by tudent learning activities (54%) course content (43%), and the information given to students (33%) (Recommendations for using the SALG, 2008). We adjusted the SALG items to match college mathematics situations. The final learning gains survey, which we call the SALG-M, consisted of four structured sections on course experiences Appendix A3: Survey Methods A3-11 and two sections on learning gains. The first four sections asked about students’ experiences of instructional practices: how much particular practices helped their learning.

The practices deal with overall instructional approach, classroom activities, tests and other assignments, and interactions during the course. Answers follow a five-point scale between “no help” and “ great help. ” Two other structured sections of the questionnaire ask about students’ gains in understanding, confidence, attitude, persistence, and collaboration. These answers vary on a fivepoint scale between “no gain” and “great gain. ” The final post-survey for the SALG-M is presented as Exhibit E3. 2. In addition to structured items, the learning gains post-survey included four open-ended questions.

Students were provided space to write about: • • • • How the class changed the ways they learn mathematics How their understanding of mathematics changed as a result of the class How the way the class was taught affected their ability to remember key ideas What they will carry with them from the class into other classes or other aspects of their life. Answers to these questions complemented numerical responses on students’ gains from their mathematics courses, helping us to better understand these results.

The learning gains survey also gathered information on students’ expected grade at the end of the course, college major, class year, gender, and whether they were pursuing teaching certification. These questions confirmed the match between pre- and post-surveys and enabled us to detect changes in students’ ideas or plans. The complete post-survey consists of the structured items in the attitudinal pre-survey and all the sections of the learning gains survey. A3. 5 Data collection

Survey data were gathered from undergraduate students studying mathematics at all the four campuses during two academic years 2008-2010. We started with online survey instruments when testing the survey instruments and also gathered pre-surveys at one campus in early fall of 2008. Due to the low response rate to the online form, we gathered the rest of the survey data as a paper-and-pencil test in class, which yielded very high response rates. The paper questionnaire was administered at the beginning and end of each course.

In the courses that were part of a multi-term sequence (e. g. , a three-quarter calculus sequence), we administered the full postsurvey only at the end of the final section, but also gave a learning gains survey at the end of each previous related section. This provided us with some longitudinal data on the evolution of students’ experiences and learning gains over multiple terms of IBL or comparative instruction. The surveys were delivered to our project collaborators at the four campuses who also mostly administered the surveys in class.

In some cases, course instructors or teaching assistants administered the survey. Instructors were given instructions on how to administer the surveys and return the completed surveys to us. We also reminded them about keeping the confidentiality Appendix A3: Survey Methods A3-12 and anonymity of students in every step. Filling out the surveys took students about 10-20 minutes; instructors were asked to offer enough class time for completing the surveys. A3. 6 Data analysis Attitudinal survey variables A3. 6. 1

Composite variables were constructed based on the attitudinal survey design and the factors of mathematical beliefs, affect, learning goals, and strategies of learning and problem-solving presented in Section A3. 3. 1. 1. Exploratory factor analysis, principal component analyses and item analyses on the attitudinal survey data were used to create the final composite variables: five measures of motivation, affect and confidence; three measures of learning goals; seven measures of beliefs about mathematics and learning; and three measures of strategies.

For each composite variable, averages of student ratings across the items represented the score for each student. This enabled us to interpret results on the same scale as that for the original attitudinal survey items. The composite variables were then used to report results on students’ attitudes and for further analysis on group differences in attitudes. Table A3. 5 displays the survey questions and items for each composite variable, the titles and descriptions of the composite variables, and the reliability scores for the composite variables (for the pre-survey data and post-survey data separately).

Table A3. 5: Composite Variables Measuring Student Beliefs, Affect, Goals and Problem-Solving Strategies Scale Count Motivation Interest Math major Math future Teaching Enjoyment Confidence Math confidence Teaching confidence Confidence in own mathematical ability Confidence in teaching math 7 7 5 2 Q9: 1,2,4, 5,6 Q9: 3,8 0. 820 0. 696 0. 826 0. 645 Interest in learning and discussing mathematics Desire to graduate with a math major Desire to pursue math in future work or education Desire to teach math Pleasure in doing and discovering mathematics 7 7 7 7 7 3 1 2 1 6 Q1: 5,6,7 Q1: 2 Q1: 1,4 Q1: 8 Q2: 1-6 0. 08 0. 439 0. 914 0. 828 0. 615 0. 928 Items Numbers Reliability Cronbach alpha Pre Post Variable Description Appendix A3: Survey Methods Table 3. 5, continued… Goals for studying math Intrinsic Extrinsic Communicating Learning new ways to think & to apply math Meeting requirements; degree, good grades Communicating mathematical ideas to others Exams, lectures, instructor activities Whole-class or small group work Active verbal interaction with other students Repeated practice, remembering Rigorous reasoning, flexibility in solving Process view; revealing mathematical ideas Product iew; recall and confirming conjectures Finding one’s own way to think & solve problems Seeking help, actively sharing with others Planning, organizing, reviewing one’s own work 7 7 7 4 4 2 Q3: 7-10 Q3: 1,3,4,6 Q3: 2,5 A3-13 0. 791 0. 724 0. 783 0. 828 0. 744 0. 810 Beliefs about learning Instructor-driven Group work Exchange of ideas 7 7 7 4 3 3 Q5: 1,6,7,8 Q5: 2,3,5 Q5: 9,10, 11 Q6: 2,6 Q6: 1,5,7, 8,9 Q8: 2,6,7,8 Q8: 1,3,5 0. 642 0. 685 0. 731 0. 667 0. 719 0. 745 Beliefs about problem-solving Practice Reasoning 7 7 2 5 0. 90 0. 734 0. 758 0. 712 Beliefs about proofs (Yoo & Smith, 2007) Constructive Confirming Strategies Independent Collaborative Self-regulatory 7 7 7 4 3 6 Q4: 5,9,11, 12 Q4: 2,4,14 Q4: 1,3,6, 7,8,10 0. 747 0. 774 0. 747 0. 775 0. 813 0. 747 7 7 4 3 0. 637 0. 692 0. 675 0. 672 A3. 6. 2 Learning gains survey variables Similar to the treatment of attitudinal variables, composite variables were constructed on the basis of the questions and structures in the SALG-M survey.

Exploratory and principal component analyses and item analysis produced five measures of instructional practices and nine measures of learning gains (see Table A3. 6). The five composite variables related to instructional practices were used in reporting results on students’ course experiences. Results on learning gains from the nine composite variables represented students’: Appendix A3: Survey Methods • • • • A3-14

Cognitive gains: mathematical concepts, mathematical thinking, application of mathematical knowledge, Affective gains: positive attitude, confidence, persistence, Social gains: collaboration, comfort in teaching mathematics, Independence in learning mathematics. Table A3. 6: Composite Variables Measuring Student Experiences and Learning Gains Variable Description Scale Count Experience of course practices (what helped me learn) Overall Teaching approach, atmosphere, pace, workload 5 5 5 5 5 7 5 4 8 6 Q1: 1-7 Q2: 3-7 Q2: 2,8,9 Q4: 1-8 Q5: 1-6 0. 898 0. 839 0. 695 0. 764 0. 96 Items Numbers Reliability (Cronbach) Active participation Personal engagement in discussion & group work Individual work Assignments Personal interactions Math concepts Math thinking Application Studying & problem-solving on one’s own Nature of tests, homework, other assigned tasks Interaction with peers & instructor, in/out of class Understanding concepts Understanding how mathematicians think Applying ideas elsewhere, understanding others’ ideas Appreciation of math Confidence to do math Persistence, stretching Working with others, seeking help Comfort in teaching math Work/organize on own

Learning gains: Cognitive gains 5 5 5 2 2 3 Q6: 1,2 Q6: 3,4 Q6: 5,6,7 0. 921 0. 819 0. 629 Learning gains: Affective gains Positive attitude Confidence Persistence Collaboration Teaching Learning gains: Independence 5 5 5 5 5 5 2 4 2 3 1 2 Q8: 3,6 Q8: 1,2,7,8 Q8: 9,14 Q8: 10,12,13 Q8: 11 Q8: 4,5 0. 821 0. 905 0. 852 0. 841 0. 806 Learning gains: Social gains Appendix A3: Survey Methods A3-15 We also report results on students’ cognitive, affective and social gains as three main areas of learning gains.

Gain in independence in learning mathematics represented a measure that is distinct from the other three main areas. Table A3. 6 displays the titles and descriptions of the learning-related composite variables, the survey items that comprise each variable, and the reliability scores for each composite variable. A3. 6. 3 Analysis methods for structured survey questions All survey data was entered by student technicians and analyzed using the SPSS computer software package. Statistical analyses included descriptive statistics of each composite variable and background variable.

Correlation analysis was used to study relationships between composite variables and their relation to background information on students’ overall college GPA and expected grade at the beginning and end of their course. Parametric (independent and pair-wise T-tests, ANOVA) or non-parametric (Chi-square, Mann-Whitney, Kruskas-Wallis) tests were used to explore group differences in students’ attitudes, experiences, and learning gains, sorted by demographic information on students’ gender, ethnicity, race, academic status, and college major.

The most important of these analyses focused on differences between IBL and non-IBL students, and between math-track students and pre-service teachers. Analysis of covariance (ANCOVA) was used to check intermediate effects (GPA, expected grade, gender) on students’ learning gains. Stepwise regression analysis was applied to examine the variation in students’ learning gains versus changes in their attitudes and self-reported class experiences. A3. 6. 4 Analysis of open-ended survey questions

The open-ended survey questions asked about students’ gains or changes in their understanding of mathematics, remembering key ideas, ways to learn mathematics, and other things they carry with themselves from a math course. Most of students’ written comments addressed reports of learning gains from a course or possible difficulties or negative experiences from a course. To analyze the written responses, we applied the same categories that were constructed for analyzing student interview data.

The preliminary categories were fleshed out with more detailed descriptions and subdivided into several subcategories of learning gains and processes using inductive content analysis (Miles & Huberman, 1994; Strauss & Corbin, 1990). As each statement was examined, the detected gains were classified into one of the preliminary categories or a new category creating during reading and analysis. Table A3. 7 summarizes the final coding scheme for learning gains reported in the open-ended answers, and the frequency with which each was reported. Appendix A3: Survey Methods A3-16 Table A3. 7.

Counts for Reported Learning Gains from Open-Ended Survey Comments Main category Subcategory Description Cognitive gains Subtotal Better recall Better knowledge, deeper understanding of mathematical concepts and ideas Thinking and problem solving skills Transferable mathematical knowledge Transferable thinking skills Did not gain cognitively Affective gains Subtotal Positive attitude towards mathematics Confidence to do math, solve math problems, and be a mathematician Less confidence, no gain in confidence Enjoyment, liking math Negative experience, liking less Interest and motivation Less interested Changes in learning Subtotal Beliefs about learning math, deeper learning, problem solving, creativity and discovery, finding own style of learning math Independence in mathematical thinking, learning or problem solving Persistence Work ethic, learned to study hard Metacognition, self-reflection Appreciation others’ thinking, learning from others No change in learning math Number of students reporting each gain once 455 82 120 119 23 20 91 158 16 38 5 28 62 8 1 428 133 93 17 21 17 66 81 2-3 times 115 2 54 42 3 1 13 24 1 8 2 13 63 33 14 1 14 1 ? 4 times 5 4 1 1 1 4 3 1 – Appendix A3: Survey Methods Table A3. 7, continued… Gains in communication skills Subtotal Speaking or presenting Writing athematics Collaboration, group work Teaching others, explaining to others Giving or receiving critique Changes in understanding the nature of mathematics Subtotal How knowledge is built; how research math is done Change in conceptualization of math No change in concepts of the nature of math Total 116 14 23 29 46 4 78 14 59 5 1235 9 4 3 2 20 20 231 A3-17 10 In all, 544 students wrote in at least one gain in cognition, affect, communication, ways of learning mathematics, and/or understanding the nature of mathematics. They reported one to as many as nine gains each. In all, 197 students reported 1-6 times each that they did not make gains or undergo changes in cognition, affect, communication, ways of learning mathematics, or understanding the nature of mathematics.

The rest of survey respondents wrote no comments. A3. 7 Reliability and validity Most of the pre- and post-surveys were administered and gathered in mathematics classes by the project coordinators of each campus, or by instructors. The surveys were completed in class, which strengthened the response rate. The coordinators were given written instructions for administering the surveys that were intended to ensure that students had enough time to answer the surveys and that their anonymity was preserved. Completed surveys were delivered to the research team by the campus coordinators, entered by trained project assistants into separate SPSS files, and checked and analyzed by the researchers.

We will describe features of our survey data from the attitudinal survey and the learning gains survey separately. A3. 7. 1 Attitudinal survey data After testing the structures and items in the attitudinal survey with a small group of students, we revised the instrument accordingly. In the full data collection, we gathered a large number of completed pre-and post-surveys, which ensured high statistical power in our results. Missing answers both in the structured attitudinal and gain survey items were rare. Even though the postsurvey was rather long, most students responded to all the structured items. The number of missing answers on the items varied between 0-49 for the main pre-survey items and 0-37 for the main attitudinal post-survey items.

These low numbers indicate that students understood the Appendix A3: Survey Methods A3-18 questions and statements in the items. All these features strengthen the reliability and validity of our attitudinal survey results. Among the combined survey responses, some students did not report specific demographic information on their: • • • • • • • • gender (49), race (148) or ethnicity (74), academic major (79), academic status (46), number of prior college mathematics courses (53) AP test score (430; many students did not take the AP test), prior GPA (291; many first year students had no prior GPA), expected grade at the beginning of a course (86).

The most common type of missing information was self-reported AP test scores and prior GPA. Thus the results reported on group differences by prior GPA largely excluded first-year students. We used AP test scores in our study of group differences only in analyzing LMT scores for preservice teachers. The number of responses about students’ beliefs on mathematical proofs (N=877) and about confidence (N=672) were somewhat smaller than those on other topics. Only students with prior experience on proofs were asked to answer proof-related survey questions. A subsection on confidence was added to the attitudinal questionnaire after some students had completed it.

For both sections, the lower number of answers implies slightly lower statistical power in comparison to the results on other survey questions. Descriptive statistics for the composite attitudinal variables showed variation among students. Responses for the items varied between the minimum 1 and maximum 7 for most items. The minimum for only one composite variable was above 2 (Reasoning 2. 2 on the post-survey). However, standard deviations for the pre-survey variables varied between 0. 88 and 2. 56, and for the post-survey variables between 0. 86 and 2. 63, indicating low or moderate variation among students in their attitudes. Cronbach’s alphas were used to study the reliabilities of the subsections and composite variables for both attitudinal and SALG-M survey instruments.

The final reliability scores for attitudinal composite variables are presented in Table A3. 5, for pre-survey and post-survey data separately. Reliability scores for the pre-survey varied between 0. 439 and 0. 914 and between 0. 615 and 0. 928 for the post-survey. Only five composite variables had a low reliability score (below 0. 7) in the pre-/post-survey data: math future interest, teaching confidence, instructor-driven beliefs, and beliefs about proofs. Appendix A3: Survey Methods A3-19 Correlational analysis showed that the composite variables had good construct validity, meaning they produced real results on students’ motivation, beliefs and strategies.

For example, on the post-survey data, both composite variables related to motivation (personal interest, math future) correlated highly positively with each other (r= 0. 417) but also with the variables on intrinsic learning goals (r=0. 552, r=0. 354) and enjoyment of learning mathematics (r=0. 718, r=0. 413). Confidence in mathematics (r=0. 507) and teaching (r=0. 309) correlated highly positively with enjoyment of mathematics. Math confidence also correlated clearly with the use of independent strategies (r=0. 445) and beliefs about math problem-solving as reasoning (r=0. 531). Beliefs that mathematics learning is an instructor-driven activity correlated highly positively with extrinsic learning goals (r=0. 54), beliefs about problem-solving as practice (r=0. 499), and beliefs about proving as confirming the truth (r=0. 359). In contrast, beliefs about math problemsolving as reasoning correlated highly positively with enjoyment of mathematics (r=0. 512) and intrinsic learning goal (r=0. 539). Moreover, use of independent strategies correlated highly positively with use of self-regulatory strategies (r=0. 606), whereas use of collaboration correlated positively with communicating goal (r=0. 289) and beliefs about mathematics learning as group work (r=0. 459) or exchange of ideas with other students (r=0. 422). A3. 7. 2 Learning gains survey data

We collected even larger numbers of responses on the SALG-M items (N=1074-1127) than on the attitudinal pre-/post-survey measures. This ensured high statistical power for results on students’ self-reported class experiences and learning gains. The numbers of analyzed responses on the SALG-M items was lower than in the attitudinal data because students could choose the response “Not applicable. ” Open-ended survey questions on the SALG-M were optional, and students commonly chose not to respond to these. Overall, 13 to 38% of IBL math-track students and 12 to 20% of non-IBL math-track students chose to write in a comment about their learning gains (depending on the question). These low percentages are typical for open-ended survey questions.

The whole scale between 1 and 5 was used by students in their answers to questions about both course experiences and learning gains. However, descriptive statistics showed a rather strong “halo” effect. The median for all the course experience variables (except Assignments other than tests) was above 3. 8, and for the learning gains variables above 3. 5 (except Application). The standard deviations ranged between 0. 82 and 0. 95 for the course experience variables and between 0. 93 and 1. 16 for the learning gains variables. These scores indicate rather low variation in students’ answers to the survey items. The reliability scores for the composite variables from the SALG-M instrument are presented in Table A3. 6.

The scores indicated even higher reliability and internal consistency than for the attitudinal variables. For the course experience composite variables, the reliability scores varied between 0. 696 and 0. 898. Reliability scores for the composite learning gains variables similarly varied between 0. 629 and 0. 921. Only one composite variable (gains in application) had a reliability score less than 0. 7. Appendix A3: Survey Methods A3-20 The SALG survey instruments are based on extensive research that support the validity of selfreport in situations where students have the ability to provide accurate information (Wentland & Smith, 1993) and where they ave few or no obvious reasons (such as adverse consequences or social embarrassment) for providing inaccurate information (Aaker, Kumar & Day, 1998). The SALG survey instruments meet standards of good validation. The developers of the instrument confirmed that most survey items functioned adequately and that item composites formed reliable subscales (Weston, Seymour & Thiry, 2006; Weston, Seymour, Lottridge & Thiry, 2006). Moreover, latent factors underlying items conformed to the hypothesized structures of the survey. We adjusted the original SALG survey instrument to fit college mathematics learning situations and refer to the result as the SALG-M.

The constructed composite variables represented the underlying structures of the SALG instrument and factors specific for the SALG-M survey. These factors were checked for construct validity by correlational analysis. For example, Active Participation, a variable denoting participation in class discussions and group work, was most strongly (positively) related (r=0. 634) to the measure of Personal Interaction (see Table A3. 6). In turn, Individual Work, measuring students’ studying and problem solving on their own, was the most strongly positively related to Assignment (r=0. 557), in particular to Assignments other than regular tests (r=0. 529).

Moreover, the Overall measure of course experience correlated highly positively (0. 505-0. 680) with all the other composite variables on course experiences. All students reported rather positive course experiences and high learning gains as result of a course. Correlational analysis further showed strong positive linkages between the learning gains variables (0. 403-0. 740) and between course experiences and learning gains (0. 314-0. 680). These results confirm construct validity of the results and the fact that positive class experiences were related to higher reported learning gains. On the other hand, students who reported positive experience in one area also did so on the other measured class experiences.

Moreover, students who reported high learning gains in one area also did so on other measures of learning gains. This again reflects the halo effect that makes it more difficult to find real differences in students’ class experiences and learning gains. A3. 7. 3 Connections between the surveys and other measures of learning gains We checked connections between the composite variables based on the structured SALG-M survey questions and numerical variables related to the open-ended answers. Correlations between the answers to structured and open-ended questions showed that higher self-assessed cognitive, affective and social learning gains were all clearly positively related to a higher number of gains each student wrote in.

This was valid both for the total count of gains reported in written comments (r=0. 133 to 0. 239) and for the separate counts of cognitive gains, affective gains and changes in students’ ways of learning math. In particular, higher self-assessed gains in math concepts and thinking were clearly positively connected (r=0. 209**) to a greater number of cognitive gains written in response to the open-ended questions. Moreover, higher self-assessed gains in collaboration were clearly positively connected (r=0. 209**) to a greater number of gains Appendix A3: Survey Methods A3-21 in ways of learning mathematics as reflected in written comments. These indicate reliability and construct validity of the results on learning gains.

We also checked how changes in attitudes related to learning gains as measured by the SALG-M instrument. The correlations (see Table A3. 8) show that students who reported higher cognitive, affective, and social learning gains also showed increases in many of the attitudinal variables. This applied particularly to enhanced motivation, enjoyment, math confidence, and intrinsic and communicating goals during a math course. Pre/post increases both in enjoyment and math confidence were clearly positively related to reported affective gains in confidence, positive attitude, and persistence. This displays construct validity of these affective measures. Table A3. : Statistically Significant Correlations between Changes in Beliefs, Motivation and Strategies and Learning Gains Correlation with Learning Gains Attitudinal Variable Math concepts & thinking interest math major math future teaching Goals Enjoyment Confidence Beliefs about learning Beliefs about problem-solving Strategies math ability group work exchange of ideas reasoning practice independent collaborative self-regulatory ** p< 0. 01 Application Affective Social Motivation 0. 186** 0. 149** 0. 115** 0. 118** 0. 170** 0. 121** 0. 228** 0. 249** 0. 142** 0. 154** 0. 205** 0. 205** 0. 134** 0. 204** 0. 140** 0. 125** 0. 199** 0. 158** 0. 107** 0. 105** 0. 143** intrinsic communicating 0. 104** 0. 204** 0. 233** 0. 121** 0. 150** 0. 155* 0. 155** 0. 129** 0. 138** 0. 182** 0. 132** 0. 66** 0. 245** 0. 145** 0. 166** 0. 186** 0. 186** 0. 121** 0. 204** 0. 156** 0. 194** 0. 166** 0. 195** 0. 144** 0. 189** 0. 189** 0. 164** 0. 188** 0. 183** Similarly, gains in mathematical concepts and thinking were positively related to increases in most of the attitudinal variables. The strongest positive relation was to increased belief in reasoning in solving math problems, but also to enhanced enjoyment and math confidence and to Appendix A3: Survey Methods A3-22 increased use of self-regulatory strategies. Positive relationships between reported gains in application and attitudinal variables showed similar but somewhat weaker correlations.

All these positive relations display good construct validity for the variables involved. Moreover, students with higher reported learning gains developed strengthened beliefs in the value of group work and exchange of ideas with other students that generally contribute to learning. Increased belief in reasoning as a way to solve math problems again enhances mathematics learning and problem solving. Moreover, students’ increased use of independent and self-regulatory strategies in learning were clearly positively related to their learning gains. The use of these strategies generally enhances learning. In particular, gains in collaboration were positively related to increased use of collaborative learning strategies.

Correlational analysis (Spearman) between the learning gains composite variables and other measures of learning outcomes indicated low to moderate connections. Self-reported cognitive, affective and social learning gains from a mathematics course did not correlate with self-reported GPA level at the beginning of a course. But grades generally measure student performance, which is not necessarily related to learning in any given course. In addition, our GPA measure excluded first-year students who could not report their prior GPA at the beginning of a course but who nonetheless reported higher gains than older students. Furthermore, our results on learning gains indicated higher learning gains among students with lower prior GPA.

These features are also perhaps reflected in the low correlations between learning gains and GPA level at the beginning of a course. We also checked the correlations (Spearman) between self-reported learning gains and selfreported AP test score, for the smaller number of students who reported an AP score. However, the correlations indicated only a weak positive relation to gains in math concepts and thinking (r=0. 106*). Again, AP test score is a measure of past mathematics performance, but does not determine learning in the present course. However, correlations between learning gains and expected grade in the course were somewhat stronger, especially to the expected grade reported at the end of the course. These correlations varied between 0. 10** and 0. 307**.

In particular, gains in mathematical concepts and thinking clearly correlated with expected grade at the end of a course (r=0. 238**). The positive correlation of expected grade was even stronger to affective learning gains (r=0. 307**) but weaker to social gains in collaboration (r=0. 10**). These correlations display clear but moderate connections between students’ self-reported learning gains and their assessment of the quality of their learning during a college math course. A3. 8 References Cited Aaker, D. A. , Kumar, V. , & Day, G. S. (1998). Marketing research. New York: Wiley. Ames, G. , & Archer, J. (1988). Achievement goals in the classroom: Students? earning strategies and motivation processes. Journal of Educational Psychology, 80, 260-267. Appendix A3: Survey Methods A3-23 Bandura, A. (1993). Perceived self-efficacy in cognitive development and functioning. Educational Psychologist, 28, 117-148. Buch, N. J. , & Wolff, T. F. (2000). Classroom teaching through inquiry. Journal of Professional Issues in Engineering Education and Practice, (July),105-109. Burn, B. , Appleby, J. , & Maher, P. (Eds. ) (1998). Teaching undergraduate mathematics. London: Imperial College Press. De Corte, E. , Verschaffel, L. , & Eynde, P. O. (2000). Self-regulation: A characteristic and a goal of mathematics education. In M.

Boekaerts, P. R. Pintrich, & M. Zeidner (Eds. ), Handbook of self-regulation (pp. 687-726). San Diego: Academic Press. Duch, B. J. , Groh, S. E. , & Allen, D. E. (2001). Why problem-based learning? In B. J. Duch, S. E. Groh, & D. E. Allen (Eds. ), The power of problem-based learning (pp. 3-11). Sterling, VA: Stylus Publishing. Entwistle, N. (1988). Motivational factors in students’ approaches to learning. In R. R. Schmeck (Ed. ), Learning strategies and learning styles (pp. 21-51). New York: Plenum. Frost, L. A. , Hyde, J. S. , and Fennema, E. (1994). Gender, mathematics performance, and mathematics-related attitudes and affect: A meta-analytic synthesis.

International Journal of Educational Research, 21, 373-386. Gillies, R. M. (2007). Cooperative learning: Integrating theory and practice. Thousand Oaks, CA: Sage Publications. Goldin, G. A. (2000). Affective pathways and representations in mathematical problem solving. Mathematical Thinking and Learning, 17, 209-219. Hanna, G. (1991). Mathematical proof. In D. Tall (Ed. ), Advanced mathematical thinking (pp. 54-61). Norwell, MA: Kluwer. Ju, M. -K. ; Kwon, O. N. (2007). Ways of talking and ways of positioning: Students’ beliefs in an inquiry-oriented differential equation class. Journal of Mathematical Behavior, 26, 267280. Knuth, E. J. (2002).

Secondary school mathematics teachers’ conceptions of proof. Journal for Research in Mathematics Education, 33, 379-405. Kroll, D. L. , ; Miller, T. (1993). Insights from research on mathematical problem solving in the middle grades. In D. T. Owens (Ed. ), Research ideas for the classroom: Middle grades mathematics (pp. 58-77). NY: Macmillan. Leder, G. C. , Pehkonen, E. , ; Torner, G. (Eds. ). (2002). Beliefs: A hidden variable in mathematics education? Dordrecht: Kluwer. Appendix A3: Survey Methods A3-24 Malmivuori, M. L. (2001). The dynamics of affect, cognition, and social environment in the regulation of personal learning processes: The case of mathematics. Research report 172.

Helsinki, Finland: Helsinki University Press. Malmivuori, M. -L. (2007). Understanding student affect in learning mathematics. In C. L. Petroselli (Ed. ), Science Education: Issues and Developments (pp. 125-149). New York: Nova Science. Marton, F. , Hounsell, D. , ; Entwistle, N. (1997). The experience of learning. Edinburgh: Scottish Academic Press. McLeod, D. B. (1992). Research on affect in mathematics education: A reconceptualization. In D. A. Grouws (Ed. ), Handbook of research on mathematics teaching and learning. 575-596. New York: Macmillan. Miles, M. B. , ; Huberman, A. M. (1994). Qualitative data analysis. California: Sage. Pintrich, P. R. (2000).

Multiple goals, multiple pathways: The role of goal orientations in learning and achievement. Journal of Educational Psychology, 92, 544-555. Pollatsek, H. , Barker, W. , Bressoud, D. , Epp, S. , Ganter, S. , ; Haver, B. (2004). Undergraduate Programs and Courses in the Mathematical Sciences: CUPM Curriculum Guide 2004. A report by the Committee on the Undergraduate Program in Mathematics of The Mathematical Association of America. MAA, Washington DC: The Mathematical Association of America. www. maa. org/cupm/ Prince, M. , ; Felder, R. (2007). The many facets of inductive teaching and learning. Journal of College Science Teaching, 36(5), 14-20. Rasmussen, C. , ; Kwon, O. (2007).

An inquiry oriented approach to undergraduate mathematics. Journal of Mathematical Behavior, 26, 189-194 Recommendations for using the SALG (2008). Recommendations for using SALG. Accessed 4/25/2011 at http://www. salgsite. org/docs/SALG Rationale 11 15 08. doc Schiefele, U. (1991). Interest, learning, and motivation. Educational Psychologist, 26(3), 299323. Schoenfeld, A. (1985). Mathematical problem solving. New York: Academic Press. Schoenfeld, A. H. (1992). Learning to think mathematically: Problem solving, metacognition, and sense making in mathematics. In D. A. Grouws (Ed. ), Handbook of research on mathematics teaching and learning (pp. 34-370). New York: Macmillan. Seegers, G. , ; Boekaerts, M. (1996). Gender-related differences in self-referenced cognitions in relation to mathematics. Journal for Research in Mathematics Education, 27, 187-215. Selden, A. , ; Selden, J. (2007). Overcoming students’ difficulties in learning to understand and construct proofs. Technical Report 1. Cookeville, TN: Tennessee Technological University. Appendix A3: Survey Methods A3-25 Seymour, E. , Wiese, D. , Hunter, A. , ; Daffinrud, S. M. (2000). Using real-world questions to promote active learning. National Meeting of the American Chemical Society Symposium, March 27, 2000, San Francisco, CA.

Smith, J. C. (2006). A sense making approach to proof: Strategies of students in traditional and problem-based number theory courses. Journal of Mathematical Behavior, 25, 73-90. Snow, R. E. , Corno, L. , ; Jackson, D. (1996). Individual differences in affective and conative functions. In D. C. Berliner ; R. C. Calfee (Eds. ), Handbook of educational psychology (pp. 243-292). New York: MacMillan. Sowder, L. , ; Harel, G. (2003). Case studies of mathematics majors’ proof understanding, production, and appreciation. Canadian Journal of Science, Mathematics and Technology Education, 3(2), 251-267. Steffe, L. P. , ; Thompson, P. W. (Eds. )(2000).

Radical constructivism in action: Building on the pioneering work of Ernst von Glasersfeld. Studies in Mathematics Education Series, 15. London: Routledge Falmer. Stipek, D. (2002). Good instruction is motivating. In A. Wigfield ; J. S. Eccles (Eds. ). Development of achievement motivation (pp. 309-332). San Diego: Academic Press. Strauss, A. , ; Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park, CA: Sage Publications. Tall, D. (1991). The psychology of advanced mathematical thinking. In D. Tall (Ed. ), Advanced mathematical thinking (pp. 3-21). Norwell, MA: Kluwer. Wentland, E. J. , ; Smith, K. W. 1993). Survey responses: An evaluation of their validity. New York: Academic Press. Weston, T. , Seymour, E. , Lottridge, S. , ; Thiry, H. (2006). The validity of SENCER-SALG online survey for course revision and assessment of student outcomes. Unpublished manuscript. Weston, T. , Seymour, E. , ; Thiry, H. (2006). Evaluation of Science Education for New Civic Engagements and Responsibilities (SENCER) project. [Report to SENCER/NCSCE] Boulder, CO: ATLAS. Yoo, S. , ; Smith, J. C. (2007). Differences between mathematics majors’ views of mathematical proof after lecture-based and problem-based instruction. In Lamberg, T. , ; Wiest, L. R. (Eds. ).

Proceedings of the 29th annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education (pp. 596-598). Stateline (Lake Tahoe), NV: University of Nevada, Reno. Zimmerman, B. (2000). Self-efficacy: An essential motive to learn. Contemporary Educational Psychology, 25, 82-91. Exhibit E 1: A t t di i t E3. 1: Attitudinal Pre-Survey r r Dear student, Our research team is studying methods of improving teaching and learning in college mathematics courses, including the methods used in the course you are taking now. Because you are enrolled in a college math course, we would like to know about your own experiences in learning mathematics.

This survey asks about your views about mathematics, your strategies for learning math, and your personal reasons for studying mathematics. Your participation is voluntary. You may skip questions you do not wish to answer, or choose not to participate. Your answers are anonymous and will not be reported in any way that may identify you individually; they will be aggregated with responses by other students from your course and other courses. Your instructor will not know how you answered. By completing this survey, in part or in whole, you agree that we may use this data to understand and improve the quality and effectiveness of mathematics instruction. Please, mark clearly the best answer to each question.

You do NOT need to fill in the bubble completely. Thank you for your candid responses! Please contact us with any questions. Sandra Laursen, study director Marja-Liisa Hassi, research associate Ethnography ; Evaluation Research University of Colorado at Boulder sandra. laursen@colorado. edu Other hassi@colorado. edu Your interest in mathematics 1. HOW LIKELY is it that you will… Not at all likely Take additional math courses after this course? Graduate with a college math major? Graduate with a college math minor? Study hard for a college math course? Read magazine or newspaper articles related to math? Bring up mathematical ideas in a non-mathematical conversation?

Participate in a club or organization related to math? Teach math in the future? Extremely likely Exhibit E 1: A t t di i t E3. 1: Attitudinal Pre-Survey r r Your enjoyment of mathematics 2. HOW MUCH do you ENJOY… No enjoyment Working on a challenging mathematical problem? Discovering a new mathematical idea? Seeing mathematics in everyday life? Perceiving beauty in mathematical ideas? Using rigorous reasoning in a math problem? Thinking about abstract concepts? Teaching mathematics to other people? Extreme enjoyment Your goals in studying mathematics 3. Below are some goals that students may have in studying mathematics. HOW IMPORTANT is each goal for YOU?

Not at all important Learning specific procedures for solving math problems Improving your ability to communicate mathematical ideas to others Getting a good grade in college mathematics courses Memorizing the sets of facts important for doing mathematics Making mathematics understandable for other people Meeting the requirements for your degree Learning to construct convincing mathematical arguments Using mathematics as a tool to study other fields Learning new ways of thinking Applying mathematical thinking outside the university context Other goals (please specify) Extremely important 5 6 Exhibit E 1: A t t di i t E3. 1: Attitudinal Pre-Survey r r Your strategies for learning mathematics 4. When you DO MATH, how often do you take each action listed below? Very seldom Study on your own. Brainstorm with other students. Try to organize or summarize your own ideas. Share problem-solving strategies with other students.

Find your own ways of thinking and understanding. Review your work for mistakes or misconceptions. Read the assigned readings. Plan a solving strategy before attacking a problem. Try to find your own way to solve a problem. Check your understanding of what the problem is asking. Use your intuition about what the answer should be. Look for an alternate strategy to solve the problem. Give up when you get stuck. Ask another student for help. Ask the instructor or TA for help. Very often Your preferences for learning mathematics 5. Indicate how much you agree or disagree: I learn mathematics BEST when… Strongly disagree The instructor lectures. The class critiques other students’ solutions.

I work on problems in a small group. The exams let me prove my mathematical skills. Groups present their solutions in class. The instructor explains the solutions to problems. The homework assignments are similar to the examples considered in class. I study my class notes. I can compare my math knowledge with other students. I explain ideas to other students. I get frequent feedback on my mathematical thinking. Strongly agree Exhibit E 1: A t t di i t E3. 1: Attitudinal Pre-Survey r r 6. Indicate how much you agree or disagree: In order to solve a challenging math problem, I NEED… Not at all To carefully analyze different possible solutions.

To have lots of practice in solving similar problems. To understand other students’ mathematical thinking. To have natural talent for mathematics. To try multiple approaches to constructing a solution. To remember a lot of examples that I might use in constructing a solution. To use rigorous reasoning. To have freedom to do the problem in my own way. To work hard Very much Your experience and views about mathematical proof 7. Have you had math classes that included mathematical proofs? yes no (Go directly to question 9) 8. The following statements reflect some students’ views about mathematical proof. How much do you AGREE or DISAGREE with each statement?

Strongly disagree The main purpose of proof is to confirm the truth of a mathematical result that is already known to be true. Proof is a tool for understanding mathematical ideas. Doing proofs well requires good recall of previous proofs of similar statements. The main purpose of proof is to explain why a certain statement is true. In math class, doing proofs means confirming conjectures that have been previously proven by an expert. There are several different ways to prove a mathematical statement. When evaluating a proof, the most important thing to look at is its logical structure. A proof is something you have to construct based on your own understanding. Strongly agree Exhibit E 1: A t t di i t E3. 1: Attitudinal Pre-Survey r r

Your confidence in doing math 9. HOW CONFIDENT are you that you can… Not at all confident Get a high grade in this course? Successfully work with complex mathematical ideas? Teach mathematics to high school students? Develop new mathematical ideas? Apply a variety of perspectives in solving problems? Present your work at the board in a math class? Work on math problems with other students? Teach math to children? Extremely confident Your math background 10. What was the highest level of math that you took in HIGH SCHOOL? Algebra, one year Algebra, two years Geometry with an algebra prerequisite Pre-calculus or trigonometry Calculus Other (please specify) 11.

Did you take the AP Calculus test? Yes No (go directly to question 14) 12. Which of the AP Calculus tests did your take? A/B B/C 13. What was your score in the AP Calculus test? 1 2 3 4 5 Exhibit E 1: A t t di i t E3. 1: Attitudinal Pre-Survey r r 14. How many COLLEGE math courses have you taken prior to this course? Please count the total number of semesters or quarters. 0 1 2 3 4 5 6 7 or more 15. What grade do you expect to receive in this course? A AB+ B BC+ C CD F Your academic background 16. What is your overall UNDERGRADUATE GPA? (estimated) 3. 8 or higher 3. 5 – 3. 79 3. 0 – 3. 49 2. 5 – 2. 99 2. 0 – 2. 49 below 2. 0 Not applicable 17. What is your class year?

First-year Sophomore Junior Senior Graduate student Other (please specify) Exhibit E 1: A t t di i t E3. 1: Attitudinal Pre-Survey r r Your academic interests 18. What is your college major? (Check ALL that apply) Math or Applied Math Physics Chemistry Engineering Computer science Other science or technical field Economics Other non-science field 19. Are you pursuing a teaching certification? no yes, elementary (grades K-6 or K-8) yes, secondary math (grades 6-12, 8-12, or 9-12) yes, secondary in a field other than math Other (please specify) Your personal background Our funding agency requires us to gather data on the gender, race and ethnicity of study participants.

Please choose the answers that best apply. 20. What is your gender? male female 21. What is your ethnicity? Hispanic or Latino Not Hispanic or Latino 22. What is your race? (please check ALL that apply) American Indian or Alaskan Native Asian Black or African American Native Hawaiian or other Pacific Islander White Exhibit E 1: A t t di i t E3. 1: Attitudinal Pre-Survey r r Assign yourself an identifier On this page, we ask for some information that will enable us to match your survey responses with those in other surveys. The information will be unique to you but will not identify you individually. * 23. Enter the following data. Please, print neatly.

FIRST two letters of your FIRST NAME Two-digit DAY of your BIRTHDAY (01 through 31) FIRST two letters of your MOTHER’S FIRST NAME FIRST two letters of the TOWN where you were BORN Course information * 24. What is this math course? Number of the course Section of the course Name of the instructor Survey completed Thank you for completing the survey! Your input is important to us, and will help us to help math instructors improve teaching and learning in their courses. If you have any questions, please contact us: Sandra Laursen, project director sandra. laursen@colorado. edu Marja-Liisa Hassi, research associate hassi@colorado. edu Exhibit E 2 Learning Gains Post-Survey ( A xhi t E3. 2: ng ns ost S (SALG-M) Dear student,

Our research team is studying methods of improving teaching and learning in college mathematics courses, including the methods used in the course you are taking now. Because you are enrolled in a college math course, we would like to know about your own experiences in learning mathematics. This survey asks about your experiences in this course. Your participation is voluntary. You may skip questions you do not wish to answer, or choose not to participate. Your answers are anonymous and will not be reported in any way that may identify you individually; they will be aggregated with responses by other students from your course and other courses.

Your instructor will not know how you answered. By completing this survey, in part or in whole, you agree that we may use this data to understand and improve the quality and effectiveness of mathematics instruction. We may compare your responses with your gains from the course, assessed by your instructor. This will be done anonymously by using the identifiers. We will not know your individual grades and your instructor will not know how you answered the questions in this survey. Please, mark clearly the best answer to each question. You do NOT need to fill in the bubble completely. Thank you for your candid responses! Please contact us with any questions.

Sandra Laursen, study director Marja-Liisa Hassi, research associate Ethnography ; Evaluation Research/University of Colorado at Boulder sandra. laursen@colorado. edu hassi@colorado. edu The course as a whole 1. HOW MUCH did the following aspects of the class HELP YOUR LEARNING? No help The overall approach to teaching and learning in the course How class topics, activities, ; assignments fit together The pace of the class The workload of the class The general atmosphere of the class The course material The mental stretch required of you The information you were given about the class when it began Other (please specify) A little help Moderate help Much help Great help NOT APPLICABLE 5 6 Exhibit E 2 Learning Gains Post-Survey ( A xhi t E3. 2: ng ns ost S (SALG-M) Class activities 2.

HOW MUCH did the following CLASS activities HELP YOUR LEARNING? No help Listening to lectures Studying on your own Participating in class discussions Participating in group work during class Explaining your work to other students Hearing other students explain their work Giving presentations in front of class Writing solutions to problems Checking solutions to problems Working on a computer Examining children’s mathematical work A little help Moderate help Much help Great help DID NOT HAPPEN 3. Please comment on how this class has CHANGED THE WAYS YOU LEARN mathematics? 5 6 Assignments and tests 4. HOW MUCH did the assignments and tests HELP YOUR LEARNING?

No help Taking tests Doing other assignments Doing homework The fit between class content and tests The match between the grading system and what you needed to work on The mental stretch required on tests Preparing class presentations The feedback you received on your written work A little help Moderate help Much help Great help DID NOT HAPPEN Exhibit E 2 Learning Gains Post-Survey ( A xhi t E3. 2: ng ns ost S (SALG-M) Support for you as a learner 5. HOW MUCH did each of the following HELP YOUR LEARNING? No help Interacting with the instructor DURING class Interacting with the instructor OUTSIDE class Interacting with teaching assistants DURING class Interacting with teaching assistants OUTSIDE class Working with peers DURING class Working with peers OUTSIDE class A little help Moderate help Much help Great help DID NOT HAPPEN Your understanding of class content 6. As a result of your work in this class, what GAINS did you make in your UNDERSTANDING of each of the following?

No gain The main concepts explored in this class The relationships among the main concepts Your own ways of mathematical thinking How mathematicians think and work How ideas from this class relate to ideas outside mathematics How children solve mathematical problems How to make mathematics understandable for other people Please comment on how YOUR UNDERSTANDING OF MATHEMATICS has changed as a result of this class. A little gain Moderate gain Good gain Great gain NOT APPLICABLE 5 6 Exhibit E 2 Learning Gains Post-Survey ( A xhi t E3. 2: ng ns ost S (SALG-M) 7. Please comment on how THE WAY THIS CLASS WAS TAUGHT affects your ability to REMEMBER key ideas. 5 6 Confidence, attitudes and abilities 8. As a result of your work in this class, what GAINS did you make in the following?

No gain Confidence that you can do mathematics Comfort in working with complex mathematical ideas Development of a positive attitude about learning mathematics Ability to work on your own Ability to organize your work and time Appreciation of mathematical thinking Comfort in communicating about mathematics Confidence that you will remember what you have learned in this class Persistence in solving problems Willingness to seek help from others Comfort in teaching mathematics Ability to work well with others Appreciation of different perspectives Ability to stretch your own mathematical capacity A little gain Moderate gain Good gain Great gain NOT APPLICABLE 9. What will you CARRY WITH YOU from this class into other classes or other aspects of your life? 5 6 Exhibit E 2 Learning Gains Post-Survey ( A xhi t E3. 2: ng ns ost S (SALG-M) Your expectation 10. What grade do you expect to receive in this course? A AB+ B BC+ C CD F Your background 11. What is your college major? (Check ALL that apply) Math or Applied Math Physics Chemistry Engineering Computer science Other science or technical field Economics Other non-science field 12. Are you pursuing a teaching certification? o yes, elementary (grades K-6 or K-8) yes, secondary math (grades 6-12, 8-12, or 9-12) yes, secondary in a field other than math Other (please specify) 13. What is your gender? male female Exhibit E 2 Learning Gains Post-Survey ( A xhi t E3. 2: ng ns ost S (SALG-M) 14. What is your class year? First-year Sophomore Junior Senior Graduate student Other (please specify) Assign yourself an identifier On this page, we ask for some information that will enable us to match your survey responses at the beginning and end of your math classes. The information will be unique to you but will not identify you individually. * 15. Enter the following data.

Please, print neatly. FIRST two letters of your FIRST NAME Two-digit DAY of your BIRTHDAY (01 through 31) FIRST two letters of your MOTHER’S FIRST NAME FIRST two letters of TOWN where you were BORN Course information * 16. What is this math course? Number of the course Section of the course Name of the instructor Survey completed Thank you for completing the survey! Your input is important to us, and will help us to help math instructors improve teaching and learning in their courses. If you have any questions, please contact us: Sandra Laursen, project director sandra. laursen@colorado. edu Marja-Liisa Hassi, research associate hassi@colorado. edu