Student Outcomes

Using Student Outcome Assessment Data for Improvement

From: ACBSP Final Document - Condition on Standard 4.3:  Selection and use of information results; explain how the business school or program has used student outcomes assessment data to improve the business program. 

Northeastern State University completed their Higher Learning Commission reaffirmation visit in March 2012. The final HLC report referred to a “culture of assessment.” Specifically, the report stated

 

“The ‘culture of assessment’ was clearly presented as NSU demonstrated its student learning outcomes in the annual OSRHE Assessment Report. Its recent admittance to the HLC’s Assessment Academy and the launch of the Red Balloon initiatives also support NSU’s effort to improve student learning.” (HLC Report, page 17).

 

HLC also recognized the importance of the University Curriculum Committee by stating

“Through the University Curriculum Committee, faculty review and revise the curriculum, student learning outcomes, and review and recommend new academic programs.” (HLC Report, page 18).

The assessment process within the College of Business and Technology has been aided significantly by the university assignment of College Student Learning Outcomes Coordinators. In each college, a Student Learning Outcome Coordinator was assigned and given a course release to manage the assessment process. The Student Outcome Learning Coordinators assist department chairs, faculty, and the dean of the college in implementing an ongoing assessment of student learning programs in their college. Specifically they are assigned to

a.     Assist faculty in all aspects of learning outcomes assessment of student learning: classroom assessment, program assessment, program review, and general education as applicable.

b.     Increase awareness and importance of student learning outcomes.

c.      Assist faculty in developing appropriate student learning outcomes at the course and program level.

d.     Provide information to faculty regarding various methods to assessing student learning.

e.     Interact with and attend meetings of the university Student Learning Committee (formerly Assessment) to improve all assessment of student learning across campus and infuse information from that committee into the fabric of the departments and college.

f.       Monitor and consult regarding the completion of various reports related to assessment of student learning: program assessment report, program review, academic efficiencies, and others. The student learning coordinators will not write the actual reports.

 

Within the College of Business and Technology, the basic model employed for assessment purposes consists of a four-stage process. Each discipline/degree program will

1.     Define learning outcomes for their major and construct a matrix indicating which courses are responsible for delivering/teaching that learning outcome.

2.     Determine what data will be collected to measure each learning outcome.

3.     Collect data.

4.     Evaluate data and implement any needed changes in the outcome/curriculum/assessment process.

For the College of Business and Technology, the development of learning outcomes is actually contingent upon the performance objectives that stakeholders desire in college graduates, i.e., what attributes, knowledge, and skills faculty and potential employers expect from someone with an MBA or BBA degree in a given major. The NSU College of Business and Technology mission statement identifies that the college intends to graduate professionals who are analytical, timely, and effective communicators and leaders with an understanding of the impact of economic, social, political, and global forces on business.

With college-level performance objectives in mind, the College of Business and Technology Core Committee developed a list of learning outcomes that should be attained by undergraduate students by the time of graduation that contribute to these performance objectives. These learning outcomes indicate what specific characteristics the student should exhibit in order to demonstrate achievement/mastery of that criterion. The learning outcomes developed are specific performance criteria that address the performance objectives in the College of Business and Technology Mission statement. In addition to learning objectives for the College of Business and Technology core curriculum, each graduate and undergraduate degree program has developed learning outcomes.

All assessment data collected are then evaluated, and needed changes in curriculum are made to address any shortcomings. The following are examples of changes that have resulted from a study of these data.

According to the Curriculum and Educational Policies Committee (CEPC) minutes from October 2006, the following changes were made to curriculum based on assessment results:

·       A survey was conducted of 115 individuals (most with bachelor’s degrees, 24 percent with post graduate degrees) from the Oklahoma Alliance for Manufacturing, the local chapter of the Association for Operations Management (APICS), and the local chapter of the Institute for Supply Management (ISM). Respondents indicated strong demand for a four-year degree encompassing both traditional business subjects and function-specific training in supply chain management. Not only did the survey show strong growth in supply chain management activities from 2003/2004 to 2006, it indicated that employers prefer to hire individuals with at least bachelor’s degrees. This survey provided the rationale for a new Supply Chain Management Major. The proposal outlining the degree carefully identified the student learning outcomes for this major and the process for evaluation. The outcomes will be evaluated by a variety of assessment instruments to allow for continuous improvement of the curriculum and instructional methodology. Evaluation of program objectives falls into three broad categories: knowledge-based, analyses, and skill-based. Knowledge-based objectives call upon students to either identify key concepts or to analyze particular situations. By their nature, analysis objectives are more subjective and open to interpretation than are identification objectives. Skill-based objectives require students to demonstrate their proficiency. Accordingly, each course in the program will evaluate student achievements utilizing objective, subjective, and demonstration methods applicable to the content of the specific course.

According to the Curriculum and Educational Policies Committee (CEPC) minutes from October 2009, the following changes were made to curriculum based on assessment results:

·       A new course, BADM 3963 Quantitative Methods for Business Decision Making, was added to the business core. This change was recommended after the College of Business and Technology advisory board assessed the BBA core and found the core lacking in quantitative skills presented. The additional course will allow the introduction of management science techniques and more in-depth work on topics introduced in two previous courses.

·       A change in requirements for admission into the BBA Degree Program was passed since assessment results have demonstrated that students are not achieving the desired outcomes in the current BBA program. A review of program trends at other universities and the literature on best practices in undergraduate business programs indicate that the NSU program does not focus sufficiently on “soft” skills such as teamwork, critical thinking, etc. This finding coincides with the assessment from the College of Business and Technology Dean’s Advisory Board.

According to the Curriculum and Educational Policies Committee (CEPC) minutes from February 2011, the following changes were made to curriculum based on assessment results:

·       A new course was added to the BBA Business Administration Major, BADM 4911 Business Administration Capstone. This course was proposed as a means to assess the major and would allow assessment of learning outcomes for the BBA Business Administration program, enhancing continuous improvement efforts for curriculum and instruction. This course will result in a completed Business Administration Senior Portfolio.

The following are examples of changes made in response to assessment data that required no submission to CEPC.

·       The Department of Information Systems and Technology (IS) completed a survey of local/regional companies who had hired or intended to hire a graduate. The survey investigated the personal and IT skills important in their decision. This information was used to shape curriculum and topics covered in IS 3063 Principles of Information Systems, IS 3213 Business Systems Analysis, and IS 4293 Business Database Management Systems, as well as to help determine which "special topic" courses to offer.

·       Ongoing assessment of the supply chain management program has implemented an ongoing assessment by industry stakeholders of curriculum and student intern performance. Examples of changes that have resulted from this assessment include the following:

o   Industry input indicated that the supply chain students lacked a strong focus on negotiation skills and cause and effect of procurement actions. The program curriculum specified that these topics are contained in MKT 3553 Pricing, Purchasing, and Inventory Management and in TECH 4753 Process of Purchasing. The supply chain curriculum team worked to improve negotiation and the effect of procurement aspects of MKT 3553.

o   Industry input also noted that students need a stronger command of data analysis and use of Excel. This was addressed by strengthening coverage of this material in MKT 3553 Pricing, Purchasing and Inventory Management and in MKT 4353 Marketing Logistics.

o   Industry input also desired Enterprise Resource Planning (ERP) to be included in the curriculum. Therefore, a new course in ERP will be proposed.

·       The model of interaction and ongoing assessment with industry stakeholders followed by the supply chain program also resulted in unsolicited letters of support. The following represents a few examples.

o   Nordam, Interiors and Structures Division, writes “As a long-time Supply Chain Management Professional, I cannot stress how important I believe it is to have access to a local Supply Chain Management curriculum. NSU’s program fills a vacancy that has existed for far too long in the Business Administration Colleges of far too many universities”. 

o   Lynnco Supply Chain Solutions, reports that the internship program started at Lynnco Supply Chain Solutions in conjunction with NSU has built a pipeline of talented NSU students to fulfill logistics needs. This program has led to six successful interns with three full time hires. 

o   After suggesting that a stronger emphasis be given to compliance in the Supply Chain Program, American Airlines goes on to say that the Supply Chain program at NSU is doing an excellent job in producing qualified graduates with solid potential. 

·       Student assessment results for MBA 5313 Managerial Accounting indicated that students enter the course with varied educational backgrounds and levels of preparation. The course material requires students to have a basic understanding of the principles of financial accounting, but many do not. Therefore, the following process was put into place:

o   During the second class meeting, all of the students take a multiple choice quiz over the material contained in principles of financial accounting. This quiz comprises 10% of the course grade. If a student scores 75% or above, a grade of 100 for the quiz is recorded; less than 75%, a grade of 0. Students earning a 0 will be allowed to remove the grade of 0 and replace it with a grade of 100 if they enroll in and complete a computer-based tutorial on the principles of financial accounting published by ALEKS Corporation.

Note on Standard 5.5 Faculty Evaluation; Assessment program not linked to faculty evaluation.

Much work was done on the criteria for annual evaluation (pre-tenure), and Post Tenure Evaluation (every 3 years, promotion, and tenure), resulting in new standards. In the new standards, effective classroom teaching is clearly defined to include evidence that expected learning outcomes were achieved on the college and the university level. As a result, assessment is directly tied to faculty evaluation. Further, the Higher Learning Commission (HLC) confirmed the change in their final team report for Northeastern State University dated June 14, 2012. According to the HLC:

“The Professional Portfolio Tenure and Promotion Review section in the Faculty Handbook (Appendix C1) describes the requirements faculty members must meet in the retention, promotion, and tenure process. There are four areas of evaluation and assessment to the process but ‘the primary focus….is teaching’ and ‘…the primary evidence of effective teaching should come from the assessment of learning outcomes/learning gains exhibited by students.’ This primary focus is also highlighted in the NSU Strategic Plan 2009-2014 Goal 1: Provide a quality curricular and co-curricular education in a flexible, student-centered environment. The Professional Portfolio requirements list specific activities and contributions faculty members must demonstrate as part of the process.” (HLC Report, pages 17-18).           

            The HLC also recognized Northeastern State’s use of the Boyer model to emphasize the importance of effective teaching. Again, according to the HLC:

“NSU follows the Boyer Model of effective classroom teaching and the scholarship of teaching. Faculty members must demonstrate the ways in which they adhere to the Boyer Model as they complete their Professional Portfolio and during their evaluation meetings with their college administrators. Mentoring committees have been established in each college to provide support and guidance for non-tenured faculty members to ensure they meet the requirements of the retention, promotion, and tenure process under the Boyer Model. Moreover, the University’s priorities for faculty research and service are based on the Boyer model of the ‘scholarship of teaching and learning,’ in keeping with the primary mission and goals of the institution. The Center for Teaching & Learning conducts an annual campus conference that allows faculty and staff to share research and practical advice in teaching, learning and other activities. In addition, the Research and Sponsored Programs office offers advice and support to faculty in finding funding for particular projects and in writing and submitting grant requests.” (HLC Report, page 18).

Specific details on the procedures for promotion at Northeastern State University are in section 3.362 of the Faculty Handbook and are stated below: (emphasis provided).

To meet the criteria set forth by the Oklahoma Board of Regents as stated in section 3.361 of the Faculty Handbook, all evaluation for tenure and/or promotion shall address whether each candidate has achieved excellence in the following areas:

§ Effective Classroom Teaching.

§ Scholarly Activities.

§ Contribution to the Institution and Profession (university & professional service).

§ Performance of Non-teaching Semi-administrative or Administrative Duties (if applicable).

Northeastern State University is committed to the advancement of knowledge through Teaching, Scholarship, and Service that supports the academy and the community. The primary focus, however, is teaching, as stated in Strategic Plan Goal 1: “Provide a quality curricular and co-curricular education in a flexible student-centered environment.”

The following list of activities is meant to provide examples of the kinds of evidence in the areas of Teaching, Scholarship, Contribution to the Institution and Profession, and Non-Teaching Duties that may be considered for inclusion in a professional file. The list is not meant to be exhaustive and examples may rise to the level of what constitutes scholarship (see Appendix C2 for scholarship defined by NSU deans). There may be other evidence, worthy of inclusion, which is not mentioned on the list. Also, the evidence on the list should not be interpreted as requirements, but as examples of the kinds of evidence candidates, mentoring committees, chairs, and deans may consider when evaluating a candidate’s performance.

A. Effective Classroom Teaching

Teaching includes all work that is intended to advance learning within an engaging, civil environment. The assessment of teaching should be evaluated as objectively as possible and take into account quantitative indicators such as contact hours, number of preparations, number of students, number of advisees, student evaluation ratings, peer/observer ratings, etc. However, the primary evidence of effective teaching should come from the assessment of learning outcomes/learning gains exhibited by students.

1.     Evidence of expected learning outcomes achieved by students and/or other outcomes achieved during the review period may be demonstrated through the following non-exclusive list of sample activities:

a.     Range of Activities: Communication of knowledge and the development of appropriate skills and attitudes are the primary objectives of all faculty members. With the exception of effective presentations (i), the following list is not meant to be required of all faculty, but to indicate the possible activities that faculty members may use to achieve these objectives.

                                                               i.     Effective presentations, whether in lecture, laboratory, studio or other venues.

                                                              ii.     “Active learning” pedagogy, such as use of active-learning techniques and tools to enhance student learning including, but not limited to, collaborative learning, problem-based learning, and student polling; integration of service learning and other community-based learning into courses; direction of laboratory-based student research, supervision of internships and co-op experiences; study-abroad activities.

                                                            iii.     Engaged teaching, course-or curriculum-related teaching/learning activities that involve students with the community in mutually beneficial ways. This includes but is not limit to, service learning and other community-based learning experiences, internships and co-op experiences, and involvement in community-based research or other special projects.

                                                            iv.     Alternative delivery, such as team teaching or co-teaching; development and/or implementation of online courses; off-campus teaching; flexible scheduling and delivery models.

                                                              v.     Academic advising and mentoring, such as helping students plan their academic programs; presenting options for a career in the discipline or selection of a graduate or professional school; advising discipline-related student clubs or associations; sharing professional experience and expertise on an individual basis. 

                                                            vi.     Involvement in special academic programs, such as development and/or implementation of special retention programs/efforts; participation in first-year programs and/or learning communities; development of courses for programs offered using alternative or innovative delivery models.  

                                                           vii.     Other (as appropriate to the discipline, department, or college).

b.     Documentary evidence of effective teaching and learning outcomes may include but is not limited to:

                                                               i.     Honors courses designed, taught, and evaluated.

                                                              ii.     New courses introduced and evaluated.

                                                            iii.     New courses designed, team-taught, and evaluated.

                                                            iv.     On-line or ITV courses created, modified, and evaluated for impact on student learning.

                                                              v.     Seminars created and directed.

                                                            vi.     Workshops created and directed.

                                                           vii.     Webinars created and directed.

                                                         viii.     Student projects directed that are not part of normal teaching duties.

1.     Theses.

2.     Written and oral examination committees.

3.     Student service learning opportunities.

4.     Student clubs supervised

5.     Student research mentoring/sponsorship.

2.     Best practices reflecting contemporary methodologies.

3.     Student course evaluations (summary sheet only), self-reflection, and modification of courses based upon those.

4.     Peer evaluations incorporated into course improvement/modification.

5.     Chair evaluations incorporated into course improvement/modification.

6.     Original materials employed to create an innovative learning environment.

7.     Evidence of effective advising.

a.     Number of advisees.

b.     Time devoted to advising.

c.      Media employed in advising.

d.     Innovative advising approaches.

8.     Methods employed accommodating student diversity.

At the college level, the criteria for promotion and tenure as they pertain to effective teaching evaluation are discussed below (emphasis provided). The performance of non-tenured faculty members is evaluated on teaching only and must achieve a rating of 2 or above. Post tenure review is evaluated every three years. When the review results in a finding that a tenured faculty member’s academic and professional performance is unsatisfactory, the faculty member will be notified of the deficiencies in performance and must be reviewed again within one year. Two consecutive unsatisfactory post-tenure performance evaluations may be grounds for dismissal or suspension.

General Criteria for Evaluations

 

A.     Teaching

To receive an acceptable evaluation in this area of job performance, a requirement for a favorable tenure or promotion decision, the faculty member must achieve a rating of at least (1) for teaching.

Rating (3) - The applicant has an exceptional or unusually high quality of work in this area and demonstrates leadership in teaching that extends beyond the classroom. Measured student performance outcomes are among the best in the professor's area and the professor attracts students to his or her program. To achieve this rating, the candidate meets the standard of Rating 2 for teaching plus shows leadership in promoting effective teaching among colleagues and/or in promoting or building his or her program.

Rating (2) - Quality of teaching is high, but not exceptional. The assessed quality of teaching exceeds the requirements of the department/program. The candidate also demonstrates growing competencies. This can be exhibited by very good teaching evaluations or student learning outcomes and/or a history of innovation or improvement in classroom performance that is deemed to be meritorious by the tenure or promotion committee.

Rating (1) - Quality of teaching is minimally acceptable. Assessed teaching quality meets the expectations and serves the purposes of the department/ program, but shows substantial possibility of improvement. A candidate can demonstrate that they have met this standard by providing convincing evidence of well-preparedness in the classroom, or compelling evidence that demonstrates efforts made to improve teaching performance.

Rating (0) - The faculty member's performance in teaching is generally unsatisfactory.

      The process of teaching evaluation clearly links effective teaching to the assessment process. The following represents examples of actions taken in this area:

      Scenarios 1 and 2 are examples of rewards for good teaching and/or significant improvement:

Scenario 1: The faculty member was characterized in annual evaluations as meticulous and dedicated in class preparation. Student evaluations were high even though average grades tended to be a bit below average. Student comments from his first semester at NSU have included: “well organized”; “good explanation of difficult concepts”; “difficult but fair”; and “interesting”. These evaluations in conjunction with stated learning objectives reflected on the syllabus and testing of those objectives reflect clear evidence of effective classroom teaching. Thus, this faculty member was awarded the 2011 Circle of Excellence award for teaching. This award recognizes outstanding faculty for their outstanding accomplishments and commitment to educating students.

Scenario 2: This faculty member experienced difficulties in the classroom when beginning tenure with NSU as demonstrated by this phrase from the annual evaluation: “A faculty member’s first year at a new institution is a time of transition. This faculty member’s first year at NSU was marked by difficulties in the adjustment to NSU and in the students’ adjustment to the faculty member. However, through diligent work the faculty member was able to overcome these difficulties. The success of the effort was demonstrated by the improved evaluation numbers and by comments received from students.” Also, “As evidence of the effectiveness of the teaching, results from common-component questions from the final exam across courses indicate favorable results. Also, results from ETS test are favorable.”   This faculty member ultimately received tenure and is a highly respected member of the CBT faculty. 

      Scenarios 3 and 4 are examples of poor performance resulting in dismissal:

Scenario 3: This faculty member was warned on numerous occasions of poor teaching technique. The warnings are summarized best by the Dean of the College of Business and Technology when writing the letter to deny tenure. The letter states, “Student concerns continue to surface in a significant number of classes. In a recent graduate course, the professor was forced to change all of the students’ grades as the syllabus was incomplete and the professor was unable to explain the criteria used to calculate grades…..During Summer 2010 the Dean’s Office received 12 written complaints from an undergraduate class about the professor’s perceived inability/unwillingness to communicate with about class assignments. A P-MBA class revolted when the professor tried to change the nature of the course half way through the semester.” After numerous sessions designed to help the professor improve teaching by providing course objectives and a clear path for the student to succeed were unsuccessful, the professor was denied tenure. 

Scenario 4: In this situation, the professor was advised on many occasions concerning his teaching performance. Typical of the concerns is the following: “Teaching evaluations are mixed and there have been complaints from undergraduate students about lack of clear expectations, instructions and timely feedback. While the professor had revised the syllabi and the tone was somewhat better, it was still difficult to read. Based on two visits to the professor’s class, spring 2009, the following observations were made:

·       Students appeared to have difficulty following the lecture and understanding questions that were posed, although a positive atmosphere was maintained and students were comfortable speaking up and trying to answer questions.

·       At one session, a paper that had been given at an academic conference was presented – it did not seem appropriate or relevant to the class. As discussed, the professor did not seem to be in touch with what expectations should be at the undergraduate level. 

      Concerns about teaching goals were expressed in a discussion. In a prediscussion report you stated that the focus was on helping the ‘best and the brightest’. After our discussion, I was under the impression that it was understood that all of our students deserve your best efforts – the idea of a ‘dual track teaching philosophy’ does not support that understanding. Your attitude toward ‘average and mediocre’ students is distressing.” This person was also denied tenure based on the lack of effective classroom teaching.