Meaningful Measures of Effectiveness
Meaningful Measures of Effectiveness. Since the mid-1980s, almost all states have enacted legislation requiring formal performance reporting systems for public higher education institutions. At this same time, accreditors began asking for proof positive of student learning outcomes.
It was called “institutional effectiveness.” Read what the Commission on Colleges of the Southern Association of Colleges and Schools says about institutional effectiveness:
SECTION 7: Institutional Planning and Effectiveness
Effective institutions demonstrate a commitment to principles of continuous improvement, based on a systematic and documented process of assessing institutional performance with respect to mission in all aspects of the institution. An institutional planning and effectiveness process involves all programs, services, and constituencies; is linked to the decision-making process at all levels; and provides a sound basis for budgetary decisions and resource allocations.
1. The institution engages in ongoing, comprehensive, and integrated research-based planning and evaluation processes that (a) focus on institutional quality and effectiveness and (b) incorporate a systematic review of institutional goals and outcomes consistent with its mission. (Institutional Planning)
2. The institution identifies expected outcomes of its administrative support services and demonstrates the extent to which the outcomes are achieved. (Administrative effectiveness)
Many institutions began studying how to implement something now referred to as assessment. Assessment was to occur for each academic and administrative unit in the university.
During this same timeframe, Mississippi College adopted James O. Nichols’ model on university assessment. Nichols, then the Director of University Planning and Institutional Research at the University of Mississippi, is known for his work in university assessment practices.
For more information on this, you may want to read James Nichols’ book entitled Assessment Case Studies: Common Issues in Implementation with Various Campus Approaches to Resolution expounding on ways to help implement the assessment model. The following are excerpts from Mississippi College assessment plans based on Nichols’ model:
ASSESSMENT YEAR: 2016-2017
DEGREE PROGRAM: B.S. in Biological Sciences
PARTICIPATION: Biology Department Undergraduate Faculty (Baldwin, Barlow, Bourassa, Broome, Carmicle, Graves, Haycraft, Hensley, Kazery, Norcross, Piletz, Reagan, Reiken, Stark, and Tullos)
| UNIVERSITY MISSION: The Department of Biological Sciences values the integration of faith and learning through teaching contemporary biology from the perspective of faculty members who are Christian in belief and conduct. Additionally, the Department stimulates the intellectual development of its students through . . . sciences (biology) and concentrated study in . . . pre-professional . . . programs such as pre-medical, pre-dental, pre-physician’s assistant, pre-medical technology, pre-physical therapy, biology teaching, etc. which are all under the umbrella of the B.S. in Biological Sciences. Lastly, the Department’s B.S. degree program in Biological Sciences offers its students opportunities for service in ways such as working as a volunteer at the Clinton Community Nature Center and as judges in science fairs at area high schools. STRATEGIC GOAL I: Academic Excellence Mississippi College will pursue excellence by employing effective faculty, staff, and administrators; recruiting students who can benefit from the university experience; providing an enriched learning environment and innovative delivery methods; and promoting outstanding scholarship, service, and creative activities that advance knowledge. STRATEGIC GOAL II: Undergraduate Curriculum The undergraduate curriculum will prepare students for a lifetime of learning and instill basic skills, knowledge, and attitudes necessary for personal development and reasoned response to a changing world. These goals will be accomplished through study in a general education component of liberal arts and sciences balanced with concentrated study in specialized fields. This educational experience will provide students the opportunity to pursue academic studies and to prepare for meaningful careers as they serve God and others. STRATEGIC GOAL IV: Faculty and Staff Mississippi College will recruit, employ and retain highly qualified Christian faculty, staff, and administrators who demonstrate interest in serving the university and community. The university will seek Christian faculty who also provide effective teaching/learning and advising of students, who support and engage in scholarship and creative activities that advance knowledge, and who seek to continue their own professional development. The university, in addition, will seek staff and administrators who support these efforts. To this end, the university will provide compensation and benefits commensurate with regional universities and each individual’s level of accomplishment and will support professional development for all personnel. UNIT MISSION STATEMENT: It is the mission of the Biology department to provide the best possible scientific education in preparation for professional school, graduate school and postgraduate careers. The Biology Department contributes specifically to the mission of Mississippi College through Strategic Goal I. |
| INTENDED OBJECTIVES/OUTCOMES: CONTENT MASTERY: Students graduating as biology majors will compare favorably in their knowledge of biology subject matter as compared with students graduating from other colleges and universities in the United States. STUDENT SATISFACTION: Students majoring in biology will be satisfied with their academic experience, which includes academic advising, and quality of instruction received from faculty members in the department. PLACEMENT: Students completing the pre-medical/pre-dental and research tracts will be successful in gaining admission to medical school, dental school or graduate school, respectively. Students completing the general biology track will be successful in gaining admission to appropriate postgraduate programs or find employment related to biology. Students preparing to teach biology will be successful in attaining state certification to teach. RESEARCH: Students will have adequate opportunity to engage in research opportunities. |
GOAL I: Students graduating as biology majors will compare favorably in their knowledge of biology subject matter with students graduating from other colleges and universities in the United States.
| OBJECTIVES WITH INTENDED OUTCOMES/ ASSESSMENT MEANS/MEASURES/STRATEGIES: | ASSESSMENT RESULTS (MID-YEAR AND END-OF-YEAR PROGRESS): | EVALUATION/ IMPROVEMENT (ACTION) PLAN/ CHANGES MADE: | STRATEGIC PLANNING AND BUDGETARY IMPLICATIONS: |
| PLAN: | RESULTS: | ACT: | ANALYSIS: |
| The Area Concentration Achievement Test (ACAT) in Biology consisting of questions from the subjects of cell biology, cell physiology, genetics, and ecology will be administered to all biology majors during the spring semester following completion of the core biology courses: Biology I (BIO 111), Biology II (BIO 112), Cell Biology (BIO 305) and Genetics (BIO 306). Students not completing the exam as described above will be tested during the semester in which they register for Biology Capstone (BIO 433). A student’s national percentile score will be included in the assessment during the year the student graduates. The criterion for favorable comparison to other colleges and universities will be a mean score of 50th percentile relative to national scores. | The ACAT was administered to biology majors completing the core biology courses and during the fall 2016 and spring 2017 Capstone Courses (BIO 433). ACAT scores are reported for the undergraduate biology majors awarded degrees during the August 2016, December 2016 and May 2017.
During 2016-2017, 50 students received a BS in Biology from MC.
Criterion met: The mean ACAT score for 2016-2017 Biology B.S. graduates was 68th percentile.
| Actions in 2016-2017 based on 2015-2016 assessment:
We administered our new reduced format ACAT exam for its third year (the new exam eliminated the topic of animal physiology because we do not teach this topic as part of the Biology Core).
We observed mean scores consistent with the first two years of the new exam format. Long-term data comparisons should not be made between the two exam formats.
One new faculty member, Dr. Courtney Haycraft, was hired to teach in the Biology Core courses that help prepare students for the ACAT. Staffing continuity is necessary to help provide a strong foundation for our students. An additional faculty member is needed to accommodate growing enrollment in the major and effectively meet the needs of our students for solid foundation at the freshman and sophomore levels. | Budget needs for 2018-2019:
Additional faculty members are needed to ensure a low student to faculty ratio for teaching in the Biology core. Further equipment upgrades and modernization of the core BIO 111 and 112 laboratories (Hederman Science 107 and 110) are needed.
|
| New format | 2012-2013 | 2013-2014 | 2014-2015 | 2015-2016 | 2016-2017 | Mean |
| N= | 54 | 47 | 50 | |||
| percentile | 62.3 | 70.2 | 70.9 | 67.8 |
The report you see above represents how the undergraduate biology faculty close the loop on the assessment of the content to be articulated through the academic program. This involves
· writing a broad goal,
· determining what measurements will be used to determine if the goal was met,
· reporting the results of the measurement, and then
· discovering how to improve on the outcomes based on those results, and
· indicating if resources will need to be allocated to accomplish the goal.
Every unit, both academic and administrative, has an assessment report similar to this one. This is how we will know that the institution is effective. Note also that the assessment plan of this unit is aligned with the university mission and its strategic goals (as discussed in Topic 2).
It is vital to define measurable ways to determine effectiveness. The measurement is best when it has an external evaluation and is not just an internal evaluation. For example, if grades are used, grades are internal and have no external validation. Therefore, they may not be an effective indication of institutional effectiveness. This plan indicates an external evaluation with an ACAT exam (Links to an external site.) .
It should be noted that this is one of the many ways this department measures its effectiveness. The former was an academic example. The following is an example from the administrative side—the Office of Student Development.
ASSESSMENT YEAR: 2016-2017
SUBMITTED BY: OFFICE OF STUDENT DEVELOPMENT
PARTICIPATION: Jonathan Nutt, Director of Student Development
| UNIVERSITY MISSION: Furthermore, the university environment promotes the spiritual, social, emotional, and physical development of its students and encourages them to utilize their skills, talents and abilities as they pursue meaningful careers, life-long learning and service to God and others. STRATEGIC GOAL: Strategic Goal V. Student Development and Services Mississippi College will seek to prepare its students for a variety of careers and service to God and others. The university will offer students personal, career and academic counseling in a Christian manner; lectures and programs that are enriching and uplifting; student organizations; and opportunities for university and community service. VIII. Christian Setting Mississippi college will manifest Christian principles through its policies toward students, faculty, staff and administrators; promote a campus atmosphere of helpfulness and caring; and encourage Christian relationships among students, faculty, staff and administrators. UNIT MISSION STATEMENT: In step with the Mission of Mississippi College, the Office of Student Development seeks to assist students in their academic, spiritual, and social development. This is achieved by advising conduct board members in carrying out the student conduct process; advising student organizations; overseeing the Intramural program; and advising non-social student organizations. As a part of the Department of Student Life, the Office of Student Development works closely with the Office of Student Activities, the Office of Residence Life, and Career Services. |
| INTENDED GOALS: 1. Goal I: Students will express confidence and satisfaction with the student conduct process and sanctioning. 2. Goal II: Students matriculating the conduct process will not repeat violations of the student code. 3. Goal III: Students participating in the club and tribe pledge process will express satisfaction and confidence in the pledging process. 4. Goal IV: Student leaders within clubs and tribes will be given opportunities for growth and development through training events, workshops and other events |
Goal I: Students will express confidence and satisfaction with the student conduct process and sanctioning.
| OBJECTIVES WITH INTENDED OUTCOMES/ ASSESSMENT MEANS/MEASURABLE STRATEGIES: |
ASSESSMENT RESULTS: |
EVALUATION/IMPROVEMENT (ACTION) PLAN/CHANGES MADE: |
STRATEGIC PLANNING AND BUDGETARY IMPLICATIONS |
| At least 85% of students navigating the conduct process will feel like they were treated fairly throughout the process.
At least 75% of the students navigating the student conduct process will recognize there are consequences for violating the Code of Conduct.
At least 75% of the students navigating the student conduct process will indicate the process helped them take responsibility for their actions. | In the Only 44% of students navigating the conduct process felt like they were treated fairly, while 22.2% were neutral on the topic.
100% of students surveyed that navigated the student conduct process recognized that there are consequences for violating the Code of Conduct.
55% of students surveyed that navigated the student conduct process responded that navigating the process increased their ability to take responsibility for their actions, while 33.3% responded they were neutral on the issue. | Due to a low response rate for this survey, the data may be skewed. Having different conduct officers handling cases may also be a reason that some students felt like they were not treated fairly. With many cases going back to the Director of Student Development during the 2017-2018 school year, this statistic could come out better in the future. Not much improvement could be made with this statistic, considering 100% of student surveyed felt like there were consequences for their actions, but the low response rate may have skewed this data.
Although this statistic is lower than projected, the presence of such a high “neutral” response may skew the data. Having the option to be “neutral” on the question may be hurting the outcome data in this survey. | No budgetary implications. The implementation of Maxient Software will hopefully allow the Office of Student Development greater capability in tracking not only trends, but also surveying students and gathering more data. The software will be implemented in summer 2017. No budgetary implications. The implementation of Maxient Software will hopefully allow the Office of Student Development greater capability in tracking not only trends, but also surveying students and gathering more data. The software will be implemented in summer 2017. No budgetary implications. The implementation of Maxient Software will hopefully allow the Office of Student Development greater capability in tracking not only trends, but also surveying students and gathering more data. The software will be implemented in summer 2017. |
It should be noted that this is one of the many ways this department measures its effectiveness.
Assessment of every university unit must be accomplished. For assessment to be meaningful, the “loop must be closed.” What this means is that goals are set, criteria for success established, results analyzed, and changes made to improve the unit. Without improvements made, assessment is of little use to the university.
One other technique used in determining if assessment is effective is to provide graphical presentations of the data. This provides a visual way of communicating assessment results. Both units have done this.
