1 Overview
The Multimedia Educational Resource for Teaching and Online Learning, MERLOT, is an international consortium of partners in higher education. Partners include state systems of higher education, individual campuses, professional societies and organizations, discipline-specific digital libraries, corporate entities, and international educational digital libraries and resources. These partners collaborate to help faculty and instructors overcome one of the major barriers to use of digital library materials: finding high quality, effective materials that meet their teaching objectives.
The task of finding online learning materials can be a hugely time-consuming activity. The search alone is arduous, but when added to the need for intensive instructor review of those materials, and that once identified they must also learn how to use the materials effectively for teaching, the task becomes formidable. It is no wonder that the hurdles to the effective use of online learning materials are many (CSHE 2004, Gibbs et al. 2004). Whether the classroom is real or virtual, faculty and instructors seek materials that support their teaching efforts, their pedagogy and student learning goals. MERLOT's services and features are designed to help faculty and instructors overcome the hurdles associated with finding good materials (e.g. lack of time, lack of organization, overwhelming numbers of unrated materials) through the integration of peer-reviewed online materials with effective teaching practices.
2 Short history of MERLOT
Created in 1997 by the California State University, MERLOT's mission is to be a free, Web-based starting place that connects faculty, staff and students to online learning materials that have been peer reviewed. In 1998, a State Higher Education Executives Organization/American Productivity and Quality Center (SHEEO/APQC) benchmarking study on faculty development and instructional technology selected the CSU-Center for Distributed Learning (CDL), the organization that initiated MERLOT, as one of six best practices centers in North America. Visits to the CSU-CDL by higher education institutions participating in the benchmarking study resulted in interest in collaborating with the CSU-CDL on the MERLOT project. As a result, the University of Georgia System, Oklahoma State Regents for Higher Education, University of North Carolina System and the California State University System joined together and created an informal consortium representing almost 100 campuses serving over 900,000 students and over 47,000 faculty.
In 1999, the four systems recognized the significant benefits of their cooperative efforts and created an initiative to expand the MERLOT collections, conduct peer reviews of the digital learning materials and submit example learning assignments. Each system contributed $20,000 in cash to develop the MERLOT software and over $30,000 in in-kind support to advance the collaborative project. The California State University System maintained the leadership of and responsibilities for the operations and expansion. In 2000, MERLOT expanded to include 23 systems and institutions of higher education and each Institutional Partner contributed $25,000 as well as provided the in-kind support of eight faculty members to participate in the peer review process and a part-time project director to coordinate MERLOT activities.
In January 2000, MERLOT initiated its peer review process. This was supported in part through a grant from the National Science Digital Library (NSDL) program of the National Science Foundation (NSF). The four systems of higher education participating in MERLOT sponsored 48 faculty members from the disciplines of biology, physics, business and teacher education to develop evaluation standards and peer review processes for online teaching-learning material. As new and other partners joined the project they contributed faculty members to participate in the expanding peer review program. By 2003, the number of discipline-based Editorial Boards supported by MERLOT increased to 14 and had expanded to include: biology, business, chemistry, engineering, health sciences, history, information technology, mathematics, music, psychology, physics, teacher education, teaching and technology and world languages.
In 2002, MERLOT extended its reach in the higher education community by adding additional levels of partnerships. Individual campuses may now participate in the MERLOT project through the Campus Partner level. In addition, MERLOT recognized the value in collaborating with professional societies, other educational organizations, and digital libraries and collections. It formalized these collaborations through memoranda of understanding with its Alliance Partner program. In 2003, MERLOT formalized a partnership program for industry and other businesses with interests in higher education. A full list of all MERLOT partners is available.
Through collaboration with institutions of higher education of all types - community colleges, liberal arts schools, comprehensive and research universities - and at all levels, i.e. individual campuses, state systems of higher education, and consortiums of colleges and universities, MERLOT achieves its goals of putting innovations in higher education and technology into practice. Strategically, this is accomplished through creating and supporting communities of users by combining their resources with those of the partners. In all situations, collaboration in MERLOT depends on community involvement. This involvement has led to a collection that is now made up of over 9,000 learning materials and over 19,000 registered users. Approximately 70% of the MERLOT collection has been through the review process. The number of individual users served by the MERLOT Web site has shown significant growth since its inception. For example, the average number of unique visitors per month increased from 9,579 in 2001 to over 17,000 per month by 2003. The average number of users who visited the site monthly in 2003 was around 27,000, almost 10,000 more than the average for 2002 (Hanley 2004).
3 Role of the community
Digital resources are usually made available either as an item in a repository of materials (digital libraries and physical libraries are examples of a repository) or in a referatory (inter-library loan services are an example or a referatory) (Erhmann et al. 2004). MERLOT, with its collection combined with the context it wraps around the objects (i.e. peer reviews, assignments, user comments and personal collections), and its member services, finds itself in a unique role, neither purely a repository nor referatory. Instead it combines aspects of each in collaboration with an active membership community to support the depth and breadth of academic program needs, reliable access to learning objects (i.e. persistence), multiple services (search, browse, descriptive and quality standards for learning objects). These services are critical in the support of faculty members and students in finding, selecting and using appropriate learning objects. Key to MERLOT's ability to provide these services, and ultimately to its sustainability as an organization, is its commitment to creating and supporting services that support the needs of its users as expressed by the users and representatives of those users - the campus and institutional partners.
3.1 Sustaining the organization
To sustain its programs and services, MERLOT's business model is built primarily around partnerships with organizations that leverage one another's resources. Initially, all Partners were institutions of higher education. Institutional Partners provide MERLOT with $25,000 per year, Campus Partners provide $6,500 per year and Sustaining Partners provide $50,000 per year. Regardless of the level, each partner also provides significant in-kind contributions to support faculty and staff participation on the Editorial Boards and other projects, such as, faculty development workshops (see Figure 1). Each of these contributions is necessary to identify, creat and sustain the kinds of services that meet the needs of the Partners.
Figure 1. MERLOT framework for collaboration
The CSU System continues to take the lead responsibility for the management, planning, and operation of MERLOT's processes and tools. As previously mentioned, MERLOT has expanded its partners to include national digital libraries (Health Education Assets Library (HEAL), the National Engineering Education Digital Library (NEEDS), and CAUSEWeb - a digital library for statistics education), international digital libraries and projects (EdNA Online - Education Network Australia, and eduSourceCanada), organizations serving higher education (The Teaching, Learning and Technology (TLT) group and Project Kaleidoscope), and from the corporate world (McGraw Hill and IBM). The following discussion focuses on the roles of institutional partners since they are the direct links to end-users, faculty, instructors and students.
3.2 Partnering with institutions of higher education: roles and responsibilities
To expand its services to serve a wider audience, in 1999 MERLOT sought out other partners who shared its commitment to improving higher education by improving access to high quality online learning resources and materials. The original Institutional Partners combined resources to move the project into the national arena. The structure and management of MERLOT reflects their commitment to the principles of user-centered design, the need for active user involvement in all stages of the project (from contributor of materials, to viewer/potential user of those materials), the need to integrate online services with active face-to-face training and community building, and the need to focus its programs around practical solutions for the problems facing higher education. The driving vision of these partners also focused MERLOT's services to serve institutions of higher education as they strive to integrate technology with their educational programs at the state-wide and campus levels, as well as at the discipline and departmental levels.
This vision and focus on solving tough problems through leveraging resources and collaborative work has been a central tenet of the management philosophy of MERLOT. As a result, organizationally and structurally, these goals were accomplished by creating the collaborative governing and management structures. Figure 1 illustrates how resources come to MERLOT and how it is governed. Advice from the various groups feeds directly to the Executive Director and MERLOT administration. Resources follow two paths: in-kind support (e.g. faculty time) goes directly to MERLOT programs, projects and initiatives; the second path, partner fees and other fees raised through grants are funneled through the administration team to support programs, initiatives and the Web site.
3.2.1 Advisory Board
Currently MERLOT is a developing non-profit organization. It maintains its strong connections to the CSU system because such relationships are critical to sustain and grow the organization. MERLOT's Advisory Board represents the broad set of stakeholders in the organization, i.e. the user community, experts in the field, as well as representatives from business entities with similar goals, interests, and constituencies. In the future, this board will transform into the Board of Directors and will be responsible for the fiduciary health of the organization, and hiring the executive director in conjunction with staff planning for the future.
3.2.2 Project Directors' Council
The main body that advises and plays a significant role in the governance of MERLOT is the Project Directors' Council. Project Directors liaise with the campus and institutional members who comprise MERLOT. Their role is to integrate MERLOT into their campus or institutional technology initiatives and keep MERLOT informed about the results of these efforts. Project Directors also work closely with the Editors' Council in supporting the creation of new editorial boards and in sustaining and growing the existing boards. Project Directors keep and report on the "pulse" of technology initiatives at the campus and system-wide levels; they identify emerging trends with regards to the technology needs of higher education, as well as individual faculty needs. Due to their campus roles (the positions of Project Directors include Vice Chancellors, Board of Regents Senior Administrators, Directors of system-wide technology, and Directors of faculty development) they are uniquely situated to communicate the needs of their constituents.
MERLOT's organizational initiatives result directly from the needs identified by the Project Directors. Through regular consultation Project Directors provide feedback on, and direction for, future projects, services and activities of initiatives. For example, MERLOT's Faculty Development Initiative was designed to provide Project Directors with support for their faculty development professionals in their efforts to promote the use of digital learning materials in teaching. Through this initiative, MERLOT provides direct face-to-face training and workshops for faculty developers on developing strategies for working with faculty to integrate online learning technologies into their teaching. And in cooperation with MERLOT, the State University of New York's (SUNY) Teaching, Learning and Technology Cooperative offers an online course for faculty developers.
At the system-wide level partners report that their involvement in MERLOT has led to significant changes, as one of the Project Directors put it: "MERLOT has been an excellent faculty development initiative for a state system, since it promotes collaboration and efficiency. MERLOT is a unique organization, approaching common issues and challenges in ways no others do. MERLOT extends our reach and resources by partnering with other systems, organizations, and agencies. I don't think we could accomplish nearly as much if we tried to do these things by ourselves" (MERLOT 2003).
3.2.3 Editorial Boards
The major responsibility for implementing MERLOT's commitment to improving accessiblity to high quality learning materials rests with the members of the Editorial Boards supported by MERLOT and its partners. Institutional and Campus partners provide Editors and some members of the Editorial Boards with financial support for their leadership in this area. The Editorial Boards are a direct link between MERLOT and the developers of materials (authors) and users of those materials. Board members are selected on the basis of their:
-
expertise in the scholarship of their field
-
excellence in teaching
-
experience in using technology in teaching and learning
-
connections to professional organizations in the discipline
-
experience in conducting peer reviews of online learning resources
In addition to conducting peer reviews Editors and Editorial Boards are responsible for building the collection of materials within their disciplines and building a learning community of their peers. With regards to these latter responsibilities, they are responsible for:
-
Establishing and facilitating affiliations between the MERLOT and appropriate professional disciplinary organizations.
-
Leading in the definition and assisting in the development of the MERLOT discipline Web site and services.
-
Identifying generic and discipline-specific issues that arise that impact MERLOT's growth, development and ability to meet the needs of members.
-
Providing information to the MERLOT Administrative Team regarding the customization of their MERLOT (Discipline Community) Web site.
-
Developing strategies and implementing activities to encourage the MERLOT (Discipline Community) members to participate in the site by developing and adding learning assignments and/or adding comments to MERLOT learning materials and resources.
The Editors lead their teams in developing short and long range plans for accomplishing the above. As users, Board Members provide MERLOT with a window into the world of faculty users. Editors, through the Editors' Council, are the conduits through which much of the feedback flows from board member to the MERLOT administrative team. Additionally, board members provide direct feedback regarding services and future needs through regular surveys and face-to-face meetings held during the annual MERLOT International Conference.
Editors and editorial board members are extremely active in disseminating information about MERLOT and its services to faculty on their campuses and within their profession. As leaders in the field of online teaching and learning they are frequent presenters at regional and professional society conferences and meetings. Like the Project Directors, they provide MERLOT with direct links to the communities of users for whom services are designed and developed. They are active in testing and designing services prior to deployment. For example, MERLOT (2004) has developed an online training tutorial on conducting peer reviews.
The prototype tutorial (used in conjunction with a mentorship program) was designed to train volunteer peer reviewers using as few resources as possible. In the mentorship program, volunteer peer reviewers are recruited and trained; this is one strategy to increase the number of editorial boards MERLOT supports. The problems associated with the growth of online journals have been instructive in the development and promotion of the MERLOT volunteer mentorship program. The problems of maintaining a constant influx of new materials and then maintaining the pace of peer reviews of those materials are problems that, to date, MERLOT has not experienced to the levels described in the description of the growth of CoRR and the Los Alamos eprint archives (Carr et al. 2000). In part this is due to the newness of the field of digital learning materials and in part because of the nature of educational technology: learning materials are simply not the same as a scholarly research paper. Additionally, MERLOT designed its process to guard against the problems described by Harnad (2000) regarding the review process, through its thorough training and because the reviews are public - directed towards the user rather than the author. So far, the strategy to increase both the size of the boards and number of boards through increased reliance on volunteers has been successful and has served as an initial step in changing the rewards for faculty involved in review activities. The dissemination activities described above are focused on building interest in MERLOT and recruiting potential volunteer peer reviewers.
Reviewing learning materials requires a skill set unlike those used in the review of research articles or funding proposals. The MERLOT process requires both knowledge of the content as well as knowledge and experience of using technology to teach that content. Since the reviews are public and their purpose is to inform potential users about the strengths and weaknesses in the use of these materials, training is essential to ensure the reviews are reliable and accurately reflect the review criteria. The review process was intended to leverage the academic culture associated with service to the discipline so this is a "natural" next step in that process.
Inherent in establishing a review process for this relatively new aspect of teaching in higher education is the challenge of establishing it as a status activity, e.g. one that counts in the tenure and promotion process. This is perhaps one of the most serious challenges faced in recruiting and keeping volunteers; the reward structure simply does not exist at this point. Some headway has been made, however. We see among our partners increased interest in the role of peer review, and some institutions are exploring how to integrate this process more formally into their reward structures.
Members of the MERLOT partners report that their involvement has had a positive impact on their teaching and careers. As one Editorial Board member put it: "Being on the Editorial Board gives me a sense of accomplishment ... I have provided the academic community with some measure of the quality of online materials that [they] might consider using in the classroom ... by using many of the materials I have found and/or reviewed ... it has helped me become a better teacher". (MERLOT 2003)
4 Future directions
As MERLOT grows and its services expand, the role of its users becomes equally more central. To date, our Campus and Institutional Partners have had direct input into and responsibility for defining, developing, implementing and testing MERLOT's services. As MERLOT's primary stakeholders, a number of communications structures have been implemented to encourage, support and in some cases require their input and participation in establishing the directions for MERLOT's growth. Given the success of this approach, collaboration will continue as the cornerstone for building on and improving MERLOT services to faculty and instructors. Innovation and scaling the project also depends on the collaborative relationships among the MERLOT community.
As MERLOT's collaborators expand to include academic professional societies and other corporate and non-profit members of the higher education community, it will be essential to develop pathways of communication that connect existing partners with new partners. MERLOT's focus on collaboration has been on establishing relationships that further common goals, promote collaborations among innovative organizations for the purpose of improving higher education, and meet the needs of the individual end user. The challenges are many. Effective collaborations require: excellent communication methods and paths, high levels of trust among participants, a commitment to leverage resources rather than compete for them, and the hard work of many. As the short history of MERLOT and its partners illustrates, collaborative efforts and the intimate involvement of communities of users in the design and development of digital library services is a successful sustainability strategy.