CineFiles is a digital image database of documents selected from the collection of the PFA Library and Film Study Center. PFA’s documentation collection holds over 200,000 documents related to films, film history, and filmmakers; to date over 50,000 of them are available online through CineFiles. CineFiles has been traditionally used by students, scholars, and educators in film studies departments at colleges and universities across the country and internationally. In 2006, BAM/PFA received a three-year grant from IMLS, “Expanding Classroom Access to Online Film-Related Resources,” to increase the relevance of the database to educators in history and social sciences. This process involved both updating the website to make it more user-friendly to all educators (with a special focus on humanities educators) and modeling CineFiles resources in the classroom to demonstrate how primary and secondary text sources can be valuable learning resources when used alongside classroom viewings.
See below for the final report PFA submitted to the IMLS at the conclusion of the grant. It provides a detailed account of the project along with relevant appendices and attachments.
Results and resources from another IMLS-funded project about working with copyright-protected materials in a digital environment can be found through the portal for the Copyright Resources Project.
FINAL REPORT TO IMLS
(Award number: LG-05-06-0167-06)
Download PDF versions of this report and appendices.
EXPANDING CLASSROOM ACCESS TO ONLINE FILM-RELATED RESOURCES
The activities described in this report were conducted by Pacific Film Archive (PFA) staff, in collaboration with local high school and college educators, as well as the UC Berkeley Information Systems and Technology (IST) Data Services staff, but there were no formal partnerships.
The purpose of the project “Expanding Classroom Access to Online Film-Related Resources” was to increase multidisciplinary use of the CineFiles film document image database by a broad audience of social science and humanities educators in secondary and higher education. By working closely with an advisory committee and with educators, we developed a model for expanding access to online film-related documents beyond the film studies community it traditionally serves. We developed criteria for document selection; processed nearly 15,000 documents and made them available via the CineFiles website, along with a new Portal for Educators featuring a Themes and Topics list and Resource Guides; and redesigned the website in line with usability results. We developed curricula using CineFiles and modeled them in the classroom. We are delighted to report that the project was very successful at achieving its goals.
This final report covers the full grant period from October 1, 2006 to September 30, 2009. The project began in October 2006 with staff meetings and discussions to plan workflow, seek consultation and hire staff for the project. The following staff positions were filled in late 2006 and early 2007: Indexer, Scanner, and Outreach Coordinator. Staff continued to meet regularly throughout the grant period to review and adjust the project work plan as needed.
Throughout the project, project staff met and consulted with the advisory committee, who gave feedback on every step of the process, through in-person meetings and phone conferences. The committee reflected a variety of teaching interests and outreach expertise, including educators, librarians, and communications specialists who gave invaluable advice on how educators discover and incorporate digital resources into teaching. The committee also lent their expertise on designing innovative, nationally adaptable selection and outreach methodologies. The committee members were: UC Berkeley Professors Richard Hutson (English Department) and Linda Williams (Film Studies and Rhetoric Departments), as well as San Francisco State University Professor Jillian Sandell (Women and Gender Studies Department), who all use media extensively in teaching; digital imaging and access expert Dr. Howard Besser, who directs NYU’s graduate program in Moving Image Archiving and Preservation; Gary Handman, who heads UC Berkeley’s Media Resources Center and is the video selector for the UC Berkeley Libraries; Catherine Candee, who directs the California Digital Library’s e-Scholarship initiative to encourage faculty design and use of digital resources; and David Greenbaum, Director, Data Services in UC Berkeley’s Information Systems and Technology division. External evaluators, contracted from local consulting firm Rockman et al. (REA), also reviewed surveys, focus group questions and usability procedures, to make sure that best practices were being followed.
Following is a summary of the project activities. It is arranged by activity, with the sequence of events arranged chronologically within each activity.
Gathering information from humanities educators
Based on recommendations from the advisory committee and REA consultants, project staff identified three key methods to collect information from high school and college educators in the humanities: focus groups, surveys, and research.
Recruiting educators to participate in the project: In April 2007, Outreach Coordinator Anuj Vaidya participated in a focus group for history educators organized by the San Francisco Education Fund. This event was very valuable to us as it gave us an opportunity to introduce our project and its goals to a broad group of local educators, and to make connections that were useful in helping us identify teachers to participate in focus groups and workshops, including representatives from the San Francisco Unified School District, the public television station KQED, and members of the UC Berkeley History-Social Science project.
During the spring and summer, project staff worked on a variety of outreach activities to identify and initiate contact with potential project participants, including meeting with administrators in the Berkeley and Emeryville Unified School districts to discuss the potential use of CineFiles in the classroom and coordinating with UC Berkeley Humanities and Social Sciences departments.
Focus Groups and Interviews: In September and October 2007, Anuj conducted focus groups and individual interviews with high school teachers and college educators, addressing questions including: How do you use films in the classroom?; What curricular tools would be most useful to you in building new curricula?; What kind of documents would be most useful in support of films?; What themes/topics or titles should we concentrate on?; What improvements would you most like to see on CineFiles?
Surveys: In November 2007, Anuj launched the first survey for high school educators. This was followed in April 2008 with a survey for college educators. In addition to addressing the questions raised in the focus groups, the surveys also gathered specific information on the resources used by educators to gather supporting materials for the classroom curriculum. Between November 2007 and April 2008, 73 responses were collected from humanities educators.
Findings from Surveys and Focus Groups: The findings from the focus groups and surveys helped project staff determine the scope and direction of future project activities. The results showed that high school educators required more curricular assistance than college educators – college educators preferred to repurpose curricular support materials to match their own lesson plans, while high school educators preferred to receive complete lesson plans with suggestions for clips to use in the classroom and discussion questions. Both high school and college educators tended to use both entire films and shorter segments from films within the classroom. College educators tended to use documents in support of classroom viewings more than high school educators. Both high school and college educators requested suggestions for films to be used for a variety of topics. Finally, both high school and college educators requested a wide range of documents (from reviews, interviews, and press kits to scholarly and shot analyses) in support of classroom viewings.
Selection criteria for documents to be added to CineFiles
The focus groups and surveys gathered specific information on film titles used regularly in the classroom to support history/social science education, and curricular areas for which educators required film title suggestions. Film titles to be added to CineFiles were based on responses to this part of the survey. Additionally, Outreach Coordinator Anuj Vaidya asked educators to send in their existing curricula so that we could mine it for titles used in the classroom.
Ultimately, based on surveys, focus groups and research, the following criteria were used to determine whether a film title and related documents were added to CineFiles:
- Was the title relevant to a subject area for which educators had requested resources?
- Did the title lend itself to creative use in the classroom?
- Did the PFA paper documentation files contain rich resources for the given title?
Processing selected documents
Over the course of the three-year grant, project staff selected, indexed, and digitally captured 14,006 documents totaling 29,735 page images. The documents that were processed fit the criteria described above. Copyright Analyst Stephanie Boris continued negotiating permission to display copyright-protected documents. Additionally, staff continued to process documents requested specifically by professors. For example, high school Language Arts, Economics, and Social Studies teachers and college professors in departments including Native American Studies, History, Landscape Architecture, American Studies, Film Studies and Ethnic Studies asked us to process specific titles they were using in courses.
The number of documents is slightly below our planned goal of 15,000 documents, while the total number of pages scanned misses our rough estimate by less than 300 pages. Even though these numbers are slightly lower than anticipated, they represent a significant range of materials of direct relevance to our target audience and have already spurred greater interest in and use of the database.
Over the three-year grant period, we built a list of 25 themes and topics (with over 100 subtopics), starting from our initial representative list of subject areas and films. Most of the titles were gathered from survey responses and curricula amassed through online research. We also added the complete filmographies of over 40 new directors to our featured directors list, and processed all documents related to these directors’ works.
Building tools for usability testing: In April 2008, Anuj began working with a volunteer consultant from UC Berkeley's School of Information Management and Systems to develop tools for website usability testing on the existing website. The first step in the process was to develop User Personae, which involved identifying the types of users who might use CineFiles and the purpose for which they might do so. Five different personae were identified as key users of CineFiles: College professors who are looking for documents to distribute to their students, college professors who are looking for documents to help develop their own lecture notes, film-related professionals, graduate students who are looking for materials to supplement their research, and high school teachers who might be looking for documents to supplement the classroom curriculum.
Anuj then built a mockup of a new CineFiles website which included new features that had been suggested through the focus groups and surveys. The mockup did not include any kind of new design, but was merely a skeleton with different features placed on the page. The usability test was then conducted using the mockup for navigation, while the existing website was used to conduct real-time searches.
The final step in developing the usability testing tools was to develop an interview script that would address the concerns for which we were seeking responses. These concerns primarily included the following: clarity of the website, ease of navigation, robustness of simple search and advanced search, usefulness of browse lists, usefulness of Portal for Educators, features requested on search results page, features requested on document view page, and print/save features for documents. Based on these concerns, a usability testing interview was developed, which was a standard script that led test users through a simple sequence of tasks.
Conducting usability tests: Anuj conducted a test interview session with a member from the advisory committee, and was evaluated on his interview methodology by both external consultants, REA (Detailed Evaluation) and Olga Amuzinskaya, Website Usability Expert (Expert Usability Report) and, to ensure that best practices were being followed in the testing. He then recruited fourteen users for testing in May 2008, with at least two from each user personae category. Anuj noted concerns and behavioral patterns revealed during their navigation of the site, searches and document viewing, and collated results from all fourteen usability interviews into one document.
Converting usability results into a new design: The next step in the process was to review the results from the usability testing with both the IST website developers and the BAM/PFA designer to get their opinions about what improvements were feasible within the given budgetary and time constraints of the project. Based on their input and the usability results, the designer developed a new design for the CineFiles website by September 2008, for which the website developers created technical specifications document and a project plan.
At the end of this process, the following updates to the CineFiles website had been decided upon:
- The look of the new CineFiles site was to be brought in alignment with the BAM/PFA website, in order to make clear the connection with the larger institution and to leverage its connection to UC Berkeley.
- Browse lists for subject, genre, and directors were to be implemented. This was an important addition to the site, since the indexing in CineFiles is done according to Library of Congress Subject Headings, which are specialized terms not in common usage. The browse lists give users the opportunity to identify terminology used to index materials in CineFiles, and therefore increase the percentage of successful searches.
- A featured document element was to be added to the homepage. This not only added an important visual anchor to the homepage, but also showcased rare documents from the collection that might otherwise not be seen by CineFiles users.
- Access to simple search, advanced search and browse lists were to be added to all pages in order to increase the website’s ease of navigation.
- A dynamic design was to be implemented for the results page, using frames, which allowed for multiple kinds of information (results lists, film citations, search and browse features, etc.) to be viewed on one page.
- Advanced sort capabilities were to be added to the Film and Document results in order to offer greater results navigation options to users.
- A dynamic design was to be implemented for the document view page, using frames, which allowed for multiple kinds of information (page images, document citation, navigation tools, print/save tools, etc.) to be viewed on one page.
- A thumbnail view of all document pages was to be implemented in the document view page, to ease navigation of the document.
- New download features, such as saving the entire document as a PDF with a complete citation, were to be implemented, allowing users more options for how to view and save the documents for future use.
- A simpler means of requesting copies of copyright-restricted documents was to be added.
In November 2008, a UC Berkeley student also employed by the BAM/PFA digital media department was hired to develop the Cascading Style Sheets (CSS) for the new web design, while IST website developers began developing a data model and data access methods for the proposed website. This involved recasting the existing document records into a new database optimized for web access. We initially expected to complete the site by the end of March 2009, allowing for testing during the month of April and a launch in mid-May. However, due to technical difficulties and staffing cuts at both PFA and at IST, the project timeline was pushed back by three months. The beta version of the website was ready for testing in July 2009. Project staff conducted post-usability testing on the website in August 2009 (see section below for more information), and the new website was launched on September 15, 2009 at the beginning of the fall semester (Launch Announcement).
Post-Usability Testing and Pre-Post Website Redesign Surveys: Following the completion of the new design, Anuj conducted two sets of surveys and a second round of usability testing on the new website. Five test subjects were recruited for the test, one each from each of the original user personae determined before. Their responses indicated that they found the new design greatly improved. User highlights of the new website were: search capabilities on every page of the website, the new featured document element, dynamic design on the results and document view pages, thumbnail navigation, and the PDF download function. All test users responded that the new website was much clearer in highlighting the database’s contents, and in terms of ease of navigation. Educators were especially excited by the Themes and Topics List under the Portal for Educators. The areas the respondents felt needed more work were: print options, more robust search capabilities, and a greater number of Resource Guides for classroom use. We hope to implement these suggested improvements in future phases of the project.
Website Redesign Surveys: Two sets of surveys were conducted, one at the end of each round of usability testing. Each survey gathered responses from 14 respondents about the ease of navigation on the website. The first survey responses were based on the mockup designs for the new website, while the second set of survey responses collected data on the completed design. The first survey set indicated that the proposed design was easy to navigate and included highly desired features such as browse lists, easy access to search function from all pages, and PDF downloads. The second survey set indicated that the redesign was highly successful, as more than 70% were satisfied with the ease of navigation on the website. Users were also satisfied with the search features, document view options (including thumbnail views), the quality of document images, and PDF download options.
Portal for Educators: Based on findings from surveys and focus groups, it was deemed that a separate Portal for Educators would be a very useful addition to the website, especially to encourage greater use of the CineFiles resource by high school humanities educators. This feature is a common feature in a number of online resources now (such as ITVS, KQED, Calisphere, etc), and has become common expectation for educators who are pressed for time and need online resources organized in a manner that affords easy navigation and quick response. Since this was a new addition to the project, we decided to set this up as a pilot with model resources that could then be expanded on in future updates to the website. The Portal for Educators included the following elements:
- A list of Themes and Topics with recommendations for films: This list was created based on results from surveys, focus groups and research. The preliminary list was then distributed to key educators and PFA curators who helped consolidate the list of themes and the film recommendations into 25 themes and topics with at least 5 film recommendations under each.
- Resource Guides: Outreach Coordinator Anuj Vaidya created Resource Guides for three titles in CineFiles that had rich resources. These guides were intended as a sample as to how CineFiles resources could be used in the classroom to address a variety of curricular areas.
- Useful Links: Other useful links on the Portal addressed issues such as media literacy education, and tips on how to successfully use primary and secondary source texts in the classroom.
Project staff undertook a variety of outreach methods to disseminate information about CineFiles and recruit participants for the surveys, focus groups, usability testing, and classroom modeling. During the first six months of the project, Anuj met with key administrators in the San Francisco Unified School District, Oakland Unified School District (OUSD), City College of San Francisco, KQED, and a number of humanities departments at UC Berkeley (including American Studies, South Asian Studies, Comparative Literature, and Political Science).
In 2007, a workshop presented in collaboration with KQED led to an invitation from City College of San Francisco to make ongoing presentations (one every semester) to its faculty in English, Film Studies, and ESL in order to introduce them to this important resource. Similarly, Anuj made two presentations to history teachers in the Oakland Unified School District (OUSD), through the teacher training program developed by the Alameda County Office of Education (ACOE) called the History Project Lesson Study. These presentations led to ongoing relationships with two instructors (one at the high school level, and one at the college level) who participated in in-class modeling throughout the project.
Anuj worked with teachers at two local high schools and two at the college level, coordinating several class visits during which he modeled the use of CineFiles in the classroom, resulting in students’ completion of projects that addressed media literacy and research methodology, covering subjects such as politics, economics, the environment, and film studies. At the college level, since initial surveys determined that presentations were more useful than curriculum support, Anuj conducted only one in-class modeling session in order to demonstrate the ease and value of the newly designed website in the classroom.
Anuj evaluated the value of CineFiles in the in-class modeling sessions through post-modeling interviews with the instructors, which are described in the Project Analysis section below.
Other presentations through the course of the project included: annual presentations about the website at the PFA Library Open House (which was open to all UC Berkeley faculty and students and the general public); a training session for the UC Berkeley History Project, which introduced the resource to 20 high school history teachers; two presentations for Alameda county high school History teachers through the History Lesson Study Project; two presentations to educators at the City College of San Francisco on techniques for integrating CineFiles resources into the curriculum; one presentation at PFA’s Visual Literacy Workshop, How to Read a Film; and a poster presentation and website demonstration at the 9th Annual University Museums and Collections (UMAC) conference, “Putting University Collections to Work in Research and Teaching.”
In order to widely publicize the redesigned resource and the new Portal for Educators, we undertook an email and snail mail outreach campaign that reached specific educational institutions, educational resource centers and a host of publications targeting educators. The current CineFiles outreach list reaches over 850 individuals, and the mailing list reaches over 250 different institutions across the country.
The audience for this project consists of social science and humanities educators in secondary and higher education, library and film professionals, students conducting research, students whose educators use CineFiles in their curriculum, and other CineFiles users and general researchers.
ANALYSIS OF PROJECT
The project’s established goal was to increase multidisciplinary use of the CineFiles film document image database by a broad audience of social science and humanities educators in secondary and higher education. To do so, we developed and conducted surveys and focus groups with high school and college educators in the humanities; developed criteria for documents to be added to the CineFiles database; indexed, scanned and made available the selected documents through the CineFiles database; conducted a usability survey on the original CineFiles website; analyzed usability results and developed a set of criteria for the redesign of the CineFiles website; worked with design and programming staff to refine desired website improvements and implement them; conducted usability testing on the new website; developed a new Portal for Educators on the website with resources including a Themes and Topics List and Resource Guides; conducted presentations on the CineFiles website for history and social science educators in college and high school; and developed curricula using CineFiles and modeled them in the classroom. We evaluated our success through usability testing, surveys, and interviews, as well as compiling use statistics.
We successfully met the project goal by achieving in large part our anticipated outcomes, which were:
- Social science and humanities educators from institutions of secondary and higher education increased their use of the CineFiles database and their ability to integrate its resources into curricula.
- High school and college students benefited from a more contextualized, media-rich educational experience, thus increasing their visual literacy skills.
- 15,000 documents from PFA’s collection of particular relevance to this target audience were selected, scanned, indexed, and made freely available via CineFiles.
- An enhanced and easier-to-use web interface, along with added content and in-depth indexing, yielded a higher percentage of precise and relevant search results in CineFiles.
- Managers of other digital imaging projects benefited from our research by learning from and adapting our selection criteria, outreach methods, and tutorials.
We evaluated the success of our in-class modeling via interviews with instructors. These interviews also gathered information on instructors’ ability to integrate CineFiles resources into curricula and on their increasing use of the database. We were pleased that all the instructors interviewed found that using documents in support of classroom viewings helped address issues of media literacy that were previously difficult to integrate into the curriculum. Document support also reinforced concepts for students, and helped activate their reading skills, which often become secondary where classroom viewing is involved. According to the instructors, while the old CineFiles website was not so intuitive, the newly designed website was much more intuitive and easy to navigate. Certain features on the redesigned website, such as saving documents as PDFs, and the Portal for Educators, were especially useful to instructors. Based on email interviews with 20 educators (10 at the college level, and 10 at the high school level), we noted an increase in college instructors’ use of CineFiles film-related documents in teaching (at least 3 instructors who had not used documents in the classroom earlier had used the website to find materials for their students, and an additional 5 instructors continued to use documents in support of classroom viewings). At the high school level, teachers had more difficulty using documents in conjunction with screenings, as they did not have the time to research and integrate the materials into the curriculum (2 out of the 10 high school teachers used CineFiles to find documents for their students). However, where Resource Guides were available, more teachers were able to integrate the resources easily (at least 5 teachers used materials suggested by Resource Guides in the classroom).
Over the course of the three-year grant, project staff selected, indexed, and digitally captured 14,006 documents totaling 29,735 page images. Although the number of documents is slightly below our planned goal of 15,000 documents, the percentage of documents meeting our selection criteria is 100%. We also continued to have success in negotiating with copyright holders for permission to display their documents. More than 12,000 of the documents added to CineFiles during the grant period are freely accessible, and an improved method of providing individual copies of documents for private research use was implemented, which has already proved extremely popular.
Website use statistics also demonstrate our achievements. Although overall numbers of users did not significantly increase, users from .edu IP addresses increased by 25 percent. Our improved method of requesting restricted documents was used over 52 times in December alone, with 90% from .edu IP addresses. We will evaluate whether our research benefits managers of other digital imaging projects once this final report is published on the BAM/PFA website in January 2010 with a brief survey.
Unanticipated Circumstances and Lessons Learned
The main unanticipated circumstance we encountered was the three-month delay in completion of the new website that resulted from technical difficulties and staffing cuts at both PFA and at UC Berkeley’s Information Services and Technologies Department, both of which were circumstances beyond our control. We handled this issue by establishing a more realistic timeline, which we were able to meet by undertaking post-redesign surveys and in-class modeling in fall instead of spring 2009.
Some of the lessons learned during the course of the project included the importance of preparation and user feedback, and the difficulty of getting responses to surveys. When surveys were sent out cold, the response rate was no more than 2%. The best response rate was through direct contact with individual educators. In fact, one of the strategies used to solicit early survey responses was through key individuals who were then asked to request three educators they knew (from different institutions, or departments, if possible) to take the survey. This ended up being a very effective way to generate survey responses. We will build additional incentives into future survey plans, to help improve response rates. We learned the importance and effectiveness of in-person interaction to effect learning outcomes through our many focus groups and in-class modeling sessions. Although such interpersonal activities are labor-intensive and therefore costly, they are much more effective at attracting busy educators’ attention and participation than online activities and outreach alone. Another lesson that seems almost unavoidable, is that no matter how much time is allotted for technical/programming activities, these steps always require more time than expected. We also learned more about the different types of curriculum support desired by high school and college instructors.
As a result of rigorous and thorough evaluation, testing, and design procedures, we have succeeded in redesigning and improving an online resource that, due its worldwide accessibility and ease of use, will have a large-scale and long-term impact around the globe as a source for film-related research and classroom materials, as well as a tool for teaching a variety of subjects related to the humanities.
Over the last three years, we have added over 1,000 new titles (and over 14,000 new documents) that are directly relevant to history and social sciences. Over 50% of these are specific titles requested by educators, who have been seeking richer resources to use around these particular films. When we launched the website in September 2009, it was already too late for a number of educators to incorporate new materials into their syllabi for the current semester. The real impact in terms of increased use of the resource among educators will be seen in Spring 2010 and beyond. A number of professors have also expressed an interest in (and one high school instructor has even implemented) a module on research methodology using the CineFiles website.
The resources on CineFiles have also had a direct impact on student learning and performance. One of the key findings from the in-class modeling demonstrated was that students understood visual content better when they watched a film if it was supported by a document that reinforced key concepts and vocabulary. Further, with 75% of the documents open to the public and available to download as PDFs for future use, the potential dispersal of CineFiles resources is enormous.
The impact of the project on the Berkeley Art Museum/Pacific Film Archive has also been tremendous. The project has given us a new awareness of the needs of educators in the 21st century, and will help us respond better to the needs of our users in the long run. Further, the infrastructure update that this project allowed will help us grow the resource in many new and interesting directions in the future.
In accordance with UC Berkeley’s and BAM/PFA’s research and teaching mission, a major long-term goal of the PFA Library is to improve digital access to and preservation of its research collections. Continued development and enrichment of the CineFiles database is key to progressing towards this goal.
With IMLS support over the last three years, PFA has significantly expanded access to film-related resources in the high school and college classroom, and increased multidisciplinary use of such materials. Information gathered throughout this project has helped us focus our future plans on areas of particular interest to our users. The explosion of online access in recent years underlines the fact that institutions need to expand their collections’ online presence in order to remain relevant and sustainable in today’s networked world. But access without context and outreach, especially in this era of web overload, cannot be as effective or focused as contextualized access that responds directly to potential users’ needs.
While simply continuing to digitize and catalog additional film-related documents is a laudable and necessary activity, PFA additionally plans to focus future growth of the project in areas of particular interest to our current and potential users, by providing educational access to moving images themselves and film-related documentation as well as providing deeper context and curriculum support through enhanced resource guides. We shall also research and develop improved methods of gathering information from and responding to users through social media. In these ways, we hope to further augment the effective use of the moving image in all areas of education and research.
Downloadable PDFs of this Report and Appendices
- Sample Survey (Grade 10)
- Results From High School and College Focus Groups
- Results From High School And College Surveys
- User Personae
- Usability Testing Mockup
- List of Usability Concerns
- Excerpt from CineFiles Usability Interview Script
- External Evaluator’s Review from Rockman et. al. (REA)
- Expert Usability Review from Usability Consultant Olga Amuzinskaya
- Usability Testing Results
- Mockup of New Website Design
- Technical Specifications Implementation
- Website Launch Flyer
- Results From First Round of Surveys
- Results From Second Round of Surveys
- Sample Resource Guide
- Sample workshop guide, Representing History, that was offered to high school history teachers in collaboration with KQED
- Curriculum From In-Class Modeling Session