Professional Development

During May 2012...

NC-LITe and WFU Summon Usability Study Findings Presented at Appalachian State University

Tuesday, May 29, 2012 4:13 pm

On Monday, May 21, Hu Womack, Kevin Gilbertson, Lauren Pressley, and I drove to the High Country of Boone to attend NC-LITe. It was doubly wonderful for me because I had the opportunity to travel with three amazing colleagues to one of my favorite places on earth.

It seems hard to take in that it has been seven years since I left my position at Appalachian as an Instruction Librarian. Our family moved from Boone three months before the Library moved into their new facility so it was fun just seeing the building! When we drove up in the parking deck and told the attendant that we were there for NC-LITe, she proclaimed in her authentic mountain brogue, “We finally got us one!” and for me, it felt like I was home.

When we entered the library, Scott Rice greeted us and escorted us to the fourth floor conference room. We were welcomed with a beautiful spread of pastries and coffee. Institutions represented were Salem College, UNC-G, High Point University, Appalachian, and WFU. The intimate group encouraged informal discussion.

The day began with around the table introductions and catch-up. Most of the shared information had to do with strategies for reaching the masses by offering consistent content for specific courses such as Expository Writing and Freshmen Seminars. (We are so fortunate at Wake to be able to tailor our Freshmen Seminars to the content taught.) It was interesting to note that each represented institution is currently using or has plans to use a discovery tool similar to Summon.

After the round table discussions, Kevin, Lauren and I presented our findings from our Summon usability study. Kevin began the discussion by offering some historical perspective regarding how our Library first used Multi-search and then implemented Summon last summer. Our tech team was forced to move quickly in order to get it up and going before Erik Mitchell left. In light of this, the tech team decided it would be particularly helpful to conduct a usability study to see how ZSR’s Summon (the “Everything” link) was being used.

After Kevin’s introduction, we used a clicker presentation to introduce our findings. (Note: having three techies on this trip was a true gift!!) Our usability study was based loosely on NC State’s usability study which was done three years ago and was sponsored by Summon (NC State was a Summon beta site). Their study was led by an outside consultant and their report can be found here:

For our study, we used eight student workers who were available for the afternoons that we conducted the usability study (thank you, ZSR student supervisors for sharing your students with us!). Each individual interview lasted 30-45 minutes and we met in the vacant office of the Instruction Suite (302B). Kevin conducted the interviews while Lauren and I took notes.

We began with what NC State called “Standard Framing” which helps to determine the student’s proficiency at navigating the Library website. This ended up being a very interesting section of the study! Students were asked to go to the Library homepage and asked to find a book about Bill Gates but not by Bill Gates and to tell where it was located in our library. A couple of students thought they had a book about Bill Gates when it was about the Bill and Melinda Gates Foundation. A couple of students did not notice that they had selected an ebook, and at the end of the interview one student asked, “What is an ebook?” One student found a book in the Professional Center Library and did not realize that it was not in the ZSR Library.

Students were given a journal article citation and were asked to locate the article. There were two students who used the “Journals” link to search for the article title and those students could not locate the article (one of those eventually used the “Everything” link and was successful). Two students used the journal names in the journal link and they were successful. One used the “Article” link and she was successful, and one started with the “Everything” link and was successful. One student started with the catalog link and after a long and winding search, located the article.

We found similar results when we asked students to find a scholarly journal article about stress among college students. One student started with the Databases link and another started with the Journals link and each typed in their subjects within the boxes; neither was successful. Students who started with the “Everything” link or the “Articles” link were successful. One student (not the first Database selector) got frustrated with the Databases link and decided to switch to JSTOR because her professor showed it to her once and she knew it was really big.

We then switched to having the students search only using the Everything link (Summon). We asked them to find a book by an author they liked. Two of the students were not successful with their search because their authors were J.D. Salinger and H.G. Wells; many sources came up about these authors, but not by these authors (to be successful, they should have used the Books/eBooks facet). One student searched for “Harry Potter” and there were six entries before a “Harry Potter” book appeared on the list.

Students were asked to find the latest issue of the Journal of the American Chemical Society. None of the students were successful with this search. What happened was that they began with a searches for the [Journal of the American Chemical Society] which produced a list, but the first three entries on that page directed them to Carpenter Library. (To correctly answer this question, they needed to click the fourth entry to see our subscription to the latest articles.) When they were not successful with the first couple of links, they switched to the “journal articles” facet where they lost the ability to sort by relevance. Summon does an excellent job searching by title, but after that, relevance is lost. Several of the students were confident that they had found the answer, when in fact they were looking at articles where the Journal of the American Chemical Society was cited.

We asked the students find to an article from the New York Times from the month and year they were born. Only two students noticed that newspapers were excluded from the Summon search results (it could be turned on from the facets). Two students were simply not successful with this search, two other students thought they were successful but their articles were from New York Times Magazine, and two students were successful because they ended up with links to Proquest Historical Newspapers.

None of the students were successful finding an article on “Is genetically modified “golden rice” safe?” As the interviewers, we really wished the student would use quotation marks to look for strings of words or “Ctrl-F” to search for words in their results. We hope that our LIB100 instructors will start teaching these skills as part of their curriculum!

We asked the students if they would recommend the Everything link to their friends, and five out of eight said they would “if they don’t know where else to start.”

We also asked the students what they would change about the Everything link and they had a few suggestions. The number one suggestion was to make it so that it could do a better job searching by relevance; they worded this concept something like this: If you search for “Harry Potter,” a “Harry Potter” book should come up to the top. A couple of students said that they wished that you could limit the category options before you do your search (searching by author, title, etc.). Another student said that it would help if the results were sorted by format with maybe the books on top followed by the journals.

After our presentation, Margaret Gregor, Appalachian’s Instructional Materials Center Librarian and Scott Rice, Appalachian’s E-Learning Librarian, presented a tool they created to provide virtual tours of the IMC. The students were required to find 16 hidden puzzle pieces and a book in the collection (the students were instructed to write down the bar code of the book they found). Scott set up a temporary password for us so that we could try it out. They reported that over 500 students had completed the tour. Here is a link to this site and I can give you a password that you can use if you are interested:

After Margaret and Scott’s discussion, we then had some group discussions about things we were doing at different institutions. That discussion was followed by lunch at “Our Daily Bread” in downtown Boone and then a tour of the Library.

Overall, it was a great way to spend the day! I’m already looking forward to NC-LITe in the fall!!



North Carolina Summon User Group Meeting

Thursday, May 24, 2012 12:18 pm

On Friday, May 18, Lauren Pressley and I went to Perkins Library at Duke University to attend the inaugural meeting of the North Carolina Summon User Group. My last visit to Perkins Library was 20 years ago, so just getting to visit their beautiful facility and amazing coffee shop was a treat in itself! I was surprised to discover the intentional dearth of signage on their campus and was very thankful that Lauren knew where we were going.

Upon registration, we were greeted by two very friendly Duke librarians and given very nice Serials Solutions portfolios as well as name tags and name table tents. The meeting began with a lunch hosted by Serials Solution. There were about 50 in attendance with several schools represented including NC State, ECU, UNC-CH, Campbell, Duke, and others.

Eddie Neuwirth, Senior Product Manager for Summon, began the program by introducing himself (he lives in Cary!) and his colleague, Vince Pella who is a Customer Service Representative and flew in from Seattle for the meeting. Eddie began by giving an overview about what is new and what’s coming with Summon. He said that Summon is currently being used by 450 libraries in 40 countries and that 38% of the largest Research Libraries in North America are using it. They are currently approaching one billion records with more than 90 content types. They push out new releases every three weeks that can be added by local Summon administrators. We were pleased to note the newest release gave the option for the Widget and Search Box Builder to “Keep Search Refinements.” This was one of our concerns that we noted in our own recent usability study and we were pleased to see that they had addressed this problem! Also, the save and preview items icons are now visible (another concern we noted in our usability study).

He then introduced a new widget that will be rolled out this summer called the “Discipline” widget that uses Ulrich’s categories to group journal titles and it uses call numbers to group books. He noted that not everything is mapped to the Discipline widget and then he gave an example of how to use it by limiting the Discipline widget to Biology and searching for “Blood Cells” (which yielded 600,000+ hits).

He then proceeded to talk about other enhancements such as spotlighting images (digital repositories) and a recommender feature that will suggest LibGuides, databases, etc. The biggest announcement was that they are working on a product targeted for 2013 that will replace the need for OPACs. He gave a sneak preview of what the records will look like and they appear much like our current catalog records. They are also working on A&I citation displays so there will be sneak previews to the abstracts and database records. Eddie also mentioned a product they are developing called Intota that is a web-scale development product that will eventually eliminate the need for integrated library systems as we know them.

After Eddie’s talk, there was an hour of lightning rounds which were five fascinating 10-minute presentations where people from different universities presented research related to their use of Summon. The first up was Patrick Carr from ECU who presented his findings on the impact Summon has had on E-Journal use. He reported that the cost per session came to $.39 and the cost per search came to $.08. They found that the use of Sage and Springer journals increased four times. The use of Elsevier increased 10-15%, but the use of JSTOR was down 10% and the use of EBSCO journals was down significantly. In a later part of the meeting, Eddie explained that the numbers being down in JSTOR had to do with the fact that JSTOR provides limited access to the bulk of their metadata (JSTOR gives Google Scholar full access to all of their metadata). In the end, the download of full-text downloads stayed flat for pre-Summon and post-Summon at 1.4 million.

The next presenter was from Duke and he demonstrated their use of Summon to include their visual collections (scanned items from special collections). They were still working on Summon’s ability to include thumbnails which they anticipated to be working by this week.

For me, the highlight of the lightning rounds was hearing Karen Cicconne from NC State. They were a beta site for Summon and thus have three years of experience under their belts. Karen presented her findings from comparing the results of an EBSCO group cluster search to the same search in Summon. She used the exact wording and punctuation used by students in the EBSCO searches to see the result in Summon. NC State offers a group of course tools for every course taught at NC State, and each course offers an EBSCO cluster at the top of the page. She found that the EBSCO searches had consistently better returns but there were a couple of searches that did better in Summon. (Side note: one of the consistent themes throughout the day was bemoaning Summon’s inability to sort by relevance.) Her next study will be comparing the results from Google Scholar searches with Summon searches. Her initial reaction is that both are good (as a reminder, they are only searching journal articles in their Summon searches).

Next up was Ginny Boyer from ECU’s Joyner Library. She did a survey with ECU’s Health Science librarians to gauge their perception of Summon and its usefulness for the medical field. They found that while most did not use it, they did not feel strongly about it either way. They did a study to find out their proficiency with the tool and found mixed results. There seemed to be little or no success with winning over the Health Science librarians to Summon. The kicker to this presentation came with Karen Cicconne from NC State spoke up and said that research shows that people in the health sciences do not need Summon, they can find everything they need in ChemAbstracts and Medline. The key to success in medical searches is MESH headings.

The last group were Anita Crescenzi and Kim Vassiliadis from UNC’s Health Science library. They did a Summon usability studies that was very elaborate in its execution with 170 applicants vying for the $20 gift card offered in appreciation for 60-90 minutes of their time. Their results were identical to ours (which we did with student assistants in no more than an hour for each session). A couple of their observations: Summon searches titles very well, but loses relevance searching after that. Their students also mentioned the desire to limit results from the beginning and the desire to have the results grouped by format such as journal article or book.

We ended the day with discussions around tables based on common interests. Lauren and I gravitated to the instruction table where we talked with Karen Cicconne (NCSU), Emily Daly (Duke), and Sarah Steele (Campbell). We found that most of us do not teach Summon, but sometimes it is used as an example of a broad search versus a targeted search in a specialized database. Karen gave a very interesting statistic that said that they found that 74% of their students began their searches on their website using the “All” option, with 40% of second clicks going to the Summon articles, 30% going to their catalog, and 30% going somewhere else on the page. If you have time, I suggest that you take a look at NC State’s Library homepage to see how they are implementing Summon:

I apologize for this very long reflection, but as you can see, it was a day packed with fascinating information. They are hoping to make this an annual event and I highly encourage others from ZSR to attend next year!

Professional Development
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
April 2015
March 2015
February 2015
January 2015
December 2014
November 2014
October 2014
August 2014
July 2014
June 2014
May 2014
April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013
January 2013
December 2012
November 2012
October 2012
September 2012
August 2012
July 2012
June 2012
May 2012
April 2012
March 2012
February 2012
January 2012
December 2011
November 2011
October 2011
September 2011
August 2011
July 2011
June 2011
May 2011
April 2011
March 2011
February 2011
January 2011
December 2010
November 2010
October 2010
September 2010
August 2010
July 2010
June 2010
May 2010
April 2010
March 2010
February 2010
January 2010
December 2009
November 2009
October 2009
September 2009
August 2009
July 2009
June 2009
May 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
August 2008
July 2008
June 2008
May 2008
April 2008
March 2008
February 2008
January 2008
November 2007
October 2007
September 2007
August 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
January 2007

Powered by, protected by Akismet. Blog with