Café Jus: an Electronic Journals User Survey

1 Introduction

Electronic scholarly journals have increasingly become the focus for research and development in recent years (see, e.g. Rowland 1995). Most work has concentrated on three topics: technical developments in networking, browsers, search engines and interfaces to provide an acceptably user-friendly electronic environment; the conversion of print-only journals to a dual format (print and electronic); an examination of input and handling requirements. User studies have necessarily lagged behind, since they could not readily be carried out until electronic journals became more generally available. However, one major user study has been reported by Stewart (1996). Much of the early research in this area (e.g. Shackel 1991) simulated, so far as possible, real usage; but the people involved were typically electronic-journal enthusiasts, rather than a cross-section of ordinary journal users.

Alongside and, in a number of cases, preceding the development of dual print/electronic versions of existing journals by commercial publishers, scholars in a range of academic fields have begun to found new, usually electronic-only, journals. Unlike commercial electronic journals, which are normally priced along similar lines to their printed counterparts, or even tied to the print subscription, most of these journals are distributed free of charge over the Internet. In a number of cases, the founders of these journals see them as offering an alternative publication route, in place of some of the increasingly costly specialist products of commercial publishers (e.g. Harnad 1994, Odlyzko 1996). However, it has been observed that most of the free electronic journals have so far attracted rather few submissions, and therefore publish relatively few articles per year (Harter and Kim 1996). It should also be noted that the two models discussed here -- free journals on the Internet, and electronic versions of existing commercially published journals, sold on subscription, under site licensing arrangements, or by 'pay-per-view' -- are not the only possible models of scholarly communication systems. A full discussion of many possibilities is given in Okerson and O'Donnell (1995). The two discussed here actually exist, however, and are therefore susceptible to user studies.

The rapidly expanding numbers of both commercial and free electronic journals have emphasised the need for user studies, whilst also increasing the possibility of carrying them out. Many libraries are now taking steps to provide their users with access to relevant electronic journals. One pressing question, as discussed by Kidd (1997), is how best to do this. In 1995, Loughborough University Library began to make available, via its World Wide Web pages, a collection of links to electronic journals. Titles were selected initially from the free journals found on the Internet. The criteria for choice were that the journals were, first, peer-reviewed and, second, related to the teaching or research interests in the University. At the same time, it became clear that there would be a major growth in the availability of electronic versions of commercially produced journals during 1996 (Woodward 1997). The Library began negotiations with publishers to allow trial use of some of their electronic journals, but this initiative was overtaken, in part, by a new project started by the Higher Education Funding Councils (HEFCs) in the UK. Their Pilot Site Licence Initiative (PSLI) was designed to allow access by universities to the electronic versions of journals produced by the Institute of Physics and Academic Press (and, subsequently, the two Blackwell imprints) on favourable terms (Bekhradnia 1995). In addition, Chapman & Hall allowed Loughborough University free access to its electronic journals for the duration of the Café Jus project.

As a consequence of these developments, a total of some 300 refereed electronic journals were available to staff and students at Loughborough by mid-1996. The Library restructured its Web electronic journal pages to take account of this increased number and diversity of types. Separate pages were created for free electronic journals, free sample issues of electronic journals to which the University did not subscribe, the prototype Blackwell's Navigator service, and for the commercial electronic journals to which the University did subscribe. The last-named page had links to the home pages of each of the publishers involved. (This was because each commercial publisher had its own access control system which required entry via its own home page.)

By 1996, conditions at Loughborough were appropriate for implementing a detailed study of users' reactions to electronic journals. A range of titles was now available across a variety of subjects, with easy access from anywhere on the campus. In anticipation of this development, the Café Jus (Commercial and Free Electronic Journals User Study) project was initiated in 1995, with funding from the British Library Research and Innovation Centre. Its main aim was to examine what problems readers with differing subject and computing backgrounds experienced when using electronic journals. A supplementary aim was to compare readers' reactions to free and commercial electronic journals. The freely available journals can generally be accessed by a direct hypertext link from the library's Web page to the journal's own home page, and they are usually in HTML; commercial journals require registration on the publisher's home page, and the full texts are generally in PDF. Were the former considered as on a par with the latter by academic readers?

2 Methodology

In carrying out the study, it was decided to concentrate primarily on students taking master's courses. The rationale was that they formed a group which was expected to make considerable use of journals, but had not yet developed set ways of acquiring information from them. In addition, volunteers from these courses were expected to provide a sufficient number of responses to make inter-subject comparison viable.

A structured questionnaire was developed which could be used in conjunction with hands-on access to an electronic journal. It is shown in Appendix 1 (**if you have Acrobat Reader, go directly to Appendix 1 ). The questions were arranged under three headings - journal content, journal appearance and facilities offered by the journal. This questionnaire was tested with volunteers from the 1995/96 cohort of master's students in four departments - Computer Studies, Human Sciences, Information and Library Studies, Physical Education & Sports Science. The students were introduced to the electronic journals in groups, which met in the computer-based teaching laboratories of the Department of Information & Library Studies. An initial presentation on the project together with documentation was provided; then each student accessed an electronic journal, noting down responses to the questionnaire as their reading progressed. If they had time, they could move on to another journal. Dealing with the students in groups made it possible to provide advice and help in real time, and also ensured completion of the questionnaires.

The pilot study indicated that the planned organization of the investigation was acceptable, but that there were minor ambiguities in some of the questions (more especially those dealing with journal facilities). After modifications to the questionnaire, the main study was launched with master's students from the 1996/97 cohort. The same four departments provided students, along with two more - Civil Engineering and Economics. Appendix 2 shows the full lists of titles accessed by at least one student during the pilot and main studies.

For comparison, it was decided to obtain feedback on electronic journal usage from research students and academic staff. In this part of the project, users were given individual instruction in the use of electronic journals, and then left to use them on their own at their own desks. The method of obtaining data here was via a log sheet on which participants were asked to keep a record of their usage of electronic journals. Volunteers were sought mainly via electronic mail messages inviting participation, but this proved to be a good deal less effective as a recruiting tool than the group approach used with master's students.

3 Results

The pilot study involved 47 master's students, who returned 55 questionnaires. There was a fairly even gender balance (male - 27; female - 20) and UK/overseas geographical distribution (UK - 27; overseas - 20). The total number of electronic journal titles accessed was 13, seven of which were commercial and the remainder free. The main study involved a further 75 master's students, who returned 85 questionnaires. The gender balance was again fairly equal (male - 40; female - 35), but there was a lower proportion of overseas students (UK - 56; overseas - 19). Altogether 34 electronic journal titles were accessed, of which 24 were commercial. The shift towards commercial titles reflects primarily the growth in the number of these available between the pilot and the main study. In the pilot study, some 60% of the respondents did not regard the electronic journal they had scrutinized as being particularly relevant to their interests. By the time of the main study, more commercial titles had come on-stream, and only about 20% of the respondents now regarded the journals they looked at as not being particularly relevant to their interests.

One change made between the pilot and the main study was to introduce a question about the computer expertise of the respondents. This was to help identify any major differences between groups of students that might affect their handling of electronic journals. The overall distribution of expertise in the main study was found to be: limited expertise - 20%; moderate expertise - 50%; considerable expertise - 30%. There was no significant difference in this distribution for male/female or UK/overseas groups. In subject terms, only Computer Studies - where all students had considerable expertise - differed from the norm.

Apart from the master's students, feedback was obtained from eight staff on 38 attempts they made to access electronic journals, and on 29 attempts made by four research students. In both cases, the great majority of the accesses were to commercial journals. Overall, staff and research students were more experienced than the master's students both in using journals and in handling information technology.

3.1 Results: master's students

3.1.1 Relevance and quality

Respondents were asked to compare the overall quality of the articles in the commercial and free electronic journals they were examining with articles in the printed journals they knew. In the pilot study, the standards of commercial and free journals was judged to be much the same. However, in the main study, about 40% of the respondents thought articles in free journals to be of a somewhat lower quality than those in printed journals (as compared with only 7% saying this of commercial electronic journals). These figures have to be treated with caution. In the pilot studies, 58% of the responses related to commercial journals, compared with 42% related to free journals. In the main study, the imbalance was considerably larger: 80% related to commercial and only 20% to free journals. Consequently, the differences as regards quality are less well established in statistical terms than the figures might suggest. In addition, judging from student comments, a particular difficulty with free journals related to the range of their contents. Free electronic journals might include - along with research articles - news items, discussion, etc. The students tended to group these together with the articles, leading them to a lower estimate of the overall quality of the journal.

It is not surprising that the contents of commercial online journals were thought to be of similar standing to those of printed journals, since they were essentially parallel publications of the same material. The likelihood of being acquainted with the electronic journal increased between the pilot and the main study, in step with the increasing number of titles becoming available. The number of respondents who already read the printed version of the electronic journal they were examining almost doubled from the pilot study (18%) to the main study (32%).

3.1.2 Access

There were continuing problems in gaining speedy access to electronic journals. These almost entirely concerned the commercial titles. In the pilot study, 9% of the respondents experienced problems in accessing free journals compared with 48% of those accessing commercial journals. The corresponding figures in the main study were 13% and 33%. In the pilot study especially, though not exclusively, the IDEAL system (used for Academic Press' electronic journals) proved to be far from ideal. There could be considerable delays in gaining access at any time of the working day, but it was particularly evident for American connections after lunch in the UK. The existence of alternative sites did not seem to improve matters. For example, one student was told to log into the US San Jose site because there were too many connections to the UK Bath site. When he tried, he was refused entry to the San Jose site and told to log into Bath.

Most students expected to be able to download full-text articles that interested them to the screen. This often proved to be time-consuming. Some longer articles took so long to download that the session timed out, and students had to log in again. In its favour, IDEAL gives file sizes, and this assisted students in deciding whether or not to download. Not all the publishers provide this kind of help.

The following comments by students, in answering a question about whether they encountered difficulties, indicate the type of access problems that were found.

• "Generally no, although the fact that the IDEAL username box isn't long enough for 'Loughborough' to fit in could cause confusion."
• "Finding it [the publisher's electronic journal site] was relatively easy, but it did take some time to load - about 5 minutes [and this was at 9:00 am!] It then took another few minutes to get from the IDEAL menu to the listing of journals."
• "Yes! Nowhere on the front screen tells you to press the SUBMIT button to get into the system. I presumed you just had to hit the ENTER key. It took 10 - 15 minutes to get to the first journal article!"
• "I could not get into the full text of one article; the message that came up was 'Adobe Type Manager 3.6. or newer, must be installed. Acrobat Reader will now quit.' This happened twice. We started to access the system at 2:00 pm; it is now 2:50 pm and I have not been able to see a full text article."
• "No, but we did have assistance."

These and similar comments can be generalized into the following complaints:

• The means for gaining entry to an electronic journal can be far from obvious, and trying to guess how to obtain access can be time-consuming.
• There is a need for prior knowledge (e.g. who the publisher is; what password should be used).
• Not all journal titles immediately indicate the limits or nature of the journal coverage (e.g. PACS-R), and a reader cannot gain an intuitive picture as with a printed journal.
• Users dislike publishers who assume that all readers have access to the latest software. They especially detest publishers who upgrade their interface requirements without adequate prior notice.
• Most of the users felt that they would have had problems accessing an electronic journal for the first time without additional guidance.

3.1.3 Layout and navigation

The most frequent queries under this heading related to legibility, followed by the comprehensibility of the graphics. There was also only muted enthusiasm for the ease of examining tabular material and for the way colour was used to assist comprehension. (Generally similar results were found in the pilot survey.) In effect, less than half of the respondents felt that deciphering the journal contents was made as easy as possible. The problems were partly related to the Acrobat reader and its relationship with Netscape. Some students commented on this directly. For example, a number tried to improve legibility by enlarging the text. One student commented that this worked, "if you understand the [Acrobat] zoom function, which is definitely not intuitive!"

Students clearly had some difficulty in understanding how some features operated. To look at this further, it was necessary to look at titles which had been examined by a number of readers independently. The two titles selected for this purpose were Construction Management and Economics and Journal of Sports Sciences. An analysis of these showed that most confusion reigned over the question of whether hypertext links existed within articles. There was also some uncertainty in replies relating to the presence of hypertext links to other documents, whether the journal allowed browsing through tables and diagrams, and whether it supported searching across articles.

Although most students regarded the various icons as easy to use, there was often some confusion until they became accustomed to them. One problem was that the icon might not immediately call to mind the operation concerned. For example, some failed to guess that a man with a spanner represented 'services'. Another problem was misinterpretation of the activity involved. For example, some expected the 'back' button to return them to the contents page rather than the previous page of text. It was also not immediately clear to many how to exit from an article in Acrobat. In addition, the help facilities were not found to be very helpful. Indeed, a number of the students did not manage to find them. As one commented, "Could not find them - they should be obvious to novice users!"

The various problems encountered by groups of users trying to read the electronic journals can best be indicated by the account appended below (in abbreviated form) of a session undertaken with a group of Computer Studies students. The session was more fraught than most, even though the students had considerably more computer expertise than the average.

The first problem in this session was that one of the commercial publishers had changed the layout and appearance of its site (without prior notification). One student was thrown off the system completely with the message, 'System error - Netscape Type 2.02', and had to log in again. Another student said he had been going round and round trying to find the search engine. The alphabetical list of journals was squashed into the bottom left-hand corner of the screen, and was almost too small to be read. In addition, the green lettering on a white background proved difficult to decipher. Another student got into the Home page of the selected journal, but could not find the contents pages, nor could he find a way of accessing a document. This problem was finally resolved by a telephone call to the publisher. A further student received the message, 'The file ATM Font Database must be in the system folder. Please re-install ATM. Acrobat Reader will now quit.' He thought this meant he should move to another machine, but it was suggested he should try another document on the same machine. He got into that satisfactorily, but then received the message, 'An error occurred and has been logged.' He then found that he could not retrieve the article he wanted. After several further attempts, he commented, 'This is getting on my nerves.' A fellow student encountered the same error message, and tried every journal on the list. The same thing happened each time. He tried logging in again, and failed to gain access at all. Several students wanted to print articles, but had to wait so long that they eventually gave up.

3.1.4 Perceived advantages to the electronic journal

• Finding specific information (e.g. appropriate articles) is seen as being easier than for printed journals. Student comments included:
• "The electronic version has excellent searching capabilities - that feature is extremely helpful and would save time in research."
• "Easier to find related articles."
• Electronic journals avoid some of the problems of using printed journals. Student comments included:
• "Can manipulate the electronic version: can view lots of journals without having to go out of your mind trying to find all the paper versions in the library."
• "The advantage of the electronic version is that it is readily available at any time as compared to the paper version, which may be in use by another person when you want to read it."

3.1.5 Perceived disadvantages to the electronic journal

• Electronic journals are not always easily usable by novices. Student comments included:
• "There are no instructions or guidelines immediately accessible to help you find out how to use the electronic journal, and to move around the system. To be honest, it's not very user-friendly."
• Some operations take more time. Student comments included:
• "It is easier to turn the page of paper journal."
• "At the moment it is too slow to connect, and disconnection occurs too easily."
• "Takes too long to move from one journal to another."
• Reading from the screen is less pleasant. Student comments included
• "To read a screen is tiring."
• "The graphics were very grey and boring."

The predominant view was that electronic journals were already useful for reading from a site with no nearby library, or for quick scanning and searching. However, they still need appreciable improvement if they are to compete with paper journals more generally. As one student commented: "Need more user trials before implementation. Avoid the 'Big Bang' introduction of the system. Patience!"

3.1.6 Comments from discussions in sessions

There were a number of comments relating directly to the publishers' approach and particularly relevant to the present project.

• The use of different passwords for different publishers was confusing.
• Some journal material was in a different format depending on when it had been input, with the older material being more restricted in its form of presentation.
• Some journals had been scanned in. This led to a generally indistinct appearance of text and graphics on screen.
• Changes made by publishers without warning are difficult even for computer-literate readers to disentangle.
• The more complex the journal, the greater the potential software problems. For example, one engineering journal required additional software (the Shockwave plug-in for Netscape) in order to view the animation. The lab machines had the wrong version, so the students lost the value-added element of this journal.
• Most respondents did not understand 'bookmarks' and 'thumbnails'.

Some of the colours used, especially for hypertext links, were difficult to see on-screen (e.g. a combination of green and brown, which was particularly unsuitable for readers with colour deficiencies).

Apart from these, the overwhelming comment was that accessing and using many of the electronic journals (primarily the commercial ones) would have been difficult, or even impossible, without the handouts and oral guidance given during the sessions.

3.1.7 Comparison of electronic and printed journals

Two-thirds of the respondents saw the electronic versions as offering, in general terms, easier access to journals. They meant by this that all the journals they wanted could, in principle, be accessed from the same terminal. This removed the need to set aside special times for visiting the library, when they might find that the journals they wanted were already in use. When asked which type of journal they found it easier to use, a somewhat smaller proportion (59%) chose the printed journal in preference to the electronic journal. Computer-oriented students, especially in Information and Library Studies, claimed to find electronic journals easier to handle. (These figures are derived from the main study, but the responses in the pilot study were very similar.)

3.2 Results: research students and staff

For the most part, the assessment of electronic journals by research students and staff proved to be very similar to that derived from master's students. The following points were either new or more strongly emphasised by this group.

3.2.1 Access

On several occasions, the potential reader either failed to gain access, or did so only after considerable effort. As one respondent commented: "Online access remains inefficient and slow. It would often be quicker to go to the library, indeed to send a carrier pigeon, than to spend countless hours watching a picture of an egg-timer, especially when the end result is an error message, one of an infinite variety of reasons material cannot be accessed, or gibberish." There was a strong feeling that the average first-time user would be unlikely to persevere unaided. Sometimes, even identifying the contents could be something of a feat. As another respondent remarked: "Without obtaining help from the Library, I think I would have given up before even finding the contents page." In addition, readers were occasionally ejected unexpectedly from the journal they were reading.

The provision of a list of journals grouped into subject categories was not found very helpful. Complaints ranged from an excessive number of subject headings to the inclusion of irrelevant titles. It was felt that access requirements overall were far from intuitive (this included use of Adobe Acrobat).

3.2.2 Format

Most complaints related to illegibility. For example, the typeface was often described as too small, and, especially on a lower-resolution screen, as indecipherable. One particular point was the difficulty of reading mathematical formulae on-screen. The use by publishers of their Web site for trying to market other products along with providing access to their journals was described as irritating. One reader who had thought of embedding his own hypertext links in an article decided that the PDF format precluded this. Slow response rates were generally remarked. For example, graphics in headers obviously slow transfer. If publishers insist in including such material, they should at least make it clear that it can be switched off.

One summary comment on access and format may be quoted: "I wish that [a specific commercial publisher] would go away and die somewhere - or at least their e-journal Web designers!"

3.2.3 Printout

One of the points made about legibility was that, if the typeface was too small, the natural response of the reader was to enlarge the portion of the text concerned. However, this left too little text on the screen to be easily readable. Consequently, the natural response was to read a print-out instead. All the staff involved said they would normally print out material for continuous reading. The exception was if the article contained important hypertext links. One reader found an issue of a journal where most of the articles were of interest. She did not have the time to read them all at work: at the same time, there were too many to print. She commented that this was an instance where a printed version of the journal would have been preferable.

In almost all cases, the time required for downloading and printing material was regarded as excessive. The explanation for this seems to lie in the large size of PDF files -- for most papers , 0.5-1.0 Mbyte -- and congestion on the network. One respondent noted that he allowed for this by making the process a background activity, while he continued with all his other computer activities.

Research students essentially echoed the comments of the staff, but with two particular emphases of their own. One related to back issues. Research students are particularly concerned with in-depth coverage of the literature. Hence they were frustrated by the paucity of back issues, and by the problems of searching across back issues where they did exist. The other query concerned the introduction of non-essential links, often to general Web sites, in the text of an article. A conscientious student who worked systematically through these would, it was noted, soon become irritated by the waste of time. One example quoted was an article about archival materials in the White House. The author had written: 'My day job is at the University of Michigan.' The link related neither to the subject of the article, nor in any specific sense to the author.

4 Conclusions

Though computer experts may be at some advantage in coping with the idiosyncrasies of electronic journals, none of the factors - computer expertise, gender, status and home/overseas background - had any relation with the identification of problems. This similarity of opinion of all groups in their assessment means that firm conclusions can be drawn on the basis of the present survey. The major points to be made are as follows.

• Low-level problems, especially delays in gaining access and impediments to moving about within and between journals, are a major demotivating factor in the use of electronic journals. Some of these problems are within the control of the publisher: not least, the nature of the interface. Others are not. For example, Academic Press improved server access efficiency appreciably between the pilot and the main studies. However, traffic between the USA and the UK grew, so delays from this cause continued.
• Lengthy on-screen reading, when an article has to be read in depth, remains unpopular. This is not helped by less than optimum decipherability of on-screen text and graphics. Such illegibility is something that publishers should be able to change without excessive difficulty. Efficient and rapid printing capabilities are vital, and will continue to be so for the foreseeable future. Printing out from electronic journals currently leaves much to be desired, especially in terms of delays, though the problems are reduced for those with access to Postscript printers. In principle, this can be helped by appropriate arrangements for local caches of the most-used journal issues, as is now normal practice at Loughborough.
• In terms of access, free journals tend to be more favourably assessed than commercial journals. This seems to be largely due to the fact that they need nothing more than a Web browser. Although it is a good printing device, the Acrobat software used by several commercial publishers does not yet adequately support user-friendly journal usage. The longer history of a number of free electronic journals, along with the dedicated attention they receive from their progenitors, means that, for them, some of the teething problems for first-time readers have already been ironed out. Respondents in this survey assessed the quality of free journals as somewhat lower than that of commercial journals. It seems from the comments that this was partly because free journals looked somewhat different, and had a different mix of contents from the commercial journals. Purely in terms of reader acceptance, there seems to be no reason why well-edited free journals should not compete with commercial journals for the attention of scholars.
• Commercial publishers currently appear to be more interested in pursuing their own learning curve than in assisting their customers. After several years of uncertainty about their involvement in electronic journals, they are now finally taking the plunge - only, it seems, to make the classic mistake of allowing technology, rather than human factors, to drive their efforts. They not only seem to have rather little knowledge of how readers operate in an electronic environment, they also seem not to be building on knowledge long since gained from the existing print-based environment.
• Use of electronic journals is going to require a considerable training exercise, especially if publishers keep changing what they are doing. Subscription agents and other intermediaries, such as BIDS and OCLC, are beginning to provide common interfaces for several publishers' journals. If they succeed this will mitigate some of the user problems identified in this study, but not all. The remaining difficulties facing readers are such that any future scenario must surely envisage the provision of more assistance and training by librarians and information scientists in their various institutions. Furthermore, the need for training will be continuous. To read print materials, a person has to learn to read once. But the continuous change in hardware, software, systems and interfaces means that in the future users will need to relearn frequently how to access the literature.
• One important factor for readers of electronic journals may be the rapidity of change, as differences found between the pilot and main studies of this project suggest. On the one hand, this is leading to the electronic provision of an increasing number of titles, which is evidently a plus factor. On the other, it is leading to continuing change in the interfaces. Though increasing sophistication is to be welcomed, change in an interface is quite likely, in the first instance, to make it less user-friendly. In addition, a reader who had adapted to one interface may be required to relearn what to do. Continuing change is therefore unlikely to be good news for readers who are concerned with the content of journals, rather than their handling. A comment on electronic journals from one of the postgraduate students at Loughborough may be relevant here:
• "They are interesting to play with, but normal people read printed journals."

It is possible to look at these comments in a different way: they highlight the actions that publishers of electronic journals need to take in order to improve the acceptability of their product among readers. The survey results are therefore reconstructed below in terms of guidelines to publishers.

1. Do not launch an electronic-journal service publicly without testing it with a pilot group of real users first; take note of their experiences in using it to redesign it before public launch. Do not allow yourself to be pushed by the technologists; remember the user.
2. Do not have too many graphics in your Web page design. Although it is possible for users to disable graphics in their Web browser, näive users will not know that, and will suffer low delivery speeds owing to the graphics.
3. Do not have unnecessary extra steps in the chain of Web pages leading from your home page to the actual full-text articles. At busy times each page in the chain may take several seconds to deliver.
4. Once the service is launched, do not make frequent changes to the interface or software. If you must change them, test the new versions again as in (1) above, give plenty of warning on your Web pages that they are going to change on a certain date, and leave an explanatory note about the changes on your home page for some time after making the changes.
5. Make journals available through all the various 'one-stop shop' vendors now being developed. In this way, each university can choose one such vendor as standard, and then be able to present all its electronic journals to its users with a standard interface and a single password. If some publishers boycott the one-stop shops, this advantage is lost and the progress of electronic journals in general will be retarded, to the detriment of all publishers.
6. Think seriously about the format in which full texts are presented. Adobe Acrobat/PDF is the current de facto standard, but it is slow to deliver articles, its interface is not as intuitive as it should be, and page integrity means that the article pages are the wrong shape for a VDU screen. Double-column layouts are particularly troublesome for users. PDF is a good document delivery system, and will probably remain standard during the transitional dual-publishing era, but for real electronic publishing, HTML, SGML, HyTime or XML will ultimately be used.

In terms of future development, the results of this study suggest that publishers face three categories of problem depending on the factors involved.

• Factors which are under the control of the publisher. These include most matters relating to design. Proper use of colour and good legibility of text are two obvious examples. Similarly, most of the navigational difficulties mentioned previously could be alleviated if publishers devoted more attention to them. Some needs, such as better backrun provision, may be costly for an individual publisher to resolve. Cooperative activity (e.g. in the case of backruns, with libraries, or with projects like JSTOR (1998)) might be appropriate.
• Factors which are partly under the control of the publisher. In these, cooperation is likely to be particularly important. For example, the inconvenience of retrieving journals by publisher's name and password might be resolved by cooperation between publishers and subscription agents. The overriding problem in this category, however, is how publishers will handle continually changing software/hardware without requiring their readers continually to learn new methods. In particular, changes should be downwardly compatible. To the extent that this implies a need for standards, it again requires increased cooperation on the part of publishers.
• Factors which are not under the control of the publisher. Network delays provide an obvious example in this category. Increasing traffic and inadequate bandwidth are a general problem in which scholarly publishers have only a limited influence.
• The present survey suggests that free electronic journals are not seen as a totally different type of entity from commercially produced electronic journals. Their lower assessment stems more from their deviations from the traditional scholarly journal pattern. Judging by developments so far, subscriptions to commercially produced electronic journals will be of the same order as to their print equivalents. Putting these two points together suggests that, for specialist topics with a restricted audience, low or zero-subscription electronic journals produced by groups of academics may well prove competitive with commercially published electronic journals, so long as they follow an appropriately scholarly approach.