[Home] [Databases] [World Law] [Multidatabase Search] [Help] [Feedback]
	England and Wales High Court (Patents Court) Decisions
You are here: BAILII >> Databases >> England and Wales High Court (Patents Court) Decisions >> HTC Europe Co Ltd v Apple Inc [2012] EWHC 1789 (Pat) (04 July 2012) URL: http://www.bailii.org/ew/cases/EWHC/Patents/2012/1789.html Cite as: [2012] EWHC 1789 (Pat)

[New search] [Printable RTF version] [Help]

IN THE HIGH COURT OF JUSTICE
CHANCERY DIVISION
PATENTS COURT

		Royal Courts of Justice Rolls Building, EC4A 1NL
		04/07/2012

Mr Justice Floyd :

Introduction

1. These three actions concern four patents owned by Apple Inc. ("Apple"): European Patents Nos. 2 098 948; 2 964 022; 2 059 868; and 1 168 859. For convenience I will refer to the patents by the last three digits of their numbers. Two of the actions (HC11C02703 and HC11C02826) were commenced by HTC Europe Co. Ltd (a UK company) as applications for revocation of the 022, 868 and 859 patents. In response, Apple sued HTC Corporation, a Taiwanese company, in a third action (HC11C0380) for infringement of those patents and the 948 patent. HTC Corporation counterclaimed for revocation of the 948 patent. I will refer to HTC Europe Co. Ltd and HTC Corporation together as "HTC". The trial of the actions therefore proceeded as if Apple were claimant and HTC defendant, with Apple opening the case and calling its evidence first. The evidence was called in three tranches: first 948, then 022 and 868, and finally 859. I heard some final speeches before the evidence on 859 was called. Mr Burkill QC argued the cases for Apple on 948 and 859, opposed by Mr Tappin QC for HTC on 948 and Mr Alexander QC for HTC on 859. Mr Thorley QC for Apple and Mr Meade QC for HTC argued the cases on 022 and 868. Apple's junior counsel was Mr Delaney; HTC's were Mr Lykiardopoulos, Mr Abrahams and Ms Jamal. I am extremely grateful to all counsel and solicitors for their highly skilled presentation of these cases. I shall have something to say about the time estimates for the hearing at the end of this judgment.

Legal principles

2. Construction. In Kirin Amgen v TKT [2005] RPC 9 the House of Lords explained that the determination of the extent of protection only involves asking what a person skilled in the art would have understood the patentee to have used the language of the claim to mean. Guidelines to assist the court in construing the patent are summarised by the Court of Appeal in Virgin Atlantic v Premium Aircraft [2010] FSR 10 at paragraph 5.
3. Novelty. The law is set out in the speech of Lord Hoffmann in Synthon v SKB [2006] RPC 10 at [22]. To deprive a claim of novelty, the prior document must contain clear and unmistakeable directions to do or make something which falls within the scope of that claim, and the disclosure must be enabling in the relevant sense.
4. Obviousness. In Conor v Angiotech [2008] UKHL 49; [2008] RPC 28 at [42] Lord Hoffmann approved the following statement by Kitchin J in Generics (UK) Ltd v H Lundbeck A/S [2007] RPC 32 at [72]:

"The question of obviousness must be considered on the facts of each case. The court must consider the weight to be attached to any particular factor in the light of all the relevant circumstances. These may include such matters as the motive to find a solution to the problem the patent addresses, the number and extent of the possible avenues of research, the effort involved in pursuing them and the expectation of success."

5. It is convenient to address the question of obviousness by using the structured approach explained by the Court of Appeal in Pozzoli v BDMO [2007] EWCA Civ 588; [2007] FSR 37. This involves the following steps:

"(1)(a) Identify the notional 'person skilled in the art'.

(b) Identify the relevant common general knowledge of that person.

(2) Identify the inventive concept of the claim in question or, if that cannot readily be done, construe it.

(3) Identify what, if any, differences exist between the matter cited as forming part of the "state of the art" and the inventive concept of the claim or the claim as construed.

(4) Ask whether, when viewed without any knowledge of the alleged invention as claimed: do those differences constitute steps which would have been obvious to the person skilled in the art or do they require any degree of invention?"

6. Common general knowledge. In relation to information in documents, the Court of Appeal in General Tire v Firestone [1972] RPC 457, noted at pages 482-3 the statement of Luxmoore J in British Acoustic Films that:

"A piece of particular knowledge as disclosed in a scientific paper does not become common general knowledge merely because it is widely read, and still less because it is widely circulated. Such a piece of knowledge only becomes general knowledge when it is generally known and [accepted without question] by the bulk of those who are engaged in the particular art; in other words, when it becomes part of their common stock of knowledge relating to the art" (square brackets added)

7. Whilst the Court of Appeal was not prepared to endorse the words "accepted without question" in the above citation, they were content with "generally regarded as a good basis for further action".
8. Both Mr Thorley and Mr Burkill reminded me of the considerations which apply in the case of an attack on lack of inventive step which starts from the common general knowledge alone, without reference to any particular citation. I set out some of these considerations in ratiopharm v Napp [2009] RPC 11 at [155] to [159].
9. Added subject matter. In Vector Corp v Glatt Air Techniques Ltd [2007] EWCA Civ 805, [2008] RPC 10, Jacob LJ approved his own earlier statement (as Jacob J) in Richardson-Vicks' Patent [1995] RPC 568 at 576 where he summarised the rule against added matter in a single sentence:

"I think the test of added matter is whether a skilled man would, upon looking at the amended specification, learn anything about the invention which he could not learn from the unamended specification."

10. Although this simple principle has been much elaborated in its application to particular types of amendment between application and granted patent, it is sufficient for the issue which arises in the present case. I would only add that it is always important to bear in mind that a claim may be broadened so as to cover additional subject matter without necessarily disclosing anything new.
11. Excluded subject matter. Article 52 of the European Patent Convention, which is given effect to by section 1(2) of the Patents Act 1977 provides:

"(1) European patents shall be granted for any inventions which are susceptible of industrial application, which are new and which involve an inventive step.

(2) The following in particular shall not be regarded as inventions within the meaning of paragraph 1:

…

(c) … programs for computers;

(d) presentations of information.

(3) The provisions of paragraph 2 shall exclude patentability of the subject-matter or activities referred to in that provision only to the extent to which a European patent application or European patent relates to such subject-matter or activities as such."

12. The law on this topic has been explained in two decisions of the Court of Appeal: Aerotel v Telco Holdings and Macrossan's Application [2006] EWCA Civ 1371; [2007] RPC 7 and Symbian v Comptroller General of Patents [2009] RPC 1.
13. In Aerotel the Court of Appeal set out a four step approach to deciding cases where the exclusions from patentability were engaged:

"(1) properly construe the claim

(2) identify the actual contribution;

(3) ask whether it falls solely within the excluded subject matter;

(4) check whether the actual or alleged contribution is actually technical in nature".

14. At [43] - [44] in Aerotel, Jacob LJ cites with apparent approval a submission made by counsel for the Comptroller as to how one identified the actual or alleged contribution for the purposes of steps (2), (3) and (4). It involves asking what the inventor has really added to human knowledge:

"The second step—identify the contribution—is said to be more problematical. How do you assess the contribution? Mr Birss submits the test is workable—it is an exercise in judgment probably involving the problem said to be solved, how the invention works, what its advantages are. What has the inventor really added to human knowledge perhaps best sums up the exercise. The formulation involves looking at substance not form—which is surely what the legislator intended.

Mr Birss added the words "or alleged contribution" in his formulation of the second step. That will do at the application stage—where the Office must generally perforce accept what the inventor says is his contribution. It cannot actually be conclusive, however. If an inventor claims a computer when programmed with his new program, it will not assist him if he alleges wrongly that he has invented the computer itself, even if he specifies all the detailed elements of a computer in his claim. In the end the test must be what contribution has actually been made, not what the inventor says he has made."

15. In Gemstar TV Guide International v Virgin Media Ltd [2009] EWHC 3068 (Ch) at [37], Mann J left open the question of the appropriate "baseline" for the purposes of determining the contribution: was it any cited prior art, or only common general knowledge? Although I did not hear full argument on this point, it seems to me that the baseline is defined by any item of prior art admissible for a novelty attack. As the quotation from Aerotel makes clear, the contribution which the English jurisprudence requires the court to consider is the actual addition to human knowledge, not the "alleged" contribution which one would discern from a reading of the patent specification. If it were the latter, then I can conceive of an argument along the lines that the skilled person would assess the alleged contribution in the light of his own common general knowledge. Once one is assessing a real contribution, however, it would seem odd not to take account of the whole, real state of the art (that is to say ignoring the deemed state of the art for novelty purposes under section 2(2) of the Act). The exercise of determining the contribution should in principle be the same as that involved in determining the difference between the prior art and the inventive concept for the purposes of obviousness. To ignore, as Apple invited me to do, the state of the art which does not form part of the common general knowledge seems to me to be entirely artificial, not least because the concept of common general knowledge is not a concept which appears in the Act or the EPC. Such a distinction would mean that an invention which was not novel nevertheless made a contribution to human knowledge, because the novelty destroying document was not part of the common general knowledge. I do not think that is what the cases, or the EPC, intended.
16. In Symbian the Court of Appeal commended, at [48] onwards, the guidance given in a number of prior English and EPO authorities on the meaning of the term "technical" for the purposes of applying the computer program exclusion. However the Court of Appeal declined to formulate any "bright line" test for what did and for what did not amount to a technical contribution in this field. Each case had to be decided by reference to its own particular facts and features, bearing in mind the guidance given in the decisions mentioned.
17. In AT&T Knowledge Ventures [2009] EWHC 343 (Pat), Lewison J (as he then was) helpfully analysed the guidance to be obtained from the authorities on the Court of Appeal's recommended reading list. He agreed with the Court of Appeal that it was impossible to define the meaning of "technical" in this context but considered that there were a number of signposts to what amounted to a relevant technical effect. These he set out at [40]:

"i) whether the claimed technical effect has a technical effect on a process which is carried on outside the computer;

ii) whether the claimed technical effect operates at the level of the architecture of the computer; that is to say whether the effect is produced irrespective of the data being processed or the applications being run;

iii) whether the claimed technical effect results in the computer being made to operate in a new way;

iv) whether there is an increase in the speed or reliability of the computer;

v) whether the perceived problem is overcome by the claimed invention as opposed to merely being circumvented."

18. There is less authority on the question of presentations of information. In Gemstar Mann J said:

"what achieves patentability is some real world technical achievement outside the information itself"

19. Whilst the European Patent Office has at times appeared to take a different view on this area of the law, the parties are agreed for present purposes that I am bound by and should apply the principles laid down in these cases.
20. The present case involves the computer programs and presentations of information (as such) exclusions.

The 948 patent

21. 948 is entitled "Touch event model". It has a priority date of 4^th March 2008. In broad terms it is concerned with computer devices with inputs which are multi-touch enabled, that is to say they are capable of responding to more than one touch at the same time.

Technical background

22. Although multi-touch devices of various kinds had been known since the early 1980s, they had become popular commercially a few years before the priority date. Amongst the commercial devices was the MERL DiamondTouch system and the iPhone 1. The patent is concerned with how the software handles this type of input. The software between the display and the user is called a graphical user interface or GUI.

The witnesses

23. On this patent, Apple called Dr Brad Karp, who is a Reader in Computer Systems and Networks and head of the Networks Research Group in the Department of Computer Science at University College London. HTC challenged both his expertise and his objectivity. As to his expertise, they pointed out that Dr Karp was essentially a network person, whose published work focussed on security of computer systems and networks. Dr Karp has never been involved in writing system software for a graphical user interface. His experience of GUIs and their toolkits was as a user, and was limited as well.
24. I accept that Dr Karp is a knowledgeable computer scientist. However I agree with HTC that his knowledge and expertise were not well suited to giving evidence as to the thinking of a scientist concerned with writing system software for a GUI. I think this is apparent from the following questions and answers in cross-examination:

Q. So really, you are not in a position to speak about knowledge and attributes of those involved in the design and implementation of GUIs in 2008?

A. I am sorry, I do not see how that follows. No, I disagree.

Q. On what basis?

A. On the basis that I have a broad knowledge of the computer industry as a systems person so, in the course of my career, I have interviewed for jobs at various companies and I have friends who were undergraduates with me who went on to work at various companies. So I have knowledge from personal acquaintance in the computer science community with who can wind up working on what at a firm that develops software, at a firm that sells a product of software. So there is an ethos in computer science, especially in the building community of people who develop software, that one often learns by doing; that after an undergraduate degree, you have knowledge in software engineering. When you join a company, you may be put on a project where you work on developing an artefact of software where you do not have research level training in research in the area of that software, but rather you are an implementer, you are a programmer, and you have broad software development expertise, as I believe I characterised in my report. Then you work on extending and enhancing some existing software artefact using your broad based knowledge of software engineering.

Q. But, of course, in relation to the design and implementation of GUIs, you have never learnt by doing?

A. I have not. I have acquaintances who were in that position at companies that they worked for.

25. As to objectivity, HTC made submissions about the time taken by Dr Karp before answering questions, about his belated reliance on a sentence in the specification on an issue of construction, about an inconsistent approach to Zotov and the patent, and so on. I did not think that any of this suggested partiality of Dr Karp. It is true that he was an extremely cautious witness, choosing his words with the utmost care. I think that Dr Karp was doing his best to answer questions objectively.
26. HTC called Dr Daniel Wigdor, who is an Assistant Professor of Computer Science at the University of Toronto. Between 2005 and 2008 he worked at MERL on the Diamond Space project, which included a multi-touch product. Between 2008 and 2010 he worked at Microsoft, developing products, including Surface (a multi-touch product), where he played a leadership role.
27. Apple's main criticism of Dr Wigdor was that he accepted that he was a creative individual and a member of the research community. They suggested I should approach Dr Wigdor's evidence of common general knowledge with caution as a result. I accept that I should approach Dr Wigdor's, and indeed all the evidence of common general knowledge with caution. Nevertheless I consider that Dr Wigdor was making a genuine effort to consider only what would be known to an uninventive member of the skilled team, drawing on his knowledge of such people when working alongside them in industry. He was a frank and very helpful expert witness.

The skilled addressee

28. 948 is addressed to a team working in industry in the development of system software of a GUI for a multi-touch device. The team would include someone with expertise in software engineering and someone with experience of implementing GUIs. The team would be concerned with developing products rather than academic research. For example academic work has continued both before and after the priority date into multi-mouse (or multi-mice) devices, but the multi-mouse has never gained acceptance in commercial products.
29. The skilled team would also have some knowledge of HCI, but would not be an HCI specialist. Dr Wigdor at one point attributed rather more HCI expertise to the skilled team, but I do not think anything turned on this.
30. The main dispute in this area concerned the level of skill of the software developer: was he or she a "soldier" or a "decision maker designer of APIs"? The soldier, in this analogy, is someone who just does what he is told. I have no doubt that the skilled person is not a mere soldier. It is clear that in real life the skilled team would have a leader with authority to take decisions. Whilst lacking inventive capacity, the leader would be able to adopt common sense and common general knowledge solutions to questions which presented themselves in the course of development.

Common General Knowledge

31. In order not to make this long judgment even less readable, I give an account of software creation and event-driven programming in Appendix 1 to this judgment. It is derived largely from the expert reports of Dr Karp and Dr Wigdor. It represents common general knowledge. I add here some more general observations.
32. A general goal of operating system designers is to ease the task of application software developers. The success of an operating system is likely to be driven by the scale of its adoption by application developers as well as end users. This can be done by providing features within the system on which application developers can build, reducing the amount of software which they have to write. The decisions taken by system developers as to what facilities to include in the system software have an impact on the cost of development of the application software. Thus the provision of a "button", a UI element, in the system software can allow the application developer to incorporate it by reference in the application, without the need to provide program code as to how it should look or how it should respond to input from the user.
33. It was common to allow for the properties of a UI element to be defined by a software developer in the UI toolkit. Properties may be various. Where a property is capable of having only two possible values, it can be defined by setting the value of a "flag" attached to the UI element. The flag is stored as a single binary bit, and is either set (1) or not set (0). The property of a button whereby it is either enabled or not enabled could be indicated by a flag.
34. Dr Wigdor explained that it was well known to use the setting of a flag on a UI element to indicate whether or not particular events should be sent to that UI element. He also explained that the practice of limiting events sent to a particular UI element as a method of simplifying the development of software was also part of the common general knowledge. In each case he gave examples. Although Dr Karp quibbled with some of the examples in his written evidence, he accepted that it was common general knowledge to use a flag so that events of a particular type were not sent to the UI element. He also accepted that it was generally known that this could be beneficial for the software developer.

The specification and claims

35. 948 is concerned with technical issues which arise with multi-touch devices. The background section of the specification explains that such devices are recognised to bring benefits, but present challenges for the design of the interface. The conventional mouse and pointer interface is only capable of interacting with a single window and application or process at a time. The assumption of a single interaction at any one time simplifies user interface design. However, in a multi-touch interface, more than one touch event can occur simultaneously at any time. This can make it difficult to split the display into different portions. Moreover, a single software element may need to process multiple touch events. However, if all software elements need to process multiple events, the software may become more costly and complex. In addition, it may become difficult to convert or "port" software designed to run with a single pointing device to a version which can run on a multi-touch device.
36. The summary of the invention explains at [0008] that, in order to simplify the recognition of single and multi-touch events, each view within a particular window can be configured as either a multi-touch view or a single touch view. In addition, each view can be configured as an exclusive or a non-exclusive view. The specification continues at [0008] as follows:

"Depending on the configuration of a view, touch events in that and other views can be either ignored or recognized. Ignored touches need not be sent to the application. Selectively ignoring touches can allow for simpler applications or software elements that do not take advantage of advanced multi touch features to be executed at the same device (and even at the same time) as more complex applications or software elements."

37. 948 proposes the use of flags associated with views on the screen. The flags are:

i) The multi-touch flag which indicates whether a particular view is allowed to receive multiple simultaneous touches;

ii) The exclusive touch flag, which indicates whether a particular view allows other views to receive touch events while the flagged view is receiving a touch.

38. The operation of the multi-touch flag is shown by the logic diagram of figure 4:

39. Thus, if a user touches a view at a second location without having released the touch at a first location within the same view, the operating system checks whether the multi-touch flag for that view is set. If it is, then the second touch event will be sent to the software element associated with that view. If it is not set, the touch event is ignored or blocked by the operating system. The benefit is explained at [0045]:

"[0045] Thus, embodiments of the present invention can allow relatively simple software elements that are programmed to handle only a single touch at a time to keep their multi-touch flag unasserted, and thus ensure that touch events that touch events that are part of multiple contemporaneous touches will not be sent to them. Meanwhile, more complex software elements that can handle multiple contemporaneous touches can assert their multi-touch flag and receive touch events for all touches that occur at their associated views. Consequently, development costs for the simple software elements can be reduced while providing advanced multi-touch functionality for more complex elements."

40. The logic of the operation of the exclusive touch flag is that a user first touches a first view, causing the operating system to send a first touch event associated with that first view. The logic diagram of figure 5 then picks up the sequence when the user touches a second view without releasing the first:

41. The above arrangement has the consequence that it is only if the exclusive touch flags for both touched views are unset that the operating system sends the touch event for the second touch to the software element associated with that view.
42. The benefit is explained at [0049]:

"[0049] Thus, the exclusive touch flag can ensure that views flagged as exclusive only receive touch events when they are the only views on the display receiving touch events. The exclusive flag can be very useful in simplifying the software of applications running on a multi-touch enabled device. In certain situations, allowing multiple views to receive touches simultaneously can result in complex conflicts and errors. For example, if a button to delete a song and a button to play a song are simultaneously pressed, this may cause an error. Avoiding such conflicts may require complex and costly software. However, embodiments of the present invention can reduce the need for such software by providing an exclusive touch flag which can ensure that a view that has that flag set will receive touch events only when it is the only view that is receiving a touch event. Alternatively, one or more views can have their exclusive touch flags unasserted, thus allowing multiple simultaneous touches at two or more of these views."

43. Thus the multi-touch flag is concerned with the situation where the second touch is to the same view, whilst the exclusive touch flag is concerned with the situation where there is a second touch to a different, second view. These flags behave independently.
44. 948 contains claims of three types: claims to a method for handling touch events (1-10); claims to a multi-touch enabled device (11-20) and claims to a computer readable medium (21-23). It is sufficient to consider the first group of claims. Apple maintain that if claim 1 is invalid, claim 2 is independently valid.
45. Claim 1 is in the following form, with added reference numerals:

"(i) A method for handling touch events at a multi-touch device, comprising:"

(ii) displaying one or more views;

(iii) executing one or more software elements, each software element being associated with a particular view;

(iv) associating a multi-touch flag or an exclusive touch flag with each view, said multi-touch flag indicating whether a particular view is allowed to receive multiple simultaneous touches and said exclusive touch flag indicating whether a particular view allows other views to receive touch events while the particular view is receiving a touch event;

(v) receiving one or more touches at the one or more views; and

(vi) selectively sending one or more touch events, each touch event describing a received touch, to one or more of the software elements associated with the one or more views at which a touch was received based on the values of the multi-touch and exclusive touch flags.

It is common ground that although feature (iv) ends with the word "touch event" it should, for consistency, read simply "touch". Claim 2 adds the feature:

"if a multi-touch flag is associated with a particular view, allowing other touch events contemporaneous with a touch event received at the particular view to be sent to the software element associated with the other views."

Construction

46. There are two principal issues on construction of claim 1.

Integer (iv) and "per view granularity"

47. This issue arises in the context of infringement. HTC contend that, when this integer is read as a whole, one sees that each displayed view is associated with a software element and each view has a multi-touch and/or an exclusive touch flag associated with it. The flag indicates whether that "particular view" is multi-touch or exclusive-touch. HTC point in particular to paragraph [0008] which says that "each view within a particular window can be configured as either a multi-touch view or a single touch view" and "each view can be configured as either an exclusive or a non-exclusive view".
48. Apple contend that the claim does not preclude more than one software element being associated with the same view. Moreover they contend that, as UI elements are hierarchical, a flag associated with UI elements at an upper level in the hierarchy may be associated with elements at a lower level. They point in particular to [0024] in the specification which says that views can be "nested". Thus, they say, a flag may be set for a group of views, and not merely on a "per view" basis. Mr Burkill gave the analogy of a number of people being associated with an address or postcode. Each of the people is associated with the address and the address tells you something about each particular person. The flag here may indicate a property of a group of views.
49. In my judgment, HTC are correct on this issue of construction. The words "each" and "particular" are words of emphasis which add something to the claim. The skilled reader would understand by reference to the teaching of the specification that the words were there to indicate that each view has a flag and the flag indicates the properties of that particular view. The specification contains no suggestion of anything other than "per-view granularity", as it was termed in the evidence. The reference to nested views does not go as far as suggesting that the views should not receive individual flags.
50. As Dr Karp explained, the skilled person would appreciate that the ability to set the flags independently for each view was important from the technical perspective. It enabled a "rich space of behaviours with respect to multi-touch input". This advantage is not achieved without per view granularity. Although Dr Karp volunteered the view under cross examination that the specification did not necessarily preclude a single flag being associated with multiple views, he did not draw attention to any teaching to that effect. To the extent that he was volunteering a construction of words in the claim, I am entitled to disregard it: the words "each" and "particular" are not terms of art. The specification simply does not deal with the notion of, or the technical consequences of, setting the flags collectively on a container basis. The skilled person would appreciate from the language used that it was not intended to be covered.
51. I think Mr Burkill's analogy of an address is unhelpful, as it focuses only on the word "associated". The technical import of a flag being associated with a particular view is that its value tells you something about that particular view which may be different from other views. An address is by definition the same for all at the address.

Integer (vi): "selectively sending one or more touch events"

52. This issue arises in the context of infringement as well. HTC submit that this feature achieves the advantages of the invention, which are described amongst other places in the specification at [0045], namely that unwanted touch events are not sent to the software element associated with a view. Thus, for example, subsequent touch events at a view for which the multi-touch flag is not set will be ignored or blocked rather than sent to the view. This, they submit, is what the skilled person would derive from [0008].
53. Apple contends that the integer is satisfied if the software decides, on the basis of the value of the flags, where to send the touch events. They point to [0039] which says that the invention involves "selectively providing touch data to various software elements in accordance with predefined settings" and [0044] which says:

"If, on the other hand, the multi-touch flag is not set, the OS can ignore or block the second touch. Ignoring the second touch can result in not sending any touch events associated with the second touch to the software element associated with the touched view. In some embodiments, the OS can alert other software elements of the second touch, if necessary."

54. Dr Wigdor thought that this passage contemplated sending the additional information that the touch event had been ignored to software elements other than views. Dr Karp thought that the passage was "consistent with delivering the touch event".
55. I think HTC are correct about this issue as well. The skilled person would understand that the purpose of the requirement for selective sending was to relieve the application software developer of having to deal with the events not selected to be sent: in other words the selection is between sending the events to the software elements and not sending them there. There is no basis in the specification for an arrangement in which the selectivity is as between different software elements. The skilled person would recognise the patent as teaching the application of the common general knowledge technique of sending or not sending events based on the value of the flag. He or she would not recognise any suggestion of a use of the flag for a selection between different routing of touch events. In my judgment the stray phrase in [0044] would not be regarded as sufficiently clear to the skilled person to help resolve the dispute about the meaning of "selectively send".

Infringement

56. Five HTC devices are in issue. Each runs an operating system called Android 2.3. Apple does not allege that there is any flag in Android 2.3 which corresponds to 948's multi-touch flag. The case of infringement depends on Apple's assertion that Android 2.3, in providing a flag called FLAG_SPLIT_TOUCH, provides a flag which works in the same way as 948's exclusive touch flag.
57. In order to understand Apple's case of infringement, one has to consider what Android calls Window and View objects. Windows are used as containers of View objects. So a screen may contain a Window, with two Views contained within it. Each Window contains an object which holds all its basic properties, including any flags. Views may be assembled into hierarchies, in which a group of views is descended from a ViewGroup. Each Window has an object known as ViewRoot to which the View hierarchy is attached.
58. In Android 2.2, which is not alleged to infringe, event handling occurred as follows. When the screen is touched, an input event is sent to a system process called InputDispatcher. This determines the topmost touchable Window within whose bounds the touch occurred. InputDispatcher then turns the input event into a MotionEvent and sends it to the ViewRoot of the appropriate Window. The MotionEvent is then passed down the View hierarchy until it reaches the View from which the touch originated. If a second concurrent touch is received, anywhere on the screen, InputDispatcher will package up information about both the first and second touches into a MotionEvent and send it to the ViewRoot of the Window where the first touch took place. Thus information about concurrent touches is always sent to the Window which was first touched. Further, that MotionEvent is then passed down the View hierarchy until it reaches the View from which the first touch originated. Thus all information about concurrent touches is always sent to the View which was first touched.
59. In Android 2.3, which is alleged to infringe, MotionEvents within a Window are still all sent to the View first touched. So information about concurrent touches is still always sent to the View first touched.
60. FLAG_SPLIT_TOUCH is a Window level flag which affects event handling between Windows, but not within Windows. The event handling within Windows remains the same as for Android 2.2.
61. If a first touch is made to View 1 in Window A and a concurrent touch made to View 2 in Window B, and FLAG_SPLIT_TOUCH is set for both Windows, InputDespatcher sends a MotionEvent relating to the first touch to the ViewRoot of Window A and a MotionEvent relating to the second touch to the ViewRoot of Window B.
62. HTC submitted that the method does not infringe claim 1 because there is no flag associated with each view to indicate exclusivity for that particular View. The behaviour of the operating system, when two or more views are touched within the Window is always the same. Accordingly the advantage of the invention is not realised: the application software developer needs to write software to deal with the concurrent touch at a different view within the Window.
63. Apple submitted that it is sufficient that the flag is set at Window level. I have rejected that interpretation of the claim. They rely on the fact that there are some cases, one of which is illustrated in the Product and Process description, where there is only one View within a Window. I think there are two reasons why this does not help them. Firstly, as Dr Karp accepted, there will be other views displayed, and the claim requires each View to have a flag. Secondly, even with that case, it is the Window and not the View with which the flag is associated.
64. I accept HTC's submissions. Given the construction of the claim which I have adopted, Android 2.3 does not infringe claim 1.
65. HTC contend that the method of Android 2.3 does not infringe claim 1 for the further reason that there is no selective sending of events. All touch events are sent to a View. They submit that this feature of Android 2.3 means, again, that it does not achieve the advantages of the patented invention, because software developers are not saved the cost and complexity of having to write code for dealing with concurrent touches.
66. Apple submit that Android 2.3 selectively sends touch events, at Window level.
67. This is essentially the issue of construction which I have decided against Apple. It follows that there is no infringement on this basis as well.
68. I was referred to a judgment of August 24 2011 in Apple Inc v Samsung Electronics Co Limited and others, Judge Brinkmann sitting in the District Court of the Hague reached, as I understand it provisionally, the same conclusion in relation to the first of these non-infringement points: see paragraphs 4.37 and 4.38 of the judgment. Although I have reached my conclusions independently, it is pleasing to have arrived at the same overall result as the District Court of the Hague.

Validity

69. HTC contend that 948 is invalid over common general knowledge alone, over the Jazz Mutant Lemur and over Zotov.

Obviousness over common general knowledge

70. HTC's favoured attack was over common general knowledge alone. Such an attack must, as I have said, be treated with caution. Nevertheless, it is entirely possible for a patent to be held invalid over the common general knowledge. In such cases it is necessary to make sure that one keeps one's eye not only on the items of common general knowledge relied on to undermine the patent, but also the other general knowledge which would have affected the thinking of the skilled person. I have endeavoured to do this.
71. Dr Wigdor approached the issue of obviousness from the point of view of a skilled team who wished either to enable developers easily to port legacy software for use with a multi-touch device or to ease the task of creating software to take advantage of the functionality which multi-touch devices offer. In either case his evidence was that the skilled person could without invention have arrived at a method of controlling event distribution which used the flags of the 948 patent.
72. Thus in the case of the legacy developer and legacy software, the software's response to the new multi-touch input must be considered. An immediate problem would be how to handle concurrent touches which the legacy application would not be designed to receive. One method would be not to allow concurrent touches at all. The legacy application could then consider itself as running on a single touch device. This is what in fact occurred in an application called DT Mouse. It is however an "all or nothing" approach.
73. Dr Wigdor then argues that it is normally the case that decisions about whether to route an event are made based on properties of the UI elements. Thus the skilled person would consider enabling the application developer to provide more fine grained control, and allow the application developer to decide for an individual UI element how to deal with multiple touches. Multiple touches could be sent to an individual UI element, or to separate UI elements.
74. The argument based on easing the development of multi-touch software runs as follows. Firstly Dr Wigdor says that it would be immediately apparent that many components of applications would not require multi-touch functionality. For example someone developing a calculator would not want the individual buttons to support multi-touch or the ability to touch separate buttons at the same time. Dr Wigdor says that it would be obvious to the skilled person that requiring authors of such software to write code to handle multiple extraneous contemporaneous touches on each UI element imposes an undue burden. He maintains that the obvious solution would be to allow application developers to opt out of simultaneous inputs within and across UI elements.
75. Dr Wigdor maintains that an obvious solution at the end of either line of reasoning is to use a flag associated with each UI element so as to prevent some types of input being sent to that element. He concludes that an obvious way to achieve the aims he has identified would be to define a flag to indicate the ability of the UI element to handle multiple touches or not (the multi-touch flag) and a flag to indicate whether concurrent touches to other UI elements are allowed (the exclusive touch flag).
76. Mr Burkill attacked this reasoning as being classic, impermissible, step-by step, ex post facto reasoning. I set out the main points below:

i) There was little or no motivation for providing legacy support in the new multi-touch environment: some developers might have preferred the approach of "all new everything" meaning that all applications would have to be specifically written for multi-touch.

ii) What excited the research community rather more was the new functionality of multi-touch, not the "more pedestrian" question of how to modify legacy software.

iii) The DTMouse application prevented multi-touch events from reaching the application software: this was the all or nothing approach which Dr Wigdor said that the skilled person would think undesirable.

iv) The alternative to DTMouse would be to send all events to the application and leave it to the application developer to write software to deal with these events.

v) The flag solution had only ever been applied to different types of events, not to events differentiated by reference to their time stamps.

vi) The skilled person would not have the idea of controlling at the UI element level.

vii) Insight would be required to see that multiple touches can be directed to the same UI element or across different UI elements.

viii) That Dr Wigdor had used hindsight.

77. I think there is force in Apple's suggestion that the primary focus of the skilled team would not be directed to legacy applications. Nevertheless, in my judgment the skilled team would have to give careful consideration to how to design the interface to ease the writing of software for the new multi-touch capability. I hold that the skilled team would see immediately that, whilst multi-touch delivered desirable additional functionality, there would be situations where individual UI elements would not want multi-touch, and situations where second concurrent touches between UI elements should not be allowed. Dr Wigdor's evidence that this problem of concurrency would be immediately apparent to a person skilled in the art was entirely convincing. Dr Karp was ready to accept that there would be instances where it would be obvious that conflicts would occur, but thought that there could be invention in perceiving that there was a general problem of conflicts with multi-touch applications. On this issue I am in no doubt that I should prefer Dr Wigdor's evidence, including his evidence that the skilled person would see the need for fine grained control.
78. The critical question on obviousness is, as it seems to me, whether the skilled person would see that the way of dealing with the need identified in the previous paragraph would be at system level, or whether the skilled person would consider, as Dr Karp suggested, that the way to do it would be to send the events to the application software and "consider that his work was done".
79. This was the main basis on which Dr Karp was cross-examined. Dr Karp recognised that the skilled person would appreciate that applications running on a multi-touch device will contain four types of UI elements:

i) UI elements which will need the ability to receive multiple concurrent touches;

ii) UI elements which require only a single touch, and multiple concurrent touches to that element will not be acted upon: e.g. keyboard buttons;

iii) UI elements which need to be able to receive input which is concurrent with other input at other UI elements: eg holding down a shift key whilst pressing a letter;

iv) UI elements whose functionality should not be invoked concurrently with that of other UI elements: for example operations which were in conflict with each other, such as "yes" and "no".

80. Dr Karp also accepted that the iPhone 1 was a very well known example of a multi-touch device at the priority date. He accepted, inevitably, that a skilled person examining such a device would readily be able to see that it exhibited all four types of behaviour. He also accepted that the system software developer would want to make sure that the applications on the multi-touch device would be able to exhibit similar behaviours to the iPhone 1.
81. Consistently with the case put to, but rejected by Dr Wigdor, Dr Karp said that the skilled team, faced with designing an operating system to support applications of this sort, would take the path of least resistance and send all input events to the application software. This would leave the application software developer the job of writing the code to deal with those events. The system software team would then say "I have done my job".
82. Dr Karp accepted that with a complex application, leaving the writing of the software to the application developer would be burdensome, and that the broad general ethos was to avoid burdening the software developer. The following passage in his cross-examination (about exclusive touch rather than multi-touch) illustrates Dr Karp's position:

Q. If you leave to it the application to have the code in [semble "to"] process events relating to all concurrent touches and work out which events should be responded to in what way, that is a pretty complex exercise, is it not, for the application software developer?

A. It requires the writing of additional code to implement that functionality.

Q. It definitely increases the complexity of the software that has to be written by the application software developer?

A. I think that the degree of complexity would depend on the application and the number of elements that we are talking about.

Q. It could be very complex indeed, could it not?

A. I could envision cases where, for a very complex application, it could be very complex, yes.

Q. That is a burden which the system software developer would wish to avoid placing on the application software developer, for the reasons we discussed yesterday?

A. There is a broad and general ethos in the design of systems software to make the writing of applications software simpler in ways -- yes, to make the writing of application simpler.

Q. Right. So I would suggest that the system software developer would be looking for ways to avoid the application software developer having to write code to deal with those sorts of issues. Do you agree?

A. In general, developers of libraries seek to make their libraries easy to use by programmers and to make the writing of applications as easy as possible and no easier.

Q. So I would suggest to you that, in those circumstances, it would be obvious to the system software developer that a way of providing for UI elements which, when touched, do not allow other UI elements to respond to subsequent concurrent touches is to have the system software not pass on the unwanted touch events to the relevant UI elements.

A. So I do not find that behaviour to be obvious.

Q. When you say "that behaviour"?

A. Sorry, I do not find that implementation of the system software to be obvious.

83. Whilst Dr Karp maintained that position, his underlying reasoning was not clear to me. His positive reasoning related not so much to whether the skilled person would see the need to introduce a system filter, but whether the skilled person would arrive at the use of a flag as the means of introducing it. As I have indicated above, Dr Karp's view was that a flag was a known means of filtering types of event, but had not been used to filter types of event based on their time stamps.
84. I was not persuaded that there was any significant difference between the types of event based on time stamp, and types of event based on any other distinguishing feature. In any case the distinction was something of a red herring, as it is not in fact necessary to use the time stamp to distinguish these types of event. This is an issue, again, on which I prefer Dr Wigdor's evidence to that of Dr Karp.
85. Mr Burkill relied heavily on the forensic question: if it was obvious why was it not done before. Why was DTMouse, with which Dr Wigdor was involved, not evidence that the invention was not obvious? Dr Wigdor explained, to my mind satisfactorily, why the situation facing the designers of DTMouse was not the same as that facing the skilled team in the obviousness inquiry here. DTMouse allowed the user of MERL Diamond Touch multi-touch device to use Microsoft Office software and the like by mouse emulation. The designers did not have access to the source code for Microsoft Office, but nevertheless wanted users to be able to use that software.
86. I also think that the forensic question has dangers in this case, as it is based on an implicit assumption that if it had been done before it would be possible to find out about it. Most developers treat their code as proprietary.
87. In my judgment the inventive concept is obvious in the light of common general knowledge. The skilled team tasked with designing an operating system for a multi-touch device would arrive at the invention by the routine application of common general knowledge design principles.
88. Claim 1 is obvious in the light of the common general knowledge.

Claim 2

89. I did not understand any obviousness attack to be directed at claim 2. It relates to a further aspect of the multi-touch flag, and is therefore not relevant to the allegation of infringement. Claim 2 therefore survives.

Obviousness over Jazz Mutant Lemur and Zotov

90. In the light of my finding in relation to obviousness over the common general knowledge, I do not think that these further citations need to be considered. Mr Tappin for HTC indicated that common general knowledge was his favoured attack. Mr Burkill for Apple maintained that these citations got HTC no further. The citations raise similar considerations as to how the skilled person would proceed, which I would be inclined to resolve in the same way. Zotov was in any event directed only to a multi-touch flag. I will not add to the length of this judgment by analysing these further citations in detail.

Excluded subject matter

91. I have construed claim 1 above. Mr Burkill did not explicitly define the contribution. He submitted that the invention met all the signposts referred to by Lewison J in AT&T. Neither side made any distinction between claims 1 and 2 for this purpose.
92. He submitted, firstly, that the invention provided a technical effect on a process carried on outside the computer, by simplifying the creation of application software. Secondly he submitted that the claimed technical effect operated at the level of the architecture of the computer, because it was implemented in the operating system. Thirdly he submitted that the claimed technical effect resulted in the computer operating in a new way, because it presented a new set of APIs to the developer, enabling the device to send touch events selectively. Fourthly he said there was an increase in the speed and reliability of the computer because the invention simplifies application coding and lastly that the invention overcame the problem it purported to address.
93. I do not accept these submissions. Lewison J's signposts are directed to determining whether a contribution is technical in nature. The anterior questions are (a) what is the contribution, and (b) whether it lies wholly within excluded subject matter?
94. It is clear that one part of the contribution of the 948 patent lies in the software which processes the multi-touch input. This is plainly excluded subject matter. The contribution also includes the advantage that it makes it easier to write software for the device. I consider that this contribution also lies wholly within excluded subject matter. The writing of programs for computers seems to me fall squarely within the exclusion of computer programs as such.
95. I turn therefore to Mr Burkill's approach, which is to consider whether there is a relevant technical effect. As to the first of his points, I do not think that ease of writing application software can be a relevant technical effect outside the computer. I accept Mr Tappin's submission that, in the context of the computer program exclusion, ease of writing computer programs cannot be a relevant technical contribution or effect. The writing of computer programs is excluded subject matter. Making it easier for one part of the software to be written is merely a re-distribution of the labour of writing the software. For completeness I would add that the structure of the software does not have any real world effect on the way the device performs. It was common ground that one could not tell whether or not any given device was using the invention.
96. The second point, that the method applies at the operating system level is correct. But not every method which operates at this level will be patentable. This signpost derives from the first of the IBM cases T 0006/83 referred to by Lewison J at [21], a case concerned with a network of computers. At [6] the Board referred to "features … not concerned with the nature of the data and the way in which the particular application program operates on them" as being patentable. In my judgment the invention of 948 is precisely concerned with the way in which the software operates on the data, namely the touch events.
97. The third point, that the claimed technical effect results in the computer working in a new way, by presenting a new API to the developer in which touch events are sent selectively, is not correct. The computer is not working in a new way in any relevant sense. There is merely a redistribution of the data processing within the device.
98. Next, there is no evidence of an increase of speed or reliability of the computer. Finally, I do not regard the last point as persuasive in a case where the problem solved is entirely within the computer.
99. I conclude that the invention, at least as claimed in claims 1 and 2, is not patentable because it is a computer program as such.

The 022 Patent

100. The 022 patent is entitled "Unlocking a device by performing gestures on an unlock image". It has a priority date of 23^rd December 2005. In broad terms it is concerned with the provision of a user interface on a touch screen device which enables the user to change the state of the device by, for example, dragging an image over the screen. The invention has been commercialised by Apple as the "slide to unlock" feature of its iPhone, which users of such devices encounter on first use.

Technical background

101. Human computer interaction, or HCI, is a science that encompasses the requirements, design, implementation and evaluation of computer systems for human use. The science of HCI involves some understanding of human psychology.
102. Computers had been able to recognise human input via touch for many years before the priority date. Touch pads are touch-sensing devices which are separate from the computer's visual display. Touch screens allow the user to input directly by touching the screen, at points defined by the graphic information on the screen. This is achievable by a variety of different technologies, the details of which are irrelevant to the decision on this patent. The input in either case can be by a stylus or pen, or by a finger or hand.
103. Some direct touch inputs involve more than just placing the finger or stylus on the relevant graphic on the screen, but allow the input to be interpreted by the movement of the finger or stylus. Thus graphic objects can be dragged over the screen by touching them, and maintaining contact whilst moving them to other locations.

The witnesses on 022 and 868

104. On this patent and on 868, Apple called Professor David Keyson. Professor Keyson is a Professor of Industrial Design Engineering at Delft University of Technology, in the Netherlands. He has been a professor there for 12 years. Professor Keyson is an expert in the field of HCI. His first degree was in political and social sciences, but he then obtained a Master's degree in Ergonomics from Loughborough University in 1987, and a PhD in Perception and Technology from Eindhoven University of Technology in 1996. His dissertation was on Touch in User Interface Navigation. Between finishing his Master's degree and starting his PhD, Professor Keyson worked as a human interface engineer at Xerox in the department of Industrial Design and Human Interface. He also worked, during his PhD studies, at the Institute for Perception Research.
105. Despite Professor Keyson's obvious experience in the field, I did not find him to be a very helpful expert witness. I have come to this conclusion for the following reasons. Firstly, in my judgment, Professor Keyson had his eye so firmly on the issues in the case that he found it difficult to respond clearly and fairly even to simple technical questions. Whatever the reasons for this, which I do not need to enquire into further, the end result was unhelpful. Secondly, it was clear to me that he was not as familiar with the contents of the cited prior art documents (particularly in the 868 case) as I would expect an expert to be. This was particularly clear in the case of the Lira citation against 868. If an expert has misunderstood the teaching of a prior document, which in my judgment Professor Keyson had, then his views as to whether something is obvious in the light of it are likely to be flawed. Thirdly, in advancing Apple's case on some issues he made factual errors. Thus, for example, he said that HTC devices constantly checked x and y co-ordinates to see whether a threshold was reached. This was incorrect. He also appeared to have an incorrect understanding of the way in which the Arc unlock feature of some of the HTC devices worked, and was reluctant to confirm the way in which they functioned. I think he made these mistakes because he was over-enthusiastic about advancing Apple's case, and less concerned with getting the basic facts correct. In the course of all this he lost objectivity. I have treated Professor Keyson's evidence with caution as a result.
106. On this patent and 868, HTC called Professor Saul Greenberg, a professor in the Department of Computer Science and an adjunct professor in the Department of Psychology at the University of Calgary. His first degree was in microbiology and immunology, but he subsequently obtained the degrees MSc and PhD in Computer Science at Calgary. He has studied and researched full time in the field of Computer Science since 1981.
107. In his expert report, Professor Greenberg said that he was aware of Apple products, as one would expect him to be, and that he had owned an iPhone since late 2010/early 2011 and an iPad since mid-2011. In paragraph 90 he said that an obvious addition to one of the items of prior art, the Neonode phone cited against 022, would be to incorporate a slider to the user interface. He added that he had made that suggestion without seeing the 022 patent. Mr Thorley said this was a misleading thing to have said, given that he had seen an implementation of 022 in the iPhone and iPad. I do not think that is a fair criticism when Professor Greenberg's report is read as a whole. It was relevant to record that Professor Greenberg had made the suggestion without knowing what the case was about. How much weight to attach to that fact when one adds the fact, which he had disclosed at the forefront of his report, that he had owned an iPad and an iPhone is another matter. Overall I found Professor Greenberg to be a balanced, careful and knowledgeable witness.

The skilled addressee and common general knowledge

108. There was no significant dispute about the identity of the skilled person in the case of the 022 patent. The 022 patent is addressed to a worker in the field of HCI who has a graduate degree in a subject within or concerned with the field of user interface design and about three years of industry experience. The skilled person would be working as part of a wider team of people having the necessary experience in hardware and electronics with whom he or she would interact as necessary.
109. The skilled person would be familiar with touch screen devices that were more sophisticated than simple point and click devices. There were a number of pen-based personal digital assistants (PDAs) on the market before the priority date. These included the Palm Pilot series, the Apple Newton and the IBM Simon. The skilled person would be familiar with these.
110. The skilled person would be wholly familiar with mouse-based techniques for input used in operating systems such as Windows. These included the sliding image used in the scroll bar in those systems. Many mouse based user-interaction techniques had been carried across into the PDA devices on the market at the priority date. Thus the devices which used the Windows CE platform, widely used on PDAs at the priority date, used similar sliders and scroll bars on their user interfaces to those found in mouse-based systems. They were made available to developers through Windows Visual Studio which could be used to develop Windows CE applications for PDAs.
111. The problem of accidental touches and accidental activation in touch screen devices was well known at the priority date. The problem of accidental activation while the device is not intended to be in use could be dealt with by the device locking up after detecting inactivity for a given period, or by the user locking the device manually. The use of physical controls such as buttons and slider toggles to transition a device from a lock state to an unlock state was common general knowledge by the priority date. Commercial examples of portable touch screen devices which used a mechanical slider to operate a keylock function were the Compaq Tablet PC TC1000 and the Dell™ Axim™ X50. Another example of a keylock function activated by a physical slider (albeit on a touch device rather than a touch screen devices) was the Apple iPod.
112. It was also common general knowledge, as Dr Keyson accepted, that touch screen devices could be unlocked using the screen itself by way of a code. This was a feature of the HP Jornada.
113. A commercial example of a touch screen device which used a combination of buttons to activate a keylock function was the Palm Treo 600 phone. Using this feature deactivated the physical buttons and screen items.
114. "Feedback" in the context of HCI means the provision of information to the user that his or her input is being recognised by the system. Thus, in text systems, the user gets immediate feedback that his letter has been typed. Likewise, with mouse systems, the user gets feedback via the cursor of the movement of the mouse. In touch systems the object under the touch reacts to the touch and its movement. The skilled person would have known at the priority date that a good general principle of interface design was to provide continuous feedback to the user for every user action. The textbooks showed, and the experts agreed, that feedback was a known and fundamental concept of good design. Nevertheless the amount of, and design of the appropriate feedback to be used would be decided on a case by case basis.
115. HTC submitted that the Plaisant citation was part of the common general knowledge. I deal with this suggestion in context below.

The specification and claims of 022

116. The specification of the 022 patent explains at [0003], by way of background, that touch screens are becoming increasingly popular for use as displays and as user input devices on portable devices such as mobile telephones and personal digital assistants (PDAs). According to the specification, a problem associated with such touch screens is the unintentional activation or deactivation of functions due to unintentional contact with the touch screen. The specification explains that the portable devices themselves, the touch screens and applications running on the devices may be locked in various ways, for example upon entering an active phone call, after a predetermined idle time, or upon manual locking by a user.
117. The specification acknowledges several well known methods for unlocking touch screen devices and applications running on them. These include pressing a predefined set of buttons simultaneously or sequentially, or entering a password. A prior publication is referred to which discloses unlocking a touch screen upon detecting touches on predetermined areas in a given order. These methods are said to have disadvantages in that the button pressing may be hard to perform, and creating, memorising and recalling passwords may be burdensome. The specification sets itself the object of a more efficient, user friendly procedure for:

"transitioning such devices, touch screens and/or applications between user interface states (e.g. from a user interface state for a first application to a user interface state for a second application, between user interface states in the same application, or between locked and unlocked states."

118. The specification also points out that there is a need for "sensory feedback to the user regarding progress towards satisfaction of a user input condition that is required for the transition to occur": in plainer language, feedback to tell you how you are getting on.
119. Although other passages of the specification are relevant, the invention claimed is best understood by reference to the description commencing at [0048] under the rubric "Unlocking a Device via Gestures". In the embodiments described, the device is changed from a locked to an unlocked state by making and maintaining contact with the touch screen in a predefined gesture. Paragraph [0051] includes the following as to the meaning of "gesture":

"As used herein, a gesture is a motion of the object/appendage making contact with the touch screen. For example, the predefined gesture may include a contact of the touch screen on the left edge (to initialize the gesture), a horizontal movement of the point of contact to the opposite edge while maintaining continuous contact with the touch screen, and a breaking of the contact at the opposite edge (to complete the gesture)."

120. The specification explains at [0036] that the device described has a contact/motion module which detects contact with the touch screen and contains components for performing various operations related to the detection of contact, such as determining if there is movement of the contact, and tracking the movement across the touch screen and determining if the contact has been broken. The specification explains that determining movement of the point of contact may include determining speed, velocity and/or acceleration of the point of contact.
121. At [0050] the specification explains the concept of "visual cues". These provide hints or reminders of the unlock action to the user. The visual cues may be textual or graphical or any combination thereof. The visual cues may be triggered by particular user actions, such as, for example, the user touching a locked touch screen.
122. The specification also explains that the device may also display an "unlock image" – see [0058]. An unlock image is a graphical, interactive, user interface object with which the user interacts in order to unlock the device. For example the user may drag the unlock image in a predefined manner to complete unlocking. At [0062], it is explained that in some embodiments the unlock action includes performing a predefined gesture with respect to the unlock image. Thus an unlock action may involve the user making contact with the unlock image, performing a predefined gesture while maintaining contact with the screen and thereby dragging the unlock image to a location which satisfies certain unlock criteria – see [62]. The location which satisfies the unlock criteria may be defined narrowly or broadly – for example it may be a particular marked location, or a quadrant of the touch screen – see [0063].
123. Paragraphs [0064] and [0065] are of particular significance to the argument on construction, so I set them out below:

"[0064] In some embodiments, the interaction includes dragging the unlock image to a predefined location on the touch screen. For example, the unlock action may include dragging the unlock image from one corner of the touch screen to another corner of the touch screen. As another example, the unlock action may include dragging the unlock image from one edge of the touch screen to the opposite edge. The emphasis here is on the final destination of the unlock image (and of the finger). Thus, the user can drag the unlock image from its initial location along any desired path. As long as the unlock image reaches the predefined location and is released at that location, the device is unlocked. It should be appreciated that the predefined location may be, as described above, defined narrowly or broadly and may be one or more particular locations on the touch screen, one or more regions on the touch screen, or any combination thereof.

[0065] In some other embodiments, the unlock action includes dragging the unlock image along a predefined path. For example, the unlock action may include dragging the unlock image clockwise along the perimeter of the touch screen (the path being the perimeter of the touch screen), from one of the corners and back. As another example, the unlock action may include dragging the unlock image from one edge of the touch screen to the opposite edge in a linear path. The emphasis here is on the path along which the unlock image (and the finger) moves. Because of the emphasis on the path, the final location to which the unlock image is to be moved may be defined broadly. For example, the unlock action may be to drag the unlock image from its initial location, along the predefined path, to any spot within a predefined region on the touch screen. The predefined path may include one or more straight lines or lines with twists and turns."

124. These paragraphs are drawing a contrast between giving emphasis to the destination and giving emphasis to the path. In the former case the user may choose any desired path provided he or she reaches the destination. In the latter case the user must follow the predefined path to reach a destination which may be defined broadly.
125. The invention is further explained by reference to Figures 4(a) and 4(b) which I reproduce below:

126. In Figure 4A, a device 400 has a touch screen display 408. The touch screen display is showing an unlock image 402 and visual cues. The unlock image has a built-in arrow to indicate the direction of movement. The visual cues include a channel 404 indicating the path of the gesture or movement along which the unlock image must be dragged. It is said that the channel is similar to the groove along which a slider switch moves. The cues also include one or more additional arrows 406 indicating the direction of the gesture or movement. The arrows may be animated. The right hand end of the channel is the predefined location to which the unlock image must be moved in order to unlock the device. It is stressed that the visual clues are merely exemplary and that more or fewer may be used.
127. The claims relied on by Apple are, firstly, the independent claim 1, 6 and 18, which are respectively a method claim, a device claim and a claim to a computer program product. Apple also rely on method claim 5 (and corresponding device claim 17) and device claim 9. No one suggested that claims 6 and 18 added anything to claim 1, or that claim 17 added anything to claim 5. Claims 1, 5 and 9 read as follows, with some additional numerals for reference purposes:
128. Claim 1:

(i) A computer implemented method of controlling a portable electronic device

(ii) comprising a touch-sensitive display,

(iii) comprising detecting contact with the touch-sensitive display while the device is in a user interface lock state;

(iv) transitioning the device to a user-interface unlock state if the detected contact corresponds to a predefined gesture;

(v) and maintaining the device in a user-interface lock state if the detected contact does not correspond to the predefined gesture,

(vi) characterised by moving an unlock image along a predefined displayed path on the touch sensitive display in accordance with the contact,

(vii) wherein the unlock image is a graphical, interactive user-interface object with which a user interacts in order to unlock the device.

129. Claim 5:

(i) The computer-implemented method of claim 1, further comprising:

(ii) displaying a first unlock image and a second unlock image on the touch-sensitive display while the device is in a user-interface lock state; and

(iii) wherein transitioning the device to a user interface unlock state comprises: transitioning the device to a first active state corresponding to the first unlock image if the detected contact corresponds to a predefined gesture with respect to the first unlock image; and

(iv) transitioning the device to a second active state distinct from the first active state if the detected contact corresponds to a predefined gesture with respect to the second unlock image.

130. Claim 9:

(i) The portable electronic device of claim 6 wherein the predefined displayed path is a channel.

Construction

"gesture"

131. Reading the expert evidence in this case, one might reasonably have come to the conclusion that there was to be a significant dispute about the meaning of the word "gesture". Thus Apple's expert, Professor Keyson, maintained that in the field of human-computer interaction, the term gesture had a specific meaning. In particular, if a device was only processing and responding to actions based on the current x,y coordinates of a graphical object, without taking into account the movement pattern, this would not constitute gestural human-computer communication. However, in opening the case Mr Thorley QC - perhaps sensitive to the impact this construction would have on Apple's case of infringement - made it clear that he was not contending that that definition applied in the context of the present case. He recognised, in my judgment correctly, that in the context of the 022 patent the term gesture was used more broadly. There was no requirement that the device should take account of any particular characteristic of the movement beyond its position. In my judgment this is quite clear from [0036] and [0051] of the specification, to which I have referred above.

"a predefined gesture"

132. This term appears in both features (iv) and (v) of claim 1, in the context of the phrase "if the detected contact corresponds [or does not correspond] to a predefined gesture". Both sides agree that what makes a gesture "predefined" is that the device has stored in it the required characteristics of the gesture which will unlock the device if the user's input matches it. HTC also accept that it is possible to build in room for error, in that the device may be arranged to accept a class of gestures. Thus the device may be arranged so that, if the predefined path is a horizontal gesture, the user will still unlock the screen with a slightly curved gesture. HTC maintain, however, that that there comes a point where the class of gestures is so widely defined that there ceases to be a predefined gesture at all.
133. Apple, on the other hand, submit that a predefined gesture is any gesture which the device will recognise as triggering the unlock function.
134. In my judgment, Apple are right about this point. HTC's construction places more weight on the word "a" than it can possibly bear. The argument loses any force it might have had once it is accepted that the claim admits of more than one gesture. I can see no practical reason why the patentee would want to limit the scope of his claims to any particular width of the class of gesture which might unlock the device.

"predefined displayed path"

135. The context of this term is also important: the claim requires "moving an unlock image along a predefined displayed path on the touch sensitive display in accordance with the contact".
136. HTC contend that this feature requires the device to define a specific path which the unlock image must follow. They submit that the requirement is not satisfied by merely providing a start and end point, leaving the user to determine the route to be followed between them. They submit that this is clear from the contrast drawn in [0064] and [0065]. HTC submit, further, that the requirement that the path be displayed means what it says, the required route of the unlock image must be visibly marked. They also submit that a displayed path is a more developed concept than a visual cue. An arrow indicating a direction of movement such as arrow 406 in Figure 4A is not a displayed path. On the other hand the channel 404 is an example of a displayed path (as well as being a visual clue).
137. Apple submit that the expression means a path which is (1) recognised by the device by reference to its start and end points and (2) displayed to the user. They submit that [0064] and [0065] are not describing separate, mutually exclusive embodiments, but a spectrum of mechanisms that could embody the invention. They submit that the use of the word "emphasis" in those paragraphs is consistent with this understanding.
138. I think it is clear from the context that the patentee is using the term "predefined … path" to indicate a requirement for a specific path, stored in the device. A path has start and end points and defines a route between them – it does not leave the user free to choose any route he likes for the path of the unlock image. If the device is neutral as to the path to be followed by the unlock image between the start and end points, then there is no predefined path.
139. Further, and as an additional requirement, the predefined path must be displayed. It follows from my view as to the meaning of "predefined path" that it is not sufficient merely to display the start and end points. A display of the path to be followed must be provided. What amounts to a sufficient display of the predefined path is better considered in the context of concrete examples. These are, after all, ordinary English words. In a rural context, a signpost pointing across an unmarked field does not display a path, even if one can see another signpost on the other side. On the other hand, a series of posts with arrows on them extending across the field may be a sufficient display of the path to be followed.

"in accordance with the contact"

140. HTC also sought to extract something from these additional words. They do not go as far as to suggest that the finger must engage the unlock image, but they submit that, reading the claim as a whole, there must be a relationship between the contact, the gesture, the path and the motion of the unlock image. This point only has significance in connection with one of the alleged infringements, the Arc unlock, and I will return to the point in that context.

"user interface lock state" and "transitioning to a user interface unlock state"

141. The "user interface lock state" is explained at [0045] to be a state where the device is powered on and operational but ignores most, if not all input. It is clear however that the lock state is not one in which the device only responds to one type of input. For example claim 5 expressly requires the device to be capable of transitioning from the unlock state to two different active states, dependent on different gestures, i.e. different inputs. Moreover, [0045] also makes clear that the device in the lock state "may respond to a limited set of user inputs" including, in addition to transitioning the device to an unlock state, powering the device off.
142. The term "unlock", and the more cumbersome "transitioning to a user interface unlock state", plainly cover changing the device from the lock state, to one where normal input is allowed. But the specification makes it clear that the term is being used more broadly than this. Thus, at [0075], it states:

"In some embodiments, the lock/unlock feature may apply to specific applications that are executing on the device 400 as a whole. In some embodiments, an unlock gesture transitions from one application to another, for example, from a telephone application to a music player or vice versa."

143. Similarly, [0005] refers to unlocking applications. [0045] says that the "lock state" may be used, amongst other things, to prevent unintentional activation of functions on the device.
144. Thus, in my judgment, an arrangement where certain functions, and only those functions, can be activated by a limited set of user inputs will involve a lock state. Effecting any one of those inputs so that the application can function normally will involve transitioning the device to an unlock state.

"Channel"

145. Claim 9 requires the predefined displayed path to be a channel. At [0067] the patent describes the channel in Figures 4A and 4B as "similar to a groove along which a slider switch slides".
146. Mr Thorley submitted, in my judgment somewhat optimistically, that the channel in the claim admitted of a certain amount of lateral freedom during the longitudinal movement, in contrast to a strictly defined groove. I do not think that this is the meaning that the skilled reader would attribute to the phrase. The displayed channel in the claim can be of any width, provided it remains recognisably a channel. I cannot see any technical reason why the skilled addressee would wish to exclude from the claim narrow channels, similar to the digital sliders with which he would already be familiar, or mechanical slider switches, nether of which provide for lateral movement. The passage at [0067] is intended, in my judgment, to draw the analogy with a mechanical slider switch which travels in a closely defined groove, not to distinguish from it.

Infringement

147. The relevant model names and numbers of HTC's devices which are accused of infringement are identified in the Product and Process Description ("PPD"). They are all mobile telephones with touch sensitive screens, with the exception of the HTC Flyer which is a portable tablet computer with a touch sensitive screen. The distinction between phones and tablets does not matter. For present purposes it is enough to state that the various devices are distinguished by reference to their use of one of three different types of unlock mechanism. The type of unlock mechanism used is dependent on the version of software which is installed on them. These three mechanisms are:

i) the Arc unlock,

ii) the Ring unlock, and

iii) the Icon mechanism.

All three mechanisms are said to infringe the independent claims 1, 6 and 18. In addition, the icon mechanism is alleged to infringe claims 5 and 17.

The Arc unlock

148. In its resting state, the locked screen in the Arc unlock mechanism looks like this:

149. The name of the operator, the time and the date are displayed on the Arc. Touching the screen at any location results in the date, time and operator name disappearing and the words "Screen locked" with a small padlock appearing on the Arc. In addition the words "Drag down to unlock" appear, with a set of three chevrons appearing on either side of the wording. The three chevrons are animated, so that the top chevron appears first, followed by the second and then by the third. The chevrons increase in intensity, so that the second is brighter than the first, and so on. The figure below shows the state of the screen once the third chevron has appeared:

150. To unlock the Arc lock screen, the user must perform a gesture which results in moving the Arc towards the bottom of the display. For example, the gesture can be initiated by touching down in the centre of the Arc and then moving vertically downwards, dragging the Arc, to unlock the device. Equally the device can be unlocked by touching down on the left hand end of the arc and dragging down in a diagonal movement to the bottom right hand corner of the screen. However there is no requirement in either case that the user lands on the Arc. The user can start the gesture in the area of the screen anywhere above the Arc and drag his/her finger down the screen towards the Arc. Once the finger passes over the Arc, the Arc is 'picked up' and the user can then drag the Arc downwards (i.e. below its starting position) in order to unlock the device. The Arc cannot be moved upwards. The chevrons move downwards with the Arc.
151. The movement of the Arc is determined by the y-coordinate of the touch only. When the user lifts his or her finger off the screen, two conditions must be satisfied for the device to unlock:

i) the point at which the user landed on the screen must be above, approximately, the point at which the lower black arc meets the sides of the screen;

ii) the distance the Arc has travelled in the y-direction from the point at which the user touched on and that at which he or she lifts off must exceed a set threshold. If the user lands on the display above the Arc, and picks it up, then the distance is measured from the top of the Arc.

If the conditions are satisfied, then the Arc continues to move on the display, and disappears from view.

Infringement - Arc unlock

152. It is sufficient to focus on the features of the claim which HTC submit are not present. Firstly, they say that the Arc unlock does not involve a predefined gesture. They say that the range of gestures which will unlock the device is too great to be described correctly as a predefined gesture. I am not persuaded by this point, which turns largely on the issue of construction which I have decided. There is a class of predefined gestures, albeit a large one, which conform to the unlocking criteria of Arc unlock mechanism.
153. Secondly, HTC submit that the unlock image, in this case the Arc, is not moved along a predefined displayed path in accordance with the contact. The Arc is not a free-moving object. The path of its movement is determined by the software. The path which the Arc has to follow in order to unlock has a start and end point and is recognised by the device. I therefore consider that the Arc, which for these purposes is an unlock image, is moved along a predefined path.
154. It is here that HTC submit that the requirement for the unlock image to be moved "in accordance with the contact", requires more of a connection between the movement of the finger and the movement of the unlock image than is present in the Arc unlock mechanism. They draw attention to the fact that although the image moves along a strictly regulated path, the contact does not. In my judgment, this argument is reading too much into "in accordance with the contact". The movement of the Arc is directed by the contact. Monitoring an aspect of the movement of the contact determines the movement of the Arc. This is enough for movement of the Arc to be said to be in accordance with the contact.
155. The critical question on this accused device is whether the predefined path is "displayed". Apple maintain that the displayed chevrons, coupled with the words which appear on the screen "Drag down to unlock" amount to a sufficient display of the path. They submit that the user would understand that he or she should drag their finger down between the chevrons. HTC submit that the chevrons are merely indicating direction, not a path. They point to the fact that the chevrons move with the Arc and are part of the unlock image, rather than a display of the path which it is to follows.
156. I have come to the conclusion that the instructions on the screen, coupled with the chevrons are sufficient indication to the user of a path which the Arc must be dragged along to amount to a displayed path. Professor Greenberg accepted that the natural inclination of the user would be to run his finger between the chevrons. It follows that the user is given a sufficient indication of the path the unlock image should follow. I am not persuaded that the fact that the chevrons move prevents them displaying a path.
157. It follows that if the patent is valid, the Arc unlock mechanism would infringe claim 1. The same is true of claim 6 and 18 which are also alleged to be infringed.

The Ring unlock

158. The Ring lock screen has a grey ring displayed in the centre of the bottom edge of the screen. Approximately one third to one half of the Ring is visible above the bottom edge of the screen. There is a semi-transparent 'arc' across the bottom of the display behind the ring. Four application icons appear above the ring.
159. A touch landing on, within or in the area immediately surrounding the Ring results in the application icons disappearing. If the user 'lifts off' without moving the Ring from its starting location, the application icons reappear, and the words "Pull the ring to unlock" appear. The figure below shows the Ring Lock Screen immediately after the touch has been 'lifted off':

160. A touch landing elsewhere on the screen and lifting off also causes the words "Pull the ring to unlock" to appear. At the same time, the Ring moves upwards from the bottom of the screen until one half to two thirds of the Ring is displayed above the bottom edge of the screen. The Ring then moves back down to its starting position. Immediately after the Ring moves back down, white lines are seen radiating outwards from the Ring in a ripple effect. The combined effect is to draw the user's attention to the Ring.
161. To unlock the screen the user makes a touch anywhere within the Ring or the area immediately surrounding it, and then, drags the Ring which follows the touch. To unlock the device, the bottom of the inside edge of the device must be visible above the bottom of the screen. If this criterion is satisfied the Ring appears to grow until it covers the whole of the screen, and the screen unlocks. If the criterion is not satisfied, the Ring reverts to its original position. Thus if the Ring is pulled sideways, the device does not unlock.

Infringement - Ring unlock

162. The points taken by HTC in relation to the Ring unlock are similar to those I have dealt with under Arc unlock. They say, firstly, that the range of gestures permitted with Ring unlock is too broad to be a predefined gesture. I do not accept this submission for the same reasons as in relation to the Arc unlock feature. It is correct that the device accepts a broad range of gestures, but on the approach I have taken, this does not avoid infringement.
163. Secondly, HTC submit that there is no predefined displayed path. No separate point arises on "in accordance with the contact", because in the case of Ring unlock the ring, which is an unlock image, moves directly under the user's finger. Unlike the Arc, however, the Ring is a free moving object. Whilst the required start of the route is defined, and the end point is very broadly defined, the device is neutral as to the route taken between the two. On the approach which I have taken to what amounts to a predefined path, the Ring unlock mechanism does not satisfy this feature.
164. If I am wrong about whether there is a predefined path, I should consider whether there is a displayed path. Apple submitted that the legend "Pull the ring to unlock" coupled with the initial up and down movements of the ring and the ripple effect were adequate to display a path.
165. Perhaps recognising the difficulty with this argument, Professor Keyson suggested that the manual provided with the phone assisted the user to understand the path. I do not think that instructions in the manual form a legitimate part of determining whether a path is displayed. Further, Professor Keyson sought to rely on the presence of the light coloured arc beneath the Ring as indicating a direction to pull. Neither point was adopted by Apple in their final submissions. In my judgment they were right to ignore these points made by Professor Keyson, which demonstrated the unbalanced approach he took to the issues.
166. It is fair to point out that Professor Greenberg was prepared to accept that the combination of indicia was sufficient to display to the user that the Ring should be pulled in an upward direction. But in my judgment this is a long way away from establishing that there is a displayed path. The user is not shown the point to which he must drag the Ring, or anything except the very broadest indication of direction, always assuming he interprets the various signals in that way. The Ring unlock feature does not have a displayed path. It therefore does not infringe claim 1, 6 or 18.

The icon mechanism

167. The icon mechanism operates by dragging an application icon into the ring to activate the application in question, directly from the lock screen. A touch landing on or in the area immediately surrounding any one of the application Icons results in the Ring moving up from the bottom of the screen towards the application icon which has been touched, until approximately three quarters of the Ring is displayed. The centre of the Ring turns a dark grey colour and a lighter grey shape appears in the centre of the Ring representing the application. The sequence of images below shows this behaviour, using the phone icon:

168. When the user lifts off, the words "drag icon into ring to activate" appear. At the same time, the image representing the application moves back to its starting position and the Ring moves back down to its starting position. Immediately after the Ring moves back down, white lines are seen radiating outwards from the Ring in a ripple effect.
169. To unlock the Ring lock screen, the user must make contact with the application icon or the area immediately surrounding it and drag the application image into a drop area. The user can move the image in any pattern on the screen before lifting off in the drop area. There is a drop area for each of the four application images. Each drop area is a rectangular area defined by x and y co-ordinates and includes some of, but not necessarily all of the Ring, and an area outside the Ring that is near to the particular application icon. If the icon is dropped in the drop area, the device unlocks and the ring behaves as in the Ring unlock case.

Infringement - Icon mechanism

170. The parties' arguments as to whether there was a predefined gesture were the same, and lead me to the same conclusion as for the previous devices.
171. Apple submitted that there was a predefined displayed path because the Ring moves towards the icon and the icon moves towards the Ring, whilst a silhouette of the icon appears within the Ring. In my judgment, as for the Ring unlock mechanism, because the icon may be moved along any path between its starting and finishing points, there is no predefined path. There is also no displayed path. The visual cues are, in my judgment, too vague to amount to a displayed path.
172. It follows that the icon mechanism does not infringe claims 1, 6 or 18. Although it is common ground that the icon mechanism has the additional features of claims 5 and 17, those claims are dependent on earlier claims, and are therefore not infringed for the same reasons.

German and Dutch decisions on construction and infringement of 022.

173. HTC drew my attention to a number of decisions on the corresponding European patent (DE) against manufacturers of devices which also use the Android operating system, Samsung and Motorola.
174. In Apple Inc. v Samsung Electronics GmbH and another, a decision of 2^nd March 2012, Judges Voss, Tochtermann and Schmidt sitting in the seventh civil chamber of the Mannheim Regional Court in Germany (Landgericht Mannheim) thought that:

"it can no longer be considered a predefined gesture "along a predefined path" … within the meaning of the patent in suit when the position-related movement of the unlock image [can be] chosen at will by the user in its entirety between the starting contact point and the end of the contact on the touch sensitive display."

175. My conclusions on the proper construction of "predefined path" are consistent with those reached by the Mannheim court. They further held that:

"The predefined path … is displayed within the meaning of the patent if the precise position-related progression of the movement of the unlock image (still) necessary for unlocking is as such visualised by the user."

176. Whilst there may be scope for argument as to what is a sufficiently precise visualisation of the movement, I believe my conclusions are also broadly consistent with this finding by the Mannheim court.
177. In Apple Inc. v Motorola Mobility Inc., a decision of 16th February 2012, Judges Guntz, Pichlmaier and Kopacek sitting in the seventh civil chamber of the Landgericht München, had to consider three separate embodiments of a Motorola phone. Two of these were held to use an unlocking mechanism with a predefined displayed path on the basis that there was a sufficient visual indication of the path to be followed, and the unlock image moved along a stored path. The third embodiment was held not to infringe. Here the user must either drag an inner circle to the contour of the outer circle and then lift his finger from the touch screen upon reaching the contour, or he can first touch a padlock symbol and immediately thereafter any point outside the outer circle to bring about unlocking. The court's reasoning at pages 30 to 31 of the translation is again broadly consistent with my own.
178. I am told that at a hearing on 4^th April 2012 there was a first oral hearing in the Landgericht München in Apple's claim against HTC on the 022 patents. The views expressed by the court on that occasion were provisional and there is no written judgment. The court expressed the view that the devices in issue in the present action, which were also in issue in the German proceedings, did not infringe because of the variety of the movements of the image which lead to unlock. I have come to a different view from the Landgericht München in relation to Arc unlock. However as there is no reasoned judgment from the court, the information supplied to me about this hearing does not cause me to reconsider my view.

Validity

179. HTC relies upon lack of novelty over a document referred to as Hyppönen, obviousness over a document and video called Plaisant and a prior phone called Neonode, added matter and excluded subject matter. An argument that the invention was obvious in the light of common general knowledge alone was not pursued at trial.

Hyppönen disclosure

180. Hyppönen is PCT Application number WO/038569. It was published on 8^th May 2003. The invention disclosed is of a method and apparatus for selecting a password applicable in particular to mobile devices which lack a physical keyboard, including stylus driven PDAs. The specification contains a discussion of the balance which needs to be struck between ensuring that the device is secure, and enabling relatively easy use.
181. In the embodiment of Hyppönen on which attention was focused, a user selects a value by selecting a graphically displayed element from a stored set on a graphical user interface. This is done by means of a graphically displayed representation of a slider on a touch sensitive display operated by a stylus. For each position of the slider, the value corresponding to that position is displayed to provide visual feedback to the user. Once selected, the value chosen is hidden from view.
182. Figure 2 gives an idea of the interface:

183. The user selects the values by operating each slider from a starting point until the value corresponding to his password is displayed. Figure 2(a) shows the starting position. In Figure 2(b) the displayed values are names of cities, and the user has selected Toronto as corresponding to his password. Figure 2(c) shows the display after the first and second values have been chosen. Once the user has selected all the sliders corresponding to the password, the device will respond to the entry of the password appropriately. As the device must be switched on to display this interface, the response to the successful insertion of the password must be to transition the device from a locked to an unlocked state.

Anticipation by Hyppönen

184. In their closing skeleton Apple submitted that Hyppönen lacked three of the features of claim 1, and accordingly did not deprive that claim of novelty. These features were that there was no predefined gesture (features (iv) and (v)), there was no predefined displayed path (feature (vi)), and the user does not interact with an unlock image in order to unlock the device (feature (vii)).
185. HTC's case depends on looking at the movement of the final slider, the user having correctly entered the relevant component of the password for all the sliders up to that point. What is then required to unlock the device is that the user touch on the slider and drag it to the point at which the final component of the password is displayed.
186. In my judgment, this final movement of the slider complies with the requirements of features (iv) and (v). The device is unlocked by the user performing a gesture. If the gesture corresponds to the user moving the slider from the starting point to the correct value, then the device will unlock. It is true that the class of gestures which may potentially unlock the device is wider than a single gesture. Thus, depending on the initial starting point of the slider, the gesture may be downward or upward. Moreover any of the sliders could be used to input the last component of the password, so the gesture may be performed at various different locations. However, consistently with the view I have taken on construction, and applied when dealing with infringement, the fact that the device may be programmed to accept a broad class of gestures does not take it outside the scope of the claim.
187. Rather less straightforward is the question of whether the unlock image is moved along a predefined displayed path. Plainly the unlock image, the slider handle, can be, and is moved along a path which is displayed. Mr Thorley submits that the random starting point of the slider, coupled with the absence of any teaching as to the direction in which the slider needs to move to achieve unlocking, and the absence of a displayed endpoint means that there is no predefined displayed path.
188. In deciding the issue of infringement of the Arc unlock device, I came to the conclusion that the directional signage and instructions in the alleged infringing device were adequate for there to be a displayed path. I did not think that the absence of a displayed end point was fatal to there being a path. The visual indicators were sufficient to show a path which the unlock image had to follow. It was not a necessary part of that finding that directional indicators were necessary for there to be a displayed path at all. Other means of indicating the existence of a path could be adequate.
189. The argument based on the disclosure of Hyppönen does not raise the question of whether there is a predefined displayed path at all: there plainly is. In the 022 patent's terms it is a channel. Moreover, at all points during the unlock procedure the unlock image remains on this predefined displayed path. The argument raises a different question, namely whether the predefined displayed path must also convey directional and end point information, or whether it is enough to present the user with an unlock image on a path, leaving it up to him to find the end point is by moving the unlock image along it.
190. I agree with HTC that there is nothing in the meaning of "predefined displayed path" to exclude the Figure 2 mechanism of Hyppönen. It is true that this amounts to a less user friendly unlocking mechanism than the depicted embodiments of 022. But, as [0045] makes clear, the locked state may also be being used to prevent unauthorised use. In these circumstances one may not wish to display to the user the direction or end point of the required movement on the displayed path. Moreover, as HTC also submit, the Hyppönen mechanism does provide a degree of user-friendliness as compared with simple memory based password systems. I can see no reason why the reader of 022 would think that the patentee was seeking to exclude embodiments which deliver such advantages.
191. Finally, it seems plain to me that the unlock image, the slider "handle" in Hyppönen, is a graphical, interactive user-interface object with which a user interacts in order to unlock the device. Apple submitted that what the user did was to move the sliders in order to make a selection from the words appearing above them. That is true, but whilst doing so the user is interacting with the slider to unlock the device.
192. It follows that Hyppönen is an anticipation of claim 1, 6 and 18. Apple submit that Hyppönen's slider is not a channel, as required by claim 9. This argument turns on Apple's construction of channel, which I have rejected. It follows that claim 9 is also not novel in the light of Hyppönen.

Plaisant disclosure

193. The citation which has been referred to in this trial as Plaisant is a video and accompanying paper dating from 1992 by Catherine Plaisant and Daniel Wallace. Because of this cross-reference, it was common ground that it was legitimate to read the paper together with the disclosure of the video.
194. Catherine Plaisant was a member of the HCI department at the University of Maryland. The co-author, Daniel Wallace was, at the time that the underlying research was done, a graduate psychology student in the Psychology Department at the same university. The skilled person would have regarded these disclosures as coming from an authoritative source. The video describes the results of a usability study of six different toggle switches to control two state (on/off) devices on a touch screen. The work was conducted in collaboration with a company which specialised in the development and marketing of integrated entertainment, security and climate control systems for homes and offices. The control of these multiple devices is effected through touch screen interfaces.
195. Plaisant explains that computer based toggle switches can be confusing in a variety of ways. One type of confusion to which she refers is that between state of activation and the label for possible action. Thus the user may not readily appreciate whether the label "ON" indicates the state of the device (i.e. already "ON") or a button to press to switch it on (i.e. currently "OFF"). Another type of confusion is uncertainty as to what to do to activate the device. She gives the example of a slider where only touches to the end of the slider were possible but "sliding" is not permitted.
196. The toggles which were the subject of the usability study are illustrated in Figure 2 of the Plaisant paper (as well in the video). They are referred to as the one-button, words, two-button, rocker, slider and lever. I show this figure below:

197. The operation of the slider toggle is described as follows:

"Slider toggle: in this toggle a sliding/dragging movement is required to change the position of the yellow pointer from one side of the toggle to the other. A simple three step animation shows the movement of the pointer along the slide. If the device is ON the pointer is on the ON side. Users can then grab the pointer and slide it to the other side. If the finger is released before reaching the other side the pointer springs back to its previous position. "

198. Plaisant explains that the usability testing showed that the toggles which pushed were preferred over the toggles that slid. She thought this was possibly explained by the fact that sliding was more complex than simply touching. They also noticed that sliders were more difficult to implement. The usability test brought to light some imperfections in their slider design. However, all subjects (spontaneously or after one trial) successfully used sliding motions to manipulate the toggles. The paper continues:

"Even if sliders were not preferred, the fact that users used them correctly is encouraging since many other controls can be designed using sliding motions. Another advantage of the sliding movement is that it is less likely to be done inadvertently therefore making the toggle very secure (the finger has to land on and lift off the right locations). This advantage can be pushed further and controls can be designed to be very secure by requiring more complex gestures (e.g. a U or W shape slider can be used for a 2 or 3 setting control respectively)."

199. The Plaisant video shows a number of slider switches organised into an array, each controlling a separate function:


Was Plaisant CGK?

200. Whilst I have no doubt that the skilled person would have regarded Plaisant as a reliable and authoritative document, I am, on balance, not persuaded that it was common general knowledge at the priority date. I accept that the work was presented at one of the most important conferences on HCI, the CHI 1992 conference, that it came from an influential group and that some teachers, such as Professor Greenberg, referred their pupils to it and included it in the video materials available as part of the Open Video Project. The work was certainly widely known and shown, widely read and viewed and widely published. To hold that the entire contents of the paper and video were part of the common general knowledge requires something more.
201. Plaisant was referred to in two editions of a text book by Shneiderman and Plaisant before the priority date, but reference to it was removed in the 2005 edition. Given the large volume of material similar to Plaisant that was available, I am not satisfied that the skilled person, faced with a problem in the field of HCI design at the priority date, would necessarily find his way to this material. I conclude that it is not common general knowledge.

Obviousness over Plaisant

202. I have identified the skilled person and the common general knowledge he or she would possess. I have also determined the inventive concept of 022 by construing the claims: neither side volunteered any more pithy paraphrase. The first difference between the disclosure of Plaisant and the inventive concept of claim 1 of 022 lies in the fact that Plaisant does not disclose the use of her slider toggles in a portable electronic device (feature (i)).
203. A second difference between Plaisant and the inventive concept of 022 is that it is not accurate to describe what is done in Plaisant as transitioning a device between a user-interface lock state and a user-interface unlock state. I have indicated above that these terms have a quite extended meaning in the 022 patent, including unlocking an application and transitioning the device from one functionality to another. Nevertheless, this difference also forms part of the gap between Plaisant and the inventive concept of claim 1.
204. Apple maintained that there was a third difference, namely that the gesture disclosed in Plaisant was "a poor implementation of a gesture". By this they meant it was safe but not user friendly. Even if correct, this is not a difference between the disclosure of Plaisant and the inventive concept, which extends to any gesture. If correct, it might be a factor which the skilled person might have in mind if deciding whether to adopt Plaisant's sliders, and will need to be considered at that stage.
205. Having identified such differences as there are, it is of course necessary, for the purpose of the next stage of the inquiry, to forget about the differences and to ask only what if anything it is obvious for a skilled person to do in the light of Plaisant.
206. HTC argue as follows. Firstly they say it would be obvious to the skilled person that the Plaisant sliders could be used on other devices, including portable devices. Secondly they say that the fact that Plaisant speaks of turning devices on and off, whereas 022 refers to lock/unlock of a user interface, is a trivial difference. Thirdly they say that the skilled person would know that accidental activation was a problem with touch screen portable devices. Reading Plaisant with this problem in mind the skilled reader would see an obvious way of dealing with accidental activation.
207. Apple contend that the starting point for considering obviousness in the present case is to provide an intuitive, user friendly graphical unlock means which avoids the possibility of unintentional activation of functions through touching the screen. The common general knowledge way of providing this was through some form of mechanical keylock function. Apple go on to build on the fact that HTC no longer pursue their attack based on common general knowledge alone. As the common general knowledge already provided sliders, such as those used in the Windows CE operating system, the only additional disclosure in Plaisant was of the use of these known sliders to provide on/off functionality, which did not make the invention obvious. The removal of the reference to Plaisant from the Shneiderman and Plaisant book meant that the authors themselves no longer regarded it as relevant to any problem. Finally the problem of accidental activation has been known for some years, yet Plaisant's on/off slider had not to anyone's knowledge been implemented on a portable device for any purpose.
208. Professor Greenberg's evidence in his first report included this:

"73. Catherine Plaisant's goal was "to select a usability-tested/error-free toggle and to better understand some of the problems and issues involved in the design of controls for a touchscreen environment" (page 667, column 2, lines 9-12 of the Plaisant Paper). The Plaisant Prior Art is merely describing a transition from one user-interface state to another. Transitioning a device from a lock state to an unlock state is simply one particular example of such a state change. In Catherine Plaisant's implementation, she has chosen to label these two states as "on/off", but the labels do not matter: her work applies equally to transitions between any two states, including "lock/unlock". Catherine Plaisant's goal also applies to any underlying functionality whose user-interface state change is controlled by toggle switches.

74. Consequently, at the level of the graphical user interface, there is no difference between the Plaisant Prior Art and the '022 Patent. It would have been obvious and trivial to the Skilled Person at the '022 Priority Date to use and implement the techniques described in the Plaisant Prior Art to transition a device from a locked to an unlocked state."

209. I am not prepared to accept that, in the 022 patent's terms, Plaisant actually discloses a change from one user interface state to another. However, that aside, I accept Professor Greenberg's evidence that the skilled person would see the general applicability of the toggles for changing the state of a device. Although it was suggested to Professor Greenberg in cross-examination that the focus of Plaisant was only on on/off switches, he plainly remained of the view set out in these paragraphs.
210. Professor Keyson pointed out in his second report that Plaisant does not disclose a transition from a user interface lock state to a user interface unlock state. The actual user interface shown in Plaisant is always unlocked. This amounts to saying that Plaisant's toggles are not controlling the lock unlock state of the Plaisant user interface, which I have already accepted is correct: they are controlling the state of the relevant devices.
211. Professor Keyson's reasons why he thought that the invention was not obvious in the light of Plaisant drew heavily on his interpretation of the term "gesture". He plainly had in mind some more developed definition of what amounted to a gesture, which is not the definition adopted by the 022 patent, or a definition which Apple themselves in the end felt able to embrace. Professor Keyson also thought that the fact that the specific embodiments of 022 only showed one-way unlocking, whereas Plaisant showed two way toggles, amounted to a relevant difference. This too is wrong, as the claims of 022 do not exclude a two-way toggle switch. Professor Keyson also drew attention to the fact that a HP Jornada pocket PC had a touch screen which could be activated by tapping the screen. Users were warned of the dangers of accidental activation while sliding the device into its protective pouch. A slider based unlock mechanism would have avoided the need for this warning, but was not implemented in the HP Jornada, or any other device before the Apple iPhone.
212. In my judgment the invention of claim 1 would be obvious to the skilled person in the light of the disclosure of Plaisant. I think the evidence of Professor Greenberg is a more realistic reflection of the approach of a skilled person to that disclosure than the evidence of Professor Keyson, which was, as I have indicated, based on misconceptions as to the nature of the inventive concept. The skilled person would readily understand that Plaisant toggle switches could be used to control the state of a device, including a user interface state within the meaning of the 022 patent. Given that it is generally known that the screen will accept input to lock and unlock the device, I can see no technical reason why the skilled person would not implement a Plaisant toggle to control locking and unlocking. The skilled person would appreciate that the slider toggle would have advantages over the on/off toggles, as it would provide protection against accidental activation of the toggle itself, as Plaisant explains. Professor Keyson was constrained to accept that if it came down to choosing one of the switches, the slider would be one of the preferable ones.
213. Turning to the points advanced by Apple, firstly, I do not accept that Plaisant's sliders would have been perceived as poor forms of gesture by the skilled person at the time. All the users managed to use them successfully. The use of sliders of the Windows CE variety was part of the common general knowledge. In addition they would be seen as easy to implement. Secondly, the skilled person would have no difficulty in visualising the lock/unlock functionality as a switch analogous to the common general knowledge mechanical switch for protecting the iPod, or the sliders used to lock touch screens. Thirdly, I do not think that much can be built, as Apple seek to do, on HTC's abandonment of their case based on common general knowledge. Mr Thorley submitted that Professor Keyson's evidence was that he could not see any benefit in the disclosure of Plaisant, when slider controls were already in use in commercial touch screen products. However, I did not gain the impression that this was what Professor Keyson was saying in the passage of evidence relied on. His evidence was that, given the Windows CE sliders, he would see no reason to go and look for the slider in Plaisant. That is a different question. HTC's obviousness case assumes, as they are entitled to, that the skilled reader faced with the known problem is reading Plaisant with interest. I cannot see how it can be said that Plaisant teaches nothing new: the Windows CE sliders did not draw the analogy with mechanical on/off sliders in the graphic way that Plaisant does.
214. Mr Thorley also relied on the following passage of cross-examination of Professor Greenberg:

Q. It was appreciated by 2005 that sliders of this nature [i.e. Plaisant] did have a place in touchscreen device ----

A. I would say it would be one of the things that could be considered when making design choices, yes.

Q. It was considered, was it not? It had been considered?

A. It certainly had been considered in this article, yes.

Q. No, it had been considered in the Palm environment we looked at with -- it was looked at with Professor Keyson.

A. So maybe I should explain what I mean by "considered"?

Q. Yes, of course.

A. We are talking about a skilled person who has to decide what features they want to implement on their touchscreen, in this case on the touchscreen device, be it portable or not, depending on -- sorry, be it portable or not. A person in that role, their job is not to just blindly say, "Here is one technique and we will apply it". Their job is to consider the interface as a whole. Part of their job and part of, in fact, the majority, a large part of what they have to do is to consider all the interaction techniques at their disposal and to make choices between them. So one could consider a broad variety of interaction techniques and then choose one which best fits, for example, the kind of interface they have, the kinds of things being done on it, the other aspects of the interface in terms of consistency, other kinds of tasks the person is performing, the context of use and so on. There are many, many decisions or many factors that a person would have to make. So when I say "considered", I mean that the skilled person would be considering this amongst one of the many other options.

215. I did not think that Professor Greenberg's explanation of "considered" really took Apple's case anywhere. It is of course relevant to the assessment of obviousness that there may be a number of possible avenues for the skilled person to go down. However, Professor Greenberg was not, in this passage, qualifying his evidence about the obvious avenue to go down from Plaisant. It was not suggested to him that the other options to which he referred in the course of explaining what he meant by "considered" would eclipse the obviousness of taking the step from Plaisant to the patent.
216. Finally, whatever may have been the reason for the removal of the Plaisant material from the Shneiderman and Plaisant textbook, I do not think that the evidence showed that the skilled reader would reject the teaching of Plaisant as lacking relevance at the priority date. The work comes from a stable to which great attention would be paid. The video shows how well the toggles work.
217. Beyond this, Apple's case of non-obviousness is based on asking the forensic question "if it was so obvious, why was it not done before"? Professor Greenberg ventured the suggestion that the invention came at a time when screens were becoming more robust, so that covers were no longer needed. The need to consider how to deal with a touchscreen without a cover was therefore not such a longstanding one. I accept that explanation. In any case I am wholly unable to see how the idea of implementing Plaisant's slider in a portable device to control locking or unlocking could be said to require an inventive step.

Neonode disclosure

218. The Neonode N1 was a mobile telephone launched in July 2004 in Sweden and by November 2004 in other European countries. The N1 incorporated a touch screen which was locked when the phone was not in use.
219. A number of versions of the unlock screen were incorporated into the phone. The first, referred to as the "text" screen had an image of a padlock and the words "Right sweep to unlock" displayed below it thus:

220. In late 2004 the text lockscreen was replaced with the "arrow" lockscreen. In this screen the text was replaced with a lighter coloured arrow, seen faintly in the following picture adjacent the user's thumb:

221. A third variation, the N1M, was launched at a later date, which is in dispute. The design was altered again by replacing the arrow with 3 small chevrons pointing from left to right. I was not persuaded that HTC had discharged the onus of proving that the chevron version was made available to the public by the priority date. The evidence of Mr Gustaffson, put in by Civil Evidence Act Notice, seemed to me to leave considerable doubt as to when this third version was made available. He could not himself remember, and the documentary support he relied on was ambiguous.

Obviousness over Neonode

222. I have dealt with the inventive concept, the skilled person and the common general knowledge. HTC rely principally on the arrow unlock feature of Neonode. The only difference between the disclosure afforded by the Neonode arrow unlock feature and the inventive concept of claim 1 is the absence of an unlock image with which the user interacts and which is moved along the predefined displayed path (feature (vi) and (vii)). The sole question is therefore whether it was obvious to add these features to Neonode, without knowledge of the invention.
223. The evidence showed that the Neonode was regarded as an innovative design when it was launched. Aspects of the design of its graphical user interface were, however, the subject of criticism at the time. I am satified on the evidence that the skilled team would appreciate that the arrow unlock feature suffered from a lack of feedback.
224. HTC's case is that it did not require invention to improve the user interface by providing an unlock image in the form of a cursor which the user could drag along the unlock arrow.
225. Professor Greenberg's evidence supported HTC's case. His evidence in his first report was the following:

"89. Upon observing the Chevrons Unlock Feature and the Text Unlock Feature I noted that a number of obvious modifications could be made to improve upon them. For example, neither unlock feature provides the user with any feedback. It would be routine, and part of the standard design process, to provide some form of object on the user interface with which the user can interact, in conjunction with feedback, so that the user knows that progress has been made towards unlocking and/or whether they are carrying out the correct action.

90. The chevrons and the text "Right sweep to unlock" provide no more than an indication of the direction in which the swipe should be made. The Skilled Person would view it as a straightforward improvement to place a feature on the user interface, the visual affordances and constraints of which were such that the userwould know clearly what they had to do to unlock the touch screen and in which specific part of the screen they had to make the required input. One possible obvious addition that I put forward to PG (before seeing the '022 Patent) that would address all of the above suggestions in this and the preceding paragraph would be to add some sort of slider to the user interface, where the user would have to drag an object/handle along a slider in order to unlock the touch screen."

226. Professor Greenberg was cross examined on the basis that he had not formed, and was incapable of forming, an objective opinion on the subject having seen the implementation of the invention in the iPhone. I have mentioned this point above when dealing with the witnesses. Professor Greenberg, rejected this suggestion:

"A. Again, I can only restate that I was putting myself in the shoes of somebody in 2005, this is a really, really basic improvement on a device, there is nothing unusual about that at all. The fact that I had an iPhone, yes I had an iPhone. Would it have changed my opinion at all if I had not? No. That is -- I will stay firm on that. That is what it is."

227. Professor Keyson's answer to this evidence centred on three main points. Firstly, he pointed out that although the provision of feedback was well known, the form which that feedback takes needs to be decided. Secondly, he said that he was not aware of any products on the market prior to Neonode which had a swipe gesture and gave visual interactive feedback. Thirdly, he observed that, despite a series of modifications to their user interface, Neonode did not implement the improvement.
228. Professor Keyson found it extremely difficult to address the question of whether it would have been obvious to provide an unlock image, at points more or less refusing to do so. I have re-read this passage of his evidence, and found little or nothing in the evidence which assists Apple's case.
229. I consider that it would be obvious to the skilled team, faced with the lateral-swipe arrow unlock of Neonode, that it could be improved by the provision of feedback. The skilled team would be aware that visual feedback for a lateral gesture could be provided by the extremely familiar sliders from his common general knowledge, such as the Windows CE slider.
230. It is true that this simple improvement was not done by Neonode. This is a secondary consideration which may in some circumstances support a case of inventiveness. On its own, which it would be in this case, it is of little weight.
231. Claim 1 is obvious in the light of Neonode.

Sub-claims

232. HTC maintain that claims 5 and 9 are obvious over Neonode and Plaisant.
233. In my judgment claim 9 is plainly obvious over Plaisant: Plaisant discloses a channel, and the channel would form part of the obvious implementation of Plaisant.
234. Claim 5 adds the feature that the device can be moved to a variety of different unlocked states from the locked state. Beyond the initial unlock screen, Neonode disclosed the use of three swipe gestures in order to unlock different applications: the start menu, the keyboard menu and the tools menu. Once it is accepted that claim 1 is obvious in the light of Neonode, it seems to me that claim 5 is obvious as well. The skilled person would readily see the applicability of the swipe-with-feedback to unlocking a plurality of applications.

Added matter

235. The added matter objection to 022 is put so clearly in HTC's skeleton argument that I will quote it verbatim:

"The application as filed discloses the use of a pre-defined path (see [0069]).

The teaching of a pre-defined displayed path is at [0071]. This teaching refers to figures 4A-4B which include visual cues which display a channel 404 indicating the path of the gesture/movement along which the unlock image 402 is to be dragged. Para. [0071] teaches only the use of a channel as the visual cue to display the pre-defined path.

The rest of the teaching and the figures also only disclose the use of channel as a visual cue to display the pre-defined path. Nowhere in the application as filed is there a disclosure of displaying a predefined path by means other than a channel.

Yet in the 022 patent as granted, claim 1 is not so limited and covers any pre-defined displayed path (whether or not a channel). There is no basis for this in the application as filed. In the patent as granted, the fact that the pre-defined displayed path is a channel has been relegated to claim 9.

Claims 1 and 6 and the dependent claims (other than claim 9) add matter."

236. I think the answer can be put with equal conciseness. It is a tenet of the argument that both documents contain a disclosure of a predefined displayed path. Both documents disclose a channel, and both parties agree that a channel is a predefined displayed path. For there to be added matter, therefore, the granted patent must disclose a predefined displayed path other than a channel. It is true that claim 1 of the granted patent covers a predefined displayed path other than a channel. However, it is not possible in my judgment for the skilled person to obtain any clear or unmistakable direction from the disclosure of the granted patent as to what displayed paths there might be, other than channels. On this subject, the reader of the granted patent would be no better informed than the reader of the application as filed. The added matter objection therefore fails.

Excluded subject matter

237. I can deal with this briefly in the light of my findings thus far. If I am wrong about anticipation, and there is some contribution to the art, the contribution lies in the way in which the user interacts with the computer, providing more feedback than Neonode, or applying Plaisant's switch to unlocking the screen of a portable device.
238. HTC put their case based on excluded subject matter primarily in relation to the contribution over Neonode. They say that the contribution is merely visual information about where the contact point is on the path at any given moment. A swipe gesture across the screen will unlock the prior art device. The contribution is the mere presentation of information about how that gesture is progressing. The claimed device and the patented device both work in the same way. All that has changed, according to HTC, is that the device is now programmed to provide visual information to the user. Thus the contribution is either a computer program as such or the presentation of information as such.
239. Apple contend that the contribution is more than this. They submit that, compared to Neonode, it is not simply a matter of giving the user more information about how to unlock the device, but it provides a better, more secure device. They rely on what Professor Keyson said:

Q. What 022 gives you by way of the unlock image and its associated channel, that you do not get from Neonode, is that the user is given some information about what to do and whether they are doing it successfully.

A. Among other things, yes.

Q. What are the other things?

A. I just explained that in my previous ----

Q. You mean their anxiety, that sort of thing?

A. It is providing an incremental path that you can start off slowly and understand so it is a very secure way of -- I mean secure in the sense it is secure from a user experience viewpoint, so it is an easy to understand way that it is a very simple way to do it and yet it is a very safe way so it is not going to copy accidentally an unintentional thing, and it teaches me how to do this sweep gesture so it creates this common understanding. So it is not just about for that moment, that unlock, it is teaching me about how to use the product as a whole, it is teaching me what a gesture is, it is teaching me about a sweep gesture which I can then apply later through the interface. The teaching experience, let us say, is not specifically limited only to the unlock mechanism.

240. Apple also rely on what Professor Greenberg said in cross-examination:

Q. Would it be fair to say that the specific embodiment of 022 provides a more intuitive user interface than was provided by Neonode?

A. Yes, that is fair.

Q. It is a better way of providing an unlock mechanism to avoid the consequences of accidental touches than is provided by Neonode?

A. Yes, that is correct.

241. I think Professor Keyson was reading far too much into the contribution of 022. Nevertheless, I think there was a contribution here which went beyond a computer program as such or the mere presentation of information. There is a sense in which the invention provides a technical effect outside the computer, namely an improved switch. Moreover this is a real world effect which is not limited to the presentation of information. Whilst the subject matter of the invention is obvious, the patent is not invalid for excluded subject matter.

The 868 patent

242. The 868 patent is entitled "Portable electronic device for photo management". It has a priority date of 6^th September 2006, between 8 and 9 months later than 022.

The witnesses

243. The expert witnesses on this patent were the same as those on 022. I have dealt with my assessment of these experts when dealing with 022 above.

The skilled addressee

244. The experts agree that the 868 patent is addressed to the same person or team as the 022 patent. I have already identified the skilled person in relation to 022 when dealing with that patent.

The specification and claims

245. 868 is a relatively long document. The disclosure encompasses a large number of aspects of the user interface of a portable device. Only a small part of it is relevant to the present dispute. Apart from the introduction, the relevant disclosure is that at [0140] to [0157]. This passage describes the invention by reference to figures 23 and 24, and I summarise it below.
246. The background section to the specification explains that, as portable devices have become more multi-functional, it has become more challenging to design a user interface. Some portable devices have simply added more push buttons, which, according to the specification, has added complexity and inflexibility. At [0005] the specification explains that, although mobile phones with built-in digital cameras have been on the market for some time, they are difficult to use for even basic photo-related operations such as displaying, deleting and sending a photo because of limitations with the cell phones' user interface.
247. At [0007] the specification identifies a need for "portable multi-function devices with more transparent and intuitive user interfaces for photo management". Beyond this, at [0017] , the specification records neutrally that an aspect of the invention involves a computer-implemented method which has the features of claim 1. After describing various other "aspects of the invention" in a similar fashion, [0021] claims that the invention provides "a transparent and intuitive user interface for managing photos on a portable electronic device with a touchscreen display."
248. One turns, then, to the description of figures 23 and 24 at [0140] onwards. Figures 23A-23H are said to describe "an exemplary user interface for viewing digital objects in a set of digital objects in accordance with some embodiments". Figure 23A shows a displayed digital object which is an "entire image" on what is obviously a touch screen display. The entire image is a picture of two people, drawn schematically. The first step performed is a de-pinching gesture near the right-hand person, causing that part of the image to be zoomed in. The effect is to show only that person, as shown in figure 23B:

249. The user then performs a right-to-left swipe gesture. This causes the image to be translated across the screen in the direction of the swipe, and the edge of the image to be displayed, along with an area beyond the edge, shown in black in figure 23D:

250. The display of the area beyond the edge in Figure 23D "lets the user know the edge of the (enlarged) digital object has been reached during the first gesture". When the user releases his or her finger (or the stylus) from the touchscreen, the image is translated back until the area beyond the edge is no longer displayed. The specification explains at [0150] that in some embodiments this reverse translation may make the edge of the image appear to be elastically attached to an edge of the touch screen display. Although the specification does not use this term, this effect is referred to as "bounce back".
251. The user then performs a second swipe gesture. This causes the image to be translated in the direction of the swipe, leaving the screen, whilst at the same time a second digital image moves onto the screen. This is shown in figure 23 G, with the second image being that of an approaching train:

252. Finally the second digital image occupies the screen in place of the first.
253. The first and second swipe gestures therefore have different effects. The first gesture only causes translation and bounce back. The second gesture causes the transition from the first image to the second image in the set. The purpose of displaying the area beyond the edge with the first swipe gesture is that it lets the user know that the edge of the digital object, which was previously outside the edge of the screen, has been reached during the first gesture. Although not explained in these terms in the specification, the edge allows the user to orientate himself or herself within the zoomed image, and avoid getting "lost".
254. It is envisaged at [0148] that the digital object is a part of a set such as an album or set of images taken with a camera in the device. The set of digital objects may be a set of web pages or a set of electronic documents. At [0154] it is explained that the first and second swipe gestures may be linked in the sense that unless the time between the two gestures is less than a predetermined value, the device will not transition to displaying the second digital object in response to the second swipe.
255. At [0155] the specification explains that in some embodiments if the entire first digital object is displayed, as in figure 23A, then the first gesture will transition the device to display another digital object in the set of digital objects.
256. Apple rely on claims 1 to 3 and 7 to 10. They accept that independent claims 1, 7 and 8 stand or fall together in relation to the obviousness attack. Claims 2 and 9 and 3 and 10 are pairs of claims. So the ones that matter are 1, 2 and 3. For the moment it is sufficient to set out claim 1, with some reference numerals added:

(i) A computer implemented method, comprising:

(ii) a device with a touch screen display:

(iii) detecting a first movement of a physical object on or near the touch screen display;

(iv) while detecting the first movement, translating a first digital object displayed on the touch screen display in a first direction,

(v) wherein the first digital object is associated with a set of digital objects; characterised in that:

(vi) in response to display of a previously hidden edge of the first digital object and continued detection of the first movement, displaying an area beyond the edge of the first digital object;

(vii) after the first movement is no longer detected, translating the first digital object in a second direction until the area beyond the edge of the first digital object is no longer displayed;

(viii) detecting a second movement of the physical object on or near the touch screen display;

(ix) and in response to detecting the second movement while the previously hidden edge of the first digital object is displayed, translating the first digital object in the first direction and displaying a second digital object in the set of digital objects.

Construction

"digital object" and "set of digital objects"

257. HTC contend that the skilled person would understand a digital object to be a digital entity which could be treated as an independent unit. Apple contend that a digital object is an object shown on the screen of the device and perceived as an object in its own right by the user of the device.
258. The difference between the parties relates to whether one can treat, as HTC wish to do, individual images on a single page such as a web page as digital objects. Prof Greenberg said under cross-examination:

"In computer systems, we rarely if ever compose a unit as a single thing. A very, very standard process is to actually build it up of constituent parts, each itself of digital objects. Like a webpage, for example, is often composed of a hierarchy of digital objects and so characterising it as a single unit is not how we would do it and it is not how we actually perceive it as well. A webpage is [made up of] many constituent parts that can be acted upon independently."

259. In my judgement, the term "digital object" was not one which had a clearly defined meaning in the art. As Prof Greenberg accepted, the term was one which, to a degree at least, took its meaning from the context in which it appeared. Nevertheless, I cannot accept Apple's construction, which focuses almost entirely on the user's perception. The patent is not primarily addressed to the user. The skilled addressee of the patent would understand its disclosure to be related to the manipulation of digital objects as he understood them. This would include digital objects within a hierarchy of digital objects on a single page, or a single page made up of a number of digital objects as well as those things which Apple contend to be digital objects.

"display of a previously hidden edge"

260. HTC contends that an edge is simply the last row of pixels on the screen. They further contend that "hidden edge" just means an edge that is off the screen, and "a previously hidden edge" means an edge that was off the screen but is now on it. Thus when an entire image is on the screen, or when an enlarged image is at the edge of the screen, none of its edges is hidden. For an edge to be hidden, at least some pixels of the image on that side of the object must be off the screen. HTC contend that this interpretation ties in with the fact that claim 1 relates to zoomed in mode. Hence the starting point for the translation initiated by the first swipe gesture is an image whose edges are off the screen.
261. Apple submit that "edge" means simply the infinitesimal boundary between where the image stops and where whatever is next to it starts. They go on to submit that whether an edge is "hidden" is a question of fact which cannot be determined simply by asking whether in fact all of the pixels of the image are on the screen. The test is whether the user is able to observe and thus know that he or she is in fact seeing the edge of an image. The test is therefore one of the user's perspective, not one of technical fact.
262. HTC point out that Apple's construction introduces a subjective element into the claim. The question, on Apple's construction, is not whether the edge is in fact hidden, but whether the user is able to identify the edge as the edge. As HTC observed, this construction gives rise to considerable difficulties. Thus a person who takes a photograph in full screen mode will know that what is shown on the screen includes the edges of the image. However if the device is passed on to somebody else, they will not know. Thus whether a device has a hidden edge depends, on Apple's construction, on what information the user has previously been supplied with.
263. Apple further submit that it is wrong to assume that claim 1 is directed to zoomed-in mode. They draw attention to claim 2, which adds the requirement that "prior to the translating while detecting the first movement, at least one edge of the first digital object extends beyond the touch screen display in the first direction." Apple submit, therefore, that it is claim 2 which introduces zoomed-in mode, and that claim 1 is not so restricted.
264. In my judgement HTC are correct on this issue of construction as well. The only basis for claim 1 is the description of the embodiments by reference to figures 23 and 24. In this description, the first gesture is always performed on a zoomed-in image. The only reference to translating an entire image is in [0155], but this approach is not the subject of claim 1 at all, does not employ a second swipe or bounce back, and does not require the user to make use of the edge or area beyond the edge.
265. I do not find the argument based on claim 2 to be very helpful. It is of course correct that it is proper to infer in some circumstances, as a matter of construction, that if a subsidiary claim introduces a feature into a claim, then that feature is not either an express or implied requirement of the claim on which the subsidiary claim depends. However, as HTC point out, claim 2 does not introduce a requirement for zoomed-in mode, but a particular arrangement of zoomed-in mode in which an edge extends beyond the touch screen display in the first direction. An image may be zoomed-in without satisfying this requirement. The only proper inference to draw as a matter of construction is that claim 1 is not limited to this particular form of zoomed-in mode.
266. Even putting these slightly Chancery arguments to one side, I do not think that the skilled person would read "hidden edge" in the way in which Apple contend. With so little in the way of disclosure in the specification to aid interpretation, the reader would in my judgment assume that the background to claim 1 was zoomed-in mode, and that a hidden edge was one outside the limits of the display. The display of the previously hidden edge is what lets the user know that he has reached the edge during the first gesture i.e. during continued detection of the first movement. This language is not apt to describe the case where the edge of the image is already present or has been reached before the first gesture commences.

Infringement

267. Apple allege that the operation of HTC's photo application, Gallery, infringes claims 1-3 and 7-10.
268. There are two relevant modes of showing images in the HTC Gallery application: full screen mode and zoomed-in mode.
269. In full screen mode, the whole of a single photo is always shown on the screen. The user can, by using a swipe gesture, move from one photo to the next in the Gallery. However the swipe does not always move the display all the way to the next photo. This is because there must be a minimum swipe length. If the threshold of the minimum swipe length is not crossed, the display slides back to the first photo.
270. The issue of infringement of claim 1 by HTC's devices turns exclusively on construction. On the construction which I have accepted, HTC's devices do not do anything "in response to the display of a previously hidden edge". They do not infringe, therefore, in full screen mode.
271. In zoomed-in mode in HTC's Gallery, the user can drag the image around the screen. Panning all the way to the edge results in the image coming to a stop. No area beyond the image is shown, and the movement does not reverse. Once this dead stop point has been reached, however, further gestures will allow movement to the next photo, or return, in the same manner as with the full screen mode and depending on whether the gesture exceeds the threshold or not.
272. This dispute again turns on construction. I did not understand Apple to advance any case of infringement if HTC was correct on construction. Apple focus on the steps after the photo has hit the dead stop. They are right to do so, as the gesture which causes the image to arrive at the dead stop plainly does not result in the display of an area beyond the edge. However, once again, these steps in the HTC Gallery do not do anything in response to the display of a previously hidden edge. At the commencement of the claim's first gesture the edge of the image is aligned with the edge of the screen. There is therefore no infringement in zoomed-in mode either.
273. The finding of non-infringement seems to me simply to reflect the fact that the patent is not directed to full screen mode, and in zoomed-mode claims a specific way of reacting to the fact that the edge has been reached during a gesture. HTC's method in zoomed-in mode reacts to the fact that the edge has been reached in a significantly different way.

Validity

274. Although HTC originally relied on other citations, and pursued some of these at trial, Mr Meade ultimately relied only on one: PCT application WO 03/081458 ("Lira").
275. Lira is concerned with the problems of viewing web pages and other documents on a small screen. A user who wishes to view a full size web page on a small screen would have to scroll around the document in order to take in its entire contents. Lira teaches an approach in which the device detects the layout of the document, compares the layout to the width of the display, and re-formats it into columns having the width of its display. The user can then navigate within the columns, reading or viewing the contents, by moving the "viewport", i.e. the display window, down the columns and across the adjacent ones.
276. When scrolling vertically in a given column, it is possible that the user would inadvertently jump to the adjacent column. In order to prevent this happening, Lira provides a horizontal movement threshold. If the threshold is not exceeded the horizontal movement is ignored. If the threshold is exceeded the display window will move to the adjacent column.

Anticipation by Lira

277. Although HTC advanced an argument of anticipation by Lira, this was contingent on the court adopting Apple's construction of claim 1. As I have accepted HTC's construction, in which the claim is limited to zoomed-in mode, which is accepted not to be expressly taught by Lira, I do not propose to consider anticipation any further.

Obviousness over Lira

278. HTC's case of obviousness over Lira was run very much as a squeeze on the scope of the claims in support of their obviousness case. On the construction which I have adopted of the claims, it would be necessary for HTC to establish that it would be obvious to add a zoomed-in mode to Lira. I should say straight away that I was not persuaded by the evidence that this was so. Lira deals with allowing the user to view a large document on a small screen by breaking the content into columns. Although zooming of digital images was part of the common general knowledge, adding a zooming facility on top of Lira's column re-structuring is, as it seems to me, adding a layer of complexity to Lira which it nowhere suggests. If the idea, for some reason, were to occur to the skilled person, then he would have to consider how this zoom facility interacted with the horizontal movement threshold. HTC did not establish to my satisfaction that it was obvious how to do this, or that doing it would result in a method within the claims.
279. Accordingly, on the construction which I have arrived at, none of the claims is obvious over Lira.

Excluded subject matter

280. HTC submit that, as with 022, the invention provides no more than a way of informing the user that he has reached the edge of an image. I think there is more to the invention than that. As I have construed the patent it provides a novel method of manipulating a zoomed image involving gestures having different effects. I think that the method cannot properly be described as a computer program as such, or the presentation of information as such.

The 859 patent

281. 859 is entitled "Portable radio communication apparatus using different alphabets". It has a priority date of 19^th July 1994. It is important to bear this date in mind, as this date is much earlier than the other patents with which the action is concerned. The invention was made when mobile phones were much cruder and heavier than those with which we are familiar today.

Technical background

282. The Global System for Mobile Communications (GSM) allows the use of the standardised digital telecommunications network for sending and receiving text messages. The service is called the Short Message Service (SMS). At the priority date SMS was specified in the standards document GSM 03.40. SMS allowed the transmission of a message of a maximum of 160 characters over the network.
283. Text messages are familiar to almost everyone today, but at the priority date not all networks supported SMS. The SMS service was undoubtedly, if not embryonic, in its infancy. Phase 1 of the GSM standard (which included SMS) had been frozen in 1990. Voice was a far more important part of GSM than SMS. Some idea of the development of SMS can be gained from the fact that the first mobile phones capable only of receiving SMS (mobile terminated) were available in 1993. By 1994 every new mobile could receive SMS. 1995 marked the beginning of the widespread adoption of text messaging by young people. By 1996 every new mobile could send and receive SMS and sending SMS between networks was generally possible. In 2008 2-4 trillion SMS were sent.
284. Not every telephone with SMS capability at the priority date was therefore capable of sending (mobile originated) text messages.

The witnesses

285. On this patent, Apple called Dr Alastair Brydon. Dr Brydon is an electronic engineer who has worked in wireless communications for over 25 years. Between 1989 and 2001 he worked at BT Cellnet (now Telefonica O2) and Nokia. Since 2001 he has provided research and consultancy services in the mobile telecommunications industry. From 1989 he was deeply concerned with the development of standards for mobile network architecture, in particular contributing to the ETSI standards, chairing the technical co-ordination group developing aspects of UMTS. Dr Brydon was an entirely fair expert witness. Mr Alexander for HTC submitted that the subject matter of 859 was not squarely within Dr Brydon's expertise. There is some force in this submission, as Dr Brydon accepted that he was not, at least before 1995, directly concerned in the design of mobile handsets. Nevertheless, as he also pointed out, the issues in this case do involve network standards, such as the SMS standard, and how far SMS had got in the marketplace. These are issues on which Dr Brydon was well qualified.
286. HTC called Mr Jan Nottelmann. Mr Nottelman is an electronic engineer. Between 1988 and 1992 he was head of research and development at DC-Development, which was a joint venture company formed by the Nordic manufacturers of analogue phones, Dancall and Cetelco (later Hagenuk). The purpose of the joint venture was developing hardware and software for Hagenuk/Cetelco's and Dancall's first generation of digital GSM mobile phones. From 1992 Mr Nottelman moved to Hagenuk/Cetelco to become co-managing director and head of product strategy and development of the Hagenuk/Cetelco mobile handsets. A large part of his evidence is directed towards the functioning of the Hagenuk phone. Mr Burkill for Apple accepted that on the whole Mr Nottelman was a fair witness. He made some criticisms of particular parts of his evidence. I will deal with most of these in context to the extent that they matter. Mr Burkill pointed out that Mr Nottelman had said that he was not aware of one of the items of prior art, Arabic Tdoc, until the present case but later said "We certainly did look at the Arabic change request also at that time, or I believe the people who worked on it looked into it at the time." I have treated the latter answer as speculation, and have not relied on it. I do not think that this isolated answer casts doubt on the integrity of his evidence.

The skilled addressee

287. The patent is addressed to an engineer with an interest in the design of mobile telephones to support the services of the GSM network. He would be likely to have a first degree in electronic engineering or computer science and a few years experience in industry. The design of the user interface of the phone would be part of the responsibilities of such a person, but he or she would not have had any special training in the user interface design.

The common general knowledge

288. GSM Technical Standard 03.40 provided, in Annex 2, for a default set of characters, each coded with a 7 bit code. I reproduce it below, with shading to make it easier to explain:

289. The rows and columns labelled b7 to b1 in the shaded parts enable one to determine the code for each character in the unshaded part. b7 to b1 represent the 7 bits used to code each character. So for example, a capital J is coded by the binary number 1001010 (from b7, the most significant bit, to b1, the least significant bit).
290. The use of Annex 2 as the data coding scheme is determined by setting a value of a field TP-UD. This may have integer values in the range 0 to 255. Setting the value at 0 indicates that the alphabet is that given in Annex 2. Other values are said to be "reserved".
291. The characters in the Annex 2 table include the ordinary English letters, A-Z and a-z. In addition there are included letters with accents that are only used in certain languages, such as é, è and ô from the French alphabet and ö and ü from the German alphabet. Finally there are some capital letters specific to Greek.
292. It was compulsory for SMS handsets to support the Annex 2 alphabet so that they could receive text messages using any of these characters. But it is important to appreciate that Annex 2 is just a coding scheme, defining the protocol for the mobile to communicate with the network. The designer of the SMS user interface could decide how many of these characters were made available to the user for composing an SMS message.
293. By the priority date the skilled person would have been aware of the moves within the GSM community towards introduction of additional character sets. Arabic TDoc, discussed below, is a specific example of one such move. Whilst generally aware of these moves the skilled person would not have been aware of the detail of every proposal. He or she would also know that no final decisions had been reached.
294. Dr Brydon explained that the SMS capabilities of GSM phones at the priority date were illustrated by the Nokia 1011 which was launched in November 1992 and that the operation of such phones would have been familiar to the skilled person. For entering characters into a phone memory, the physical buttons on the keypad would be pressed a number of times. The 1011 had a fairly basic SMS capability, but it included the ability to write and send SMS messages. Although the phone had a language selection button, this related to the language of, for example, menu options. It did not affect the characters which would appear when writing SMS messages.
295. A specific question is whether the Hagenuk MT900 mobile phone, on which HTC rely for the purposes of their obviousness case, was part of the common general knowledge. The phone had a relatively small market share, less than 1% on Dr Brydon's unchallenged calculations. However it had been the subject of review, some of it commendatory in relation to its menu system, in an article in a consumer magazine, Connect. Engineers would attend annual trade fairs such as CeBIT in Hannover to increase their awareness of competitiors products and "play with them". Thus Cetelco would send "a coachload of engineers" to CeBIT. Moreover other manufacturers did appear to have followed hot on the heels of the Hagenuk with at least some of its novel features. Yet further, the number of different brands of mobile phones on the market was relatively small at the priority date, and an even smaller number of manufacturers were behind those brands.
296. The evidence persuaded me that the MT900 was sufficiently widely known by the priority date for the skilled person to be aware of its existence. The evidence showed that the designer of a user interface for a mobile phone would make it his business to keep up with what the competition were doing in that area. Although Dr Brydon thought that this might be done on a more local basis, I prefer Mr Nottelman's evidence on this issue, given his greater exposure to design of handsets. Beyond these rather general conclusions, I consider it is safer to consider in context whether a particular piece of information which could be gleaned from the MT900 or its manual was part of the common general knowledge.

The specification and claims

297. The specification of 859 begins by describing the SMS facility of GSM, as well as the construction of a contemporary mobile phone. It explains that text messages are composed by inputting one character at a time through a keypad on the phone while confirming it on the display. The keypad shown has groups of three Latin characters on each key, together with a number. So the numeral 1 appears with the letters ABC. At [0009] the specification explains that, since only the Latin alphabet is indicated on the keypad, inputting by other languages cannot be performed.
298. The objects of the invention are stated at [0011] to [0012] of the specification. These are in summary (1) to provide a phone in which a transmitting message may be formed, and (2) to provide a phone in which inputting in a language other than English is possible.
299. The specification then sets out to describe the invention by reference to three embodiments. In the first embodiment, described by reference to Figure 1 and 2, the user operates a language selection key and can then, by operation of the volume control, select one of a variety of languages from those displayed on the screen. He or she then confirms the selection by operation of the END key. The specification then explains that the control of the phone arranges for the section of memory corresponding to the selected language of the alphabet memory storing the multilingual alphabet to be used at the time of inputting by character. Thus, at [0022] the specification says that if Greek is selected as the language and numeral key one is pressed once, an "a" will appear instead of an "A".
300. In the second embodiment, described by reference to Figures 3 and 4, a desired language is selected by means of a language selection operation key and an alphabet symbol selection key. This causes the display of a character string corresponding to the selected language at the bottom of the screen. A desired character can be selected by operation of the volume control, and confirmed by pressing the END key.
301. In the third embodiment the language is selected based on the nationality derived from the user's SIM card.
302. Apple say that, in addition to claim 1, claims 2, 4, 6 and 7 are all, at least potentially, independently valid. So I set them out below with, in the case of claim 1, added numerals for identification:
303. Claim 1 provides:

(i) A portable radio communication apparatus comprising:

(ii) an antenna for transmitting and receiving a radio frequency message signal;

(iii) radio/modulator-demodulator means for demodulating a received radio frequency message signal by converting its frequency and for modulating a message signal to be transmitted to effect its frequency conversion into a radio frequency;

(iv) message memory means for storing messages which are received or to be transmitted;

(v) and display means for displaying the messages which are received or to be transmitted;

(vi) characterised by alphabet memory means for storing a multilingual alphabet comprising sections with alphabetical notation for each language;

(vii) selection means for selecting a section corresponding to a language of the multi-lingual alphabet to be used in forming a message to be transmitted at the time of inputting by character; and

(viii) control means for sequentially selecting characters from the section corresponding to a language of the multi-lingual alphabet and for forming and displaying messages which are to be transmitted.

304. Claim 2 provides:

2. The apparatus according to claim 1, wherein the alphabet memory means comprise a ROM for previously storing a plurality of linguistic alphabets; and wherein the control means are adapted to cause the display means to display the names of languages of the multi-lingual alphabet stored in the alphabet memory and select and confirm any of the plurality of charactersets by the selection means.

305. Claim 4 provides:

4. The apparatus according to claim 2 or 3 wherein the alphabet memory means comprise a ROM for previously storing a plurality of sections, and wherein the control means are adapted to form a message in the selected language while displaying on the display means the character set of the selected language name.

306. Claim 6 provides:

6. The apparatus according to any of claims 1 to 5, wherein the alphabet memory means comprise a ROM for previously storing a multi-lingual alphabet, and wherein the control means are adapted to input a home system information by the selection means and select and confirm any of the plurality of sections stored in the alphabet memory means in accordance with a nationality-based management table.

307. Claim 7 provides:

7. The apparatus according to any of claims 1 to 6, wherein said selection means comprise an IC card interface section capable of mounting a subscriber ID card; and wherein the control means have the following functions:

inputting the home system information from the subscriber ID card through the IC card interface section, selecting and confirming any of the sections stored in the alphabet memory means in accordance with a nationality information obtained from the home system information against the nationality-based management table, and displaying the selected language on the display means (7).

308. Claims 2 and 4 are relied on by Apple to reinforce an argument they run on construction. Claims 6 and 7 are directed to the third embodiment, in which the language is derived from the SIM card. Claim 7 is not alleged to be infringed.

Construction

309. The pre-characterising part of claim 1 is a rather long-winded way of calling for a mobile telephone with two-way SMS capability. There was no dispute about the meaning of any of the terms used in that part. The principal dispute on construction concerns the meaning of "alphabet memory means for storing a multi-lingual alphabet comprising sections with alphabetical notation for each language" in integer (vi). The remainder of the claim just means that the user can select which alphabet he or she wants to use and the phone will allow him to select the right characters for composing an SMS message in that language.
310. Apple submit that the patent, in requiring a multilingual alphabet with sections with alphabetical notation for each language, requires a plurality of alphabets. The requirement is not satisfied, for example, if the group of characters for, say, English, is supplemented by the additional characters needed for French or German. The memory has to store, for each language, the complete set of characters for that language. To put it another way, overlap is not permitted.
311. It is not part of Apple's case that the sections of memory containing the individual alphabets are arranged in any particular spatial relationship. The skilled person would understand that functional separation was what was required. Nevertheless, the memory, on Apple's construction, has to contain the letter A for English, A for German and A for French. Apple draw particular attention to [0022] which refers to Greek.
312. HTC contend that the patent is simply not concerned with implementation at this level of detail. There is no reason why the skilled person would consider that the patentee was requiring him to duplicate the common letters in memory. The nature and object of the invention and the specific embodiments were all described at such a high level that the skilled person would not extract Apple's intended meaning from the claim.
313. Apple advanced linguistic arguments. Thus Apple drew attention to the word "each" in the integer itself and "a language" in integer (vii). Apple also point out that claim 2 calls for "a plurality of linguistic alphabets" and "a plurality of charactersets". They also sought to extract something from other passages in the specification, the Figures and even the title of the specification itself. I did not find any of this to be convincing.
314. In my judgment claims 1, 2 and 4 are all drafted at a much higher level of generality. I think one can be led astray by too much lawyerly linguistic analysis, particularly if it is performed on feature (vi) in isolation from the rest of the claim. The patent is concerned with giving the user the ability to select a language. When a particular language is selected, the alphabet for that language will come from the section of the multilingual alphabet with the characters for that language. There is no requirement for, and the skilled person would not understand from the patent that there was any point in, a separate and distinct set of characters for each language.

Infringement

315. The HTC devices accused of infringement of 859 are described in a product and process description. They are the Evo 3D, Sensation, Desire, Desire HD, Desire S, Desire Z, Wildfire S and Incredible S. The case in relation to a ninth device, the Flyer tablet, was not pursued.
316. Although HTC mentioned in their closing skeleton argument a number of tu quoque arguments on the interpretation of other features of the claims in the event that I adopted Apple's preferred narrow construction of feature (vi) claim 1, its evidence was notably silent on the issue of infringement even if feature (vi) was construed narrowly. As I have not adopted Apple's narrow construction, I am grateful to be relieved of the responsibility of considering the other arguments on infringement any further. All the devices fall within the scope of the 859 patent on the broad construction which I have adopted.

Validity

317. HTC contend that the invention as claimed in each of the claims relied on is invalid for obviousness from each of three starting points: GSM Technical Standard 03.40, Arabic TDoc and the Hagenuk MT900. I have adequately dealt with the inventive concept, the skilled person and the common general knowledge above.

Obviousness over GSM TS 03.40

318. The GSM standard defines how a mobile handset communicates with the network so as to provide an SMS service. It does not go any further and lay down guidance on the design of the handset.
319. The skilled person would understand from TS 03.40 that it provided a mechanism for adding further alphabets to the coding system, which could be provided by reference to a further value or values for the TP-UD data field. Dr Brydon accepted this, whilst pointing out correctly that this was only in terms of the coding system for communicating with the network. It was suggested to Mr Nottelmann that the reserved values might have other uses, but he considered it unlikely.
320. Particularly in view of this last point, I do not think it is helpful to consider the obviousness case over TS 03.40 independently of that over Arabic TDoc. It is common ground that the reader of the latter would be aware of and combine its teaching with the former. Arabic TDoc involves the use of the data field to allow for a second, separate, coding alphabet. Moreover, if HTC cannot succeed on Arabic TDoc, it is unlikely that they could succeed on TS 03.40 on its own.

Obviousness over Arabic TDoc

321. Arabic TDoc is what is known as a "change request" in GSM. The change request was made because the standards group concerned, known as MoU, had expressed interest in the enhancement of the SMS service to cater for coding of messages using other alphabets than the default alphabet in TS 03.40 and TS 03.41. The document contains a proposal from Modarabtel, a Tunisian telecommunications company, for the inclusion of the Arabic alphabet.
322. The proposal suggests that the Data Coding Scheme parameter should be extended to be able to refer to an Arabic Data Coding Scheme. Annex X to the proposal is in the same format as annex 2 to TS 03.40, except that the non-shaded portion consists of mainly Arabic characters:

323. The change request is applicable also to TSM 03.41, which is a cell broadcast specification. The characters marked 1) are reserved for expansion. Note 2 specifies that the characters of the set, when displayed, should approximate to the appearance of the relevant characters specified in certain international and national standards. Note 3 explains that Arabic characters, like joined up writing, have a different shape according to their position in a word and whether they are in isolated form. In the coding scheme the isolated form is used, but the note recommends that devices supporting the Arabic character set should be able to provide context analysis in order to display the letters correctly according to their position in the word. Note 4 points out that the device should be bi-directional in order properly to display Arabic right to left writing.
324. HTC's case of obviousness is that when the skilled person sees that it is proposed to have a separate alphabet coding table for the Arabic language for send and receive over the radio interface, it would be obvious to allow the user to switch between these languages when composing messages as well.
325. Apple's answer is that nothing in Arabic TDoc proposes or requires the user to be able to send (as opposed to receive) in Arabic at all. Moreover the document does not require provision for switching between languages when composing and sending.
326. I think the first of these submissions, although technically accurate, does not represent the reaction of the skilled person to the document as established by the evidence. In paragraph 190 of his first report Dr Brydon had said this:

"190. In particular, it would not be obvious in 1994 that a user could change the character set used to construct messages on a mobile telephone by selecting a different language setting. The approach adopted at that time by GSM telephones, such as the Nokia 1011 (see paragraph 57), was to cycle through a fixed set of characters (drawn from a variety of languages) as the user made successive presses of a given key on the keypad. A comparison of Figure 5 (paragraph 57) with Figure 8 (paragraph 64), above, shows that there was no relationship between the mapping of characters to keys in a GSM telephone and the coding of characters in the default SMS character set and, indeed, not all of the SMS character set was available. An obvious implementation of a mobile telephone in the light of the Arabic TDoc would be simply to extend the mapping of key presses to characters, for example, adding the Arabic characters to the end of the list of characters associated with each key in Figure 5. A more attractive option for Arabic markets would be to have handsets that followed exactly the same tried and tested principle, but had Arabic markings on the keypad and presented the Arabic letters with the first few presses of a key, followed by Latin letters (if required at all) for subsequent key presses."

327. Whilst in that passage Dr Brydon maintained that it was not obvious to implement a different language setting, he did propose two ways of implementing writing in Arabic which he said which were obvious in the light of TDoc.
328. Dr Brydon also accepted that, when looking at the new character set in Arabic TDoc, that there were a lot more characters involved by which I took him to mean a lot more than would naturally have been made available to a user from the Annex 2 character set. That would make the scrolling solutions he had suggested, which were no more than an extension of the existing scrolling system implemented on Nokia phones, "unwieldy". He said that this did not necessarily lead the skilled person to consider a language switchable phone. He later accepted that if the character sets were split by language, there would be some logic in using those different character sets for input. When pressed as to why it was not entirely natural to follow this logic, he pointed to the fact that it had not been done before.
329. Mr Nottelmann was of the view that Arabic TDoc did make it obvious to make a language switchable phone having an Arabic mode and a mode for Western European languages. He was cross-examined on the basis that it was premature to be thinking about implementing such a phone, given that TDoc was only a proposal. He was inclined to accept that TDoc was not of immediate interest. However his evidence that it would be technically obvious to implement the dual mode phone which he proposed was not, in my judgment, seriously undermined.
330. Ultimately Dr Brydon did accept that if a decision had been taken to allow input in both Western and Arabic characters, it would be obvious to have separate language modes selectable by the user following Mr Nottelamn's approach.
331. In the end I have come to the conclusion that the evidence shows that the invention of claims 1, 2 and 4 was obvious in the light of Arabic TDoc. The skilled person would appreciate from reading the notes in the document that what was contemplated expressly included the ability to receive messages in both Arabic and Western characters. If he had any doubts about the matter, the reference to the ability for the display to operate bi-directionally would dispel them. This would, in my judgment, immediately prompt the skilled person to consider arranging the device so that messages could be composed using both character sets. Dr Brydon's paragraph 190 accepts that this is so. His oral evidence recognised that the suggestion of scrolling through large numbers of letters to reach the Arabic would be unwieldy. Given that the mode of the phone would have to be altered for Arabic input anyway (for right to left display), the concession that it would be obvious to make the Arabic mode one in which the annex X character set is used and the Western mode one in which the Western character set is used, was more or less inevitable.
332. Claims 6 and 7 add the feature of automatic language selection from the user's home SIM card. HTC contend that language selection from a SIM card, at least for text entry into a phone book or the like, was part of the common general knowledge and that it required no invention to apply this idea to language selection for SMS.
333. An automatic language selection feature from the SIM card was present on the Hagenuk MT 900. Mr Nottelmann said that the feature was common general knowledge, but despite trying, he had not been able to identify any other phone available at the priority date which shared this feature.
334. Dr Brydon's evidence was that the country-specific information on the SIM card was included for the purposes of the network. Using it for the internal purposes of the phone was not generally done and he was not aware of any examples of such use, other than the MT900.
335. It is therefore critical to establish whether the automatic language selection feature of the MT 900 was part of the common general knowledge. There can be no dispute that the feature was known, in the sense of made available to the public: but as the authorities make clear, that is not enough. Moreover the fact that the skilled person would have been aware of the MT900 is not the same thing as knowing of the automatic language selection feature, which would require that the skilled person either read the manual, or examined a physical phone. Common general knowledge is of course not confined to that which the skilled person keeps in his head, and the notion extends to information which the skilled person would know exists and know where to find: see Laddie J in Raychem Corporation's Patents [1998] RPC 31 at 40.
336. I do not think it was established that the automatic language selection feature of the MT900 formed part of the common general knowledge. It is something which the skilled person could have found out about if sufficiently interested in the MT900. But in my judgment it is not a piece of information which it would be right to assume that the skilled person knows exists or would know where to find.
337. Nevertheless, as Kitchin J pointed out in Generics (UK) v Daiichi Pharmaceutical [2008] EWHC 2413 (Pat) [2009] RPC 4 at 40, it may be the case that someone faced with a particular piece of prior art would find it obvious to seek out further information. That does not make the information common general knowledge. In the present case, the evidence that it was common practice for manufacturers to follow relevant developments on the mobile phones of others, whilst not making every feature of a rival's phone part of the common general knowledge, may mean that no invention is involved in learning about and incorporating a feature in a design which is being considered.
338. I accept that there are real dangers of hindsight creeping in when an obviousness argument involves a second or subsequent step. However I have in the end concluded that if one accepts, as I have, that the provision of a language selection means for composing text messages was obvious, then I cannot see why it is inventive for the skilled person to conduct a review of text entry systems available on the market and consider adopting any relevant features of those systems. If the skilled person conducted such an obvious review, he or she would immediately see the application of the language selection feature in the MT900. I therefore conclude that claims 6 and 7 are obvious in the light of Arabic TDoc.

Obviousness over the Hagenuk MT900

339. The Hagenuk MT 900 phone was launched in 1992. Although the phone is pleaded as a prior use, there is no dispute that the use should be considered in conjunction with its user manual. Thus the disclosure on which it is legitimate for HTC's purposes to rely is limited to what the skilled person would be able to glean and write down from examining the phone in use, in combination with the manual: see Lux Traffic Controls v Pike Signals [1993] RPC 107 at 134-6.
340. The MT900 had a small dot matrix display. It had two soft keys positioned underneath the display and dedicated "BOOK", "MENU" and "EXIT" keys, together with a numeric keypad. When the user pressed the MENU key, the main menu was shown in the display. The user could then scroll through the menus and sub-menus using certain numeric keys, and select items using another numeric key. One of the main menu items was "LANGUAGE". When this was selected the menu would look like this:

341. If AUTOMATIC was selected, the operation menu language would be based on network operator information read from the SIM card. Alternatively the user could select his language manually. The available languages were English, German, French, Spanish, Portuguese, Italian, Danish and Swedish. When a particular language was selected, whether automatically or manually, all operational menu items were displayed in that language. The selection of a language, whether automatically or manually, also determined the alphabet available to the user when entering and editing text in order to store names in the PHONE BOOK.
342. A user could enter a name in the PHONEBOOK using the Text Editor. As shown below, the available characters were displayed in a horizontal line across the top of the MT900's display. An arrow indicated the character for selection. The example below demonstrates the character "R" being indicated for selection. The user could point the arrow to different characters for selection by using the relevant numeric keys to scroll left or right, respectively, and then select it by pressing another numeric key. The selected character is then displayed in the lower part of the display. In the example the name MILLER has been entered, the arrow indicating the most recently selected character, R.

343. The available characters when English was the selected language were shown by this table:

344. The device operated in the same way if German was the selected language, except that now the phone gives some special characters used in the German language, ä, ö and ü, both in lower and (by pressing the right soft key) upper case:

345. The special characters shown by other languages are shown in the following table:

346. All of the above would be readily apparent to the skilled person on examination of the MT 900 and, if necessary, its manual. The differences between the MT900 and the inventive concept of claim 1 turn on construction:

i) On HTC's construction of feature (vi), which I have held to be correct, the only difference between the disclosure of the MT900 and the inventive concept of claim 1 is that the MT900 did not have SMS capability. In particular Dr Brydon accepted, correctly in my judgment, that the text entry method of the MT900 was the same as that described for the second embodiment in the specification of the 859 patent.

ii) On Apple's construction of claim 1 there is a further difference. It is not known how the MT900 stored its alphabets, and accordingly there is no disclosure of storing it in the way Apple maintains the claim requires, that is to say with a different address for common letters in each language. The experts agreed that it was likely that it was done in the more efficient way, that is to say with overlap of the common letters.

347. HTC's case of obviousness starts with the proposition that it was entirely natural at the priority date to be considering the provision of two-way SMS messaging capability on any mobile phone. I think this is an irresistible conclusion. The skilled person would have been aware of the fact that Nokia had implemented this feature. Given that by 1996 all new mobile phones had this capability, many if not all manufacturers must have been working on this facility before the priority date.
348. HTC go on to submit that it would be obvious to implement SMS on the MT900, and in doing so to keep the same multilingual text-entry system as is used for making entries in the phone book. This was Mr Nottelman's view of the preferred way forward, as it maintained consistency over the user interface. Dr Brydon was prepared to accept that, if the skilled person decided to implement SMS on a phone, one of the logical options was to use the existing text entry system. He considered the MT900 system to be a little cumbersome or unwieldy, whilst at the same time accepting that it was the method suggested in the second embodiment in the patent.
349. In my judgment, one of the obvious ways of approaching text entry on the MT900 if adapted for SMS would be to retain the existing method of character and language selection. Some, albeit secondary, support for this conclusion is to be found in the fact that this is what happened when the next generation of Hagenuk phones emerged. I would have reached the same conclusion even in the absence of that secondary evidence.
350. Accordingly the inventive concept of claims 1, 2 and 4 is rendered obvious by the MT900 on the construction which I have arrived at. Claims 6 and 7 are also invalid, as these features are present in the MT900 language selection method.
351. I turn briefly, therefore, to consider whether the patent is saved by Apple's construction, which requires the alphabets to be divided into sections. As I have said, the experts agreed that the likely arrangement in the MT900 was that the alphabets were not stored separately. This is what the skilled person is likely to have assumed. Dr Brydon agreed however that one of the obvious ways in which it would be apparent one could proceed would be to store the character set for each language in a separate file. The points he made were essentially three. Firstly, the skilled person might be deterred from adopting that route as it is inefficient in memory terms. Secondly, this was not the way it seems to have been done in the MT900. Thirdly, that doing it in the "patented" way brings benefits.
352. As to the first of these points, whilst perceived inefficiency might deter a skilled person from actually adopting the patented method, I would not consider a patent whose only contribution was inefficiency to involve an inventive step. On this aspect Dr Brydon's cross-examination went like this:

Q. What I want to put to you is that it would have been obvious, that one way of doing it which would have been immediately apparent would have been to provide a different characterset for each of the different languages, even though many of the characters were common?

A. Yes, it would.

Q. It would equally have been obvious that although one could have done it that way, and the facility existed to do it that way, that would be inefficient because you would have duplication and so you would need a large number of charactersets.

A. Yes, it would.

Q. It is one of those situations, it is a little bit like it being obvious to walk to Newcastle, it is obvious to do so but everyone knows it is better to take the train?

A. Yes.

353. Viewed in that light, I do not think that the second point adds anything. It is not surprising that the MT900 adopted the more efficient way of doing things. Were those the only two points made, I would conclude that the patent does not involve an inventive step. The patent system does not exist to allow people to monopolise products and processes simply because they are less advantageous than known ones.
354. It is therefore necessary to examine with some care the third of Dr Brydon's points which concerns benefits which the patent brings. At paragraph 153 of his first report he drew attention to the fact that the approach in the patent brings with it flexibility, including the ability to remap the keys to completely different alphabets such as Greek and Cyrillic. It thus allows a complete different language with different characters to be "slotted in".
355. Dr Brydon accepted that these benefits were not spelled out in the patent, but maintained that "the patent used examples such as Greek and Russian to illustrate what it is getting at".
356. The evidence showed that as soon as one contemplated the addition of a character set which had no overlap with the western languages used in the MT900, such as Arabic or Russian, it would be inevitable that one would have to expand the memory in such a way as to accommodate these characters. This would, in my judgment, create a separate section in the Apple sense. The question is whether this is something which would have occurred to the skilled person in 1994.
357. In my judgment, although the issue was not a pressing one in 1994, the skilled person who knew of the modes of operation of the MT900 would see without difficulty that it could be expanded to character sets such as Russian or Arabic if the need arose. It follows that the claims are all invalid in the light of the MT900 on Apple's construction as well.

Excluded subject matter

358. I have construed the claims above. HTC submit that the contribution lies in the particular way in which the characters have to be stored in memory in order to fall within the claims. This brings with it a benefit in terms of flexibility in adding languages. They further submit that a particular arrangement of data storage in memory falls squarely within excluded subject matter. Moreover the contribution has no real world effect - Dr Brydon confirmed that the user experience is unaffected by whether or not the characters for a particular language use shared common characters or have their own separate set of characters.
359. I think these submissions understate the contribution of 859. Hagenuk does not disclose an SMS messaging capability. That produces an effect outside the computer, and is enough to take the invention outside the exclusion. 859 is not invalid on this ground.

Time estimates

360. At one point the present trial was estimated at 6-8 days. This was, or became, a seriously inaccurate estimate on any basis. There were four patents in issue, all of obvious commercial importance to both parties. HTC, for their part, were relying on several documentary prior art citations per patent, as well as attacks based on prior use and the common general knowledge alone. The technology involved was not all entirely straightforward. Apple, for their part, were relying on independent validity of multiple sub-claims. Both parties filed voluminous expert evidence, generally three reports from each of three pairs of independent experts. The parties wished to cross-examine the opposing experts on this extensive material. The parties also estimated that the court could get on top of this material in two days, later collapsed to a day and a half because of a need for an expert to return to the United States. This time estimate was also completely unrealistic. A longer time estimate for reading does not cost the parties anything. In the result the trial had to be interrupted to allow me more time to read and understand this material. This is highly disruptive.
361. The court will always be sympathetic to attempts by parties to resolve patent disputes with strict limits as to the number of citations and claims, the evidence which may be adduced, and the time which is to be taken in court with cross examination and speeches. Very careful consideration needs to be given to match reading time estimates and trial estimates to the way in which the case is in fact being conducted.
362. Finally, this is a case where there should plainly have been a pre-trial review in accordance with the guidance in the Chancery Guide. The guidance in paragraph 3.20 is only mandatory in the case of cases lasting more than 10 days, but applies in other cases where the circumstances warrant it. The parties should have appreciated that the present case would last 10 days or more. Moreover, and in any event, the circumstances of the present case plainly warranted a pre-trial review.

Overall conclusions

363. My principal conclusions are as follows:

i) 948

a) 948 is not infringed by the HTC devices;

b) Claim 1 (but not claim 2) of 948 is invalid for obviousness over common general knowledge;

c) Claims 1 and 2 are invalid for excluded subject matter.

ii) 022

a) Claims 1, 6 and 18 of 022 are infringed by the Arc mechanism, but 022 is not infringed by the other unlock mechanisms;

b) Claims 1, 6, 9 and 18 of 022 are anticipated by Hyppönen;

c) Claims 1 and 9 (but not claim 5) of 022 are obvious in the light of Plaisant;

d) All the claims of 022 are obvious in the light of Neonode;

e) 022 is not invalid for excluded subject matter.

iii) 868

a) 868 is not infringed by the HTC devices;

b) 868 is valid over Lira;

c) 868 is not invalid for excluded subject matter.

iv) 859

a) All the claims of 859 are invalid over Arabic TDoc;

b) All the claims of 859 are invalid over the Hagenuk MT900;

c) If 859 had been valid it would have been infringed by the HTC devices;

d) 859 is not invalid for excluded subject matter.

364. I will hear counsel on the details of the order to be drawn up to reflect these findings.
Appendix 1 Summary of common general knowledge of software creation and event driven programming
365. Typical software for a computer is structured in layers, each of which consists of multiple modules. Each layer provides abstractions that higher layers may build upon. The lowest software layer is the operating system (OS). The OS is allowed to directly manipulate hardware e.g. to read external input and produce external output through input/output hardware devices, such as the display, the touch screen, the network interface, etc.. Device drivers comprise the bottom layer of the OS, closest to the hardware; it is these routines that directly read and modify the hardware's state.
366. A "user interface toolkit" ("UI toolkit") is a piece of software that is usually built atop the system software and provides a set of user interface elements ("UI elements"), such as the adornments of windows in which application software runs, buttons, checkboxes, and scroll bars, which application software developers may assemble to create their application software. This assembly is accomplished using the UI toolkit's "application programming interface" ("API"), which application software may use to command the UI toolkit. The application developer must learn these APIs (typically by reading documentation and possibly by reading example code that uses them); he or she benefits by being able to reuse functionality previously written by others.
367. Finally, above these layers sit the applications, which accomplish tasks on behalf of a user (e.g., an editor, a web browser, or an email reading/sending application). All layers above the operating system may invoke system calls provided by the OS and receive input and event notifications from the OS. Applications may invoke library routines in the UI toolkit.
368. A good API should be flexible, in that it should allow the programmer to implement varied functionality. It should also be simple, in that it should offer economy of mechanism, so as not to be onerous or error-prone for the programmer to use.
369. The overall software architecture described above is common to both desktop class computers and portable electronic devices: both types of system typically feature these three layers of software (and their subdivisions).
370. Input to computer software may arrive through a variety of hardware devices, e.g., through a network interface, a mouse, a mechanical keyboard, touch screen, etc. The overall flow of input processing through computer software begins in the OS, where a device driver is notified by the hardware that input has occurred. The device driver may process the raw, low-level input from the hardware into a form that is easier for an application to process. The OS then passes this processed input up out of the operating system to either a run-time library in the UI toolkit or directly to the application (depending on how the application is implemented). The run-time library may further convert the input into a form even easier for an application to process.
371. In mouse-based input, when the user moves the mouse, the hardware reports movement events, typically as distances in x- and y-coordinate spaces. The device driver for the mouse is notified of these events by the hardware. The device driver then constructs one or more data structures describing the change to the mouse's position, and typically passes the data structure(s) to a run-time library responsible for handling mouse input. The device driver and OS may translate movement (or button press or release) reports made by the mouse hardware in a mouse manufacturer-specific format into a hardware-neutral format; in this way, the device driver insulates run-time libraries and applications from differences in mouse hardware across manufacturers. The run-time library then makes the description of the mouse input events available to the relevant application. For example, an application may be notified that the mouse pointer has come within a portion of the display that it controls. Button presses ("clicks") are processed according to a similar data flow; many OSs pass separate events describing the "button down" and "button up" (released) events to the run-time libraries.
372. Touch screen input is processed broadly similarly to mouse-based input, with a few noteworthy differences. One significant difference between the two is that a touch screen allows a user to place a finger directly at a location on a display, whereas a mouse only allows the user to move the mouse continuously from a prior location. In this sense, a newly made touch inherently indicates input at absolute coordinates, whereas movement of a mouse inherently indicates input through relative coordinates vs. the previous position of the mouse. Mouse-based systems typically only allow the user to manipulate a single location (pointer or cursor) on the display (putting aside multi-mouse or multi-input device systems, which have not seen broad adoption).
373. Early touch-screen systems shared this single-pointer constraint—they recognised only a single point of touch at a time. Given that a user may attempt to touch more than one location on a touch screen at the same time, some later touch-screen systems support the recognition of inputs at multiple sites on the display concurrently. The low-level data structures that represent mouse and touch inputs may differ in form as a consequence—with data structures describing the latter incurring additional complexity, as they may aggregate information about multiple touches into a single data structure (though they need not do so). Simpler alternatives (though still more complex than single-pointer, mouse-based systems) would be to ignore subsequent touches until a first touch is released, or to treat the first touch as having been released immediately prior to handling a second touch.
374. An "event" is a signal which describes an occurrence within the computer. For present purposes it may be the click of a mouse or a touch on a touch screen display. These are input events.
375. A graphical user interface (GUI) consists of software, including applications and run-time libraries, which manage the content of the display and process mouse or touch-screen input. The GUI allows users to interact with a portable electronic device or desktop computer. It further allows applications to invoke GUI run-time libraries to display graphical content and text, and to be notified of user input. Users can interact with a GUI by manipulating displayed content with a mouse or touch screen: e.g., they can select menu items to invoke operations in an application, click on buttons or other displayed controls represented graphically on the display, and select data in the application on which to perform an operation.