Putting ChatGPT to the Patent Analysis Test

ChatGPT, the recently launched Artificial Intelligence (AI) chatbot, has taken the Internet by storm. Based on OpenAI’s generative pre-trained transformer (GPT) technology, it uses both supervised learning and reinforcement learning to handle simple questions and complex tasks. ChatGPT has even passed some of the world’s toughest exams, such as the US Medical Licensing Exam and Wharton MBA Exam. Although revolutionary, the technology opens questions for intellectual property (IP) professionals around how useful it can be for patent research and analysis. Our in-house experts put it to test to see how well ChatGPT could conduct patent research and discovered that the technology: 

• Does not provide relevant patent search results 
• Provides product search results that are too broad 
• Provides incorrect bibliographic information 
• Lacks the capability to correctly interpret a patent and make logical inferences 
• Collects all input data and information, compromising its confidentiality 

AI chatbot technology is in its infancy. Even though it can pass rigorous standardized exams, it needs to mature before it can be used to make patent or IP portfolio decisions—if ever. For this review, we tested ChatGPT around three foundational patent workflows: 

• Prior Art Searching 
• Product Search 
• Patent Drafting 

Prior Art Searching:  

Searching for prior arts is a critical aspect of a patent application, used to ascertain the novelty of an invention during the prosecution stage and validating/invalidating a patent during the litigation stage. To see how ChatGPT handles prior art queries, we started with simple patent search questions.  

Input: Can you identify patents disclosing geographic map with 3D buildings?

Output: Yes, you can search for patents disclosing geographic maps with 3D buildings by using patent databases such as the US Patent and Trademark Office (USPTO) or the European Patent Office (EPO). You can use keywords such as “geographic map,” “3D building,” “virtual map,” “3D mapping,” etc. to search for relevant patents. Additionally, you can also use advanced search techniques such as Boolean operators and field-specific search terms to narrow down your search results.

Input: Can you provide any US patents on this topic?

Output: Here are a few examples of US patents related to geographic maps with 3D buildings:
US Patent No. 8,948,966: “Method and Apparatus for Creating and Displaying Three-Dimensional Maps”
US Patent No. 9,344,711: “System and Method for Providing Virtual Tours of Real-World Locations”
US Patent No. 9,917,963: “Systems and Methods for Displaying Three-Dimensional Models of Structures in an Interactive Map”
US Patent No. 10,087,341: “Method and System for Displaying a Three-Dimensional Map of a Real-World Environment”
Note: The above patents are only examples and should not be taken as legal advice. It is always recommended to perform a comprehensive search and consult with a patent attorney before making any decisions related to intellectual property.

Input: What are relevant patent classifications around this topic?

Output: Patent classifications are used to categorize and organize patents based on their technological area. Relevant patent classifications for geographic maps with 3D buildings include:
International Patent Classification (IPC):
G01C 21/00: Representation of geographical maps or plans
G06T 7/00: Data processing or generation, in general, of three-dimensional representations, e.g. for simulation purposes
United States Patent Classification (USPC):
715/700: Data processing: presentation processing of document, operator interface processing, and screen saver display processing
705/38: Computer database management or file management
705/14: Presentation of data
705/17: Presentation of data (e.g., by graphic representation)
705/27: Representing three-dimensional objects
Note: Patent classifications may vary across different countries and can change over time, so it is recommended to consult the most up-to-date version of the relevant classification system for the latest information.

Input: Can you identify non-patent literature disclosing geographic map with 3D buildings?

Output: Yes, there are many non-patent literature sources that discuss and demonstrate the use of geographic maps with 3D buildings. Here are a few examples:

1.“Google Earth and 3D Buildings: Experiences and Possibilities” by A. Székely and J. Kavasidis (2010) – This article describes the use of Google Earth for visualizing 3D buildings and provides examples of how this technology can be used in urban planning and architecture.
2.“3D Buildings in Google Maps” by Google (2020) – This web page provides information on how to view and interact with 3D buildings in Google Maps, as well as how to create and contribute to 3D models.
3.“3D Buildings Visualization in Online Maps” by E. Sintas and J. Lopez (2014) – This paper discusses various approaches to visualizing 3D buildings in online maps, including the use of photogrammetry and laser scanning data.
4.“The Use of 3D Building Models for Urban Planning and Management” by H. Ledoux et al. (2012) – This article provides an overview of the use of 3D building models for urban planning and management, including the use of geographic information systems (GIS) and virtual reality tools.

I hope these sources are helpful to you!

Findings:

• ChatGPT did not perform well on critical workflows of Prior Art Searching, serving inaccurate information that mislead and prolong the searching process.
• ChatGPT should only be used for an initial understanding of a technology area, determining relevant keywords or synonyms, and identifying assignees around the target claim or feature.
• At this stage, ChatGPT does not look promising to find relevant prior art. It does not respond well to complex patent search queries, making it useless for this process. Human expertise and guidance are still needed to formulate the complex and focused search queries required to find relevant prior art.
• There are accuracy challenges with the prior art cited by ChatGPT. In the examples above, the tool provides patent numbers that are irrelevant to the domain in question. Moreover, the tool gives incorrect bibliographic details for the patent number of interest. This misleads the prior art research, wasting precious time rather than speeding up the process.
• ChatGPT reports the incorrect patent classification codes for a given technology. And it should be noted that patent classification codes are publicly available.

Product Searching:

Product searching is essential for assessing the market relevance of a patent or an allowed application. This includes a patent portfolio analysis where the scope of the study helps identify high value and low value assets. Here’s how ChatGPT performed when conducting product searching:

Input: Provide a summary of the following patent claim:” A method, comprising:
creating a composite manifest that identifies a plurality of service groups that are components of a composite application, each of the service groups associated with one or more policies, the composite manifest comprising: (i) a service manifest including service metadata, wherein the service metadata includes service type, health properties, pertinent load balancing metrics, and service configuration files, (ii) an application manifest including service composition metadata, wherein the service composition metadata includes security and isolation policy, placement constraints, configuration overrides, service types of constituent services, and load balancing domains into which the constituent services are placed, and (iii) a cluster manifest including physical infrastructure metadata, wherein the physical infrastructure metadata includes machine names, machine types and description, network topology;
identifying the constituent services based upon the application manifest;
deploying the identified constituent services on a plurality of nodes.”

Output: The patent claim describes a method for creating a composite application by identifying service groups and their associated policies using a composite manifest that includes a service manifest, application manifest, and cluster manifest. The identified constituent services are then deployed on multiple nodes based on the application manifest. The composite manifest contains detailed information about the service metadata, service composition metadata, and physical infrastructure metadata.

Input: Do you know any companies using similar technology in cloud domain?

Output: There are many companies that are using similar technologies to the patent claim in the cloud domain. Some examples of companies that offer cloud-based solutions for deploying and managing composite applications with multiple service groups include:

• Amazon Web Services (AWS): AWS offers several services that allow for the deployment and management of composite applications, such as AWS Elastic Beanstalk, AWS App Runner, and AWS CloudFormation.
• Microsoft Azure: Azure provides various services for deploying and managing composite applications, such as Azure App Service, Azure Kubernetes Service (AKS), and Azure Resource Manager.
• Google Cloud Platform (GCP): GCP provides several services for deploying and managing composite applications, such as Google Kubernetes Engine (GKE), Google App Engine, and Cloud Deployment Manager.

These are just a few examples, but there are many other cloud providers and companies.

Input: Please provide details of AWS products that are implementing similar technology.

Output: AWS offers several products that implement similar technology to the patent claim you provided. These products include:

• AWS Elastic Beanstalk: This service allows you to quickly deploy and manage web applications on the AWS cloud. Elastic Beanstalk supports a variety of popular programming languages and web application frameworks, and provides features such as auto-scaling, load balancing, and automated deployment.
• AWS App Runner: This service allows you to easily build, deploy, and scale containerized web applications on the AWS cloud. App Runner automatically builds and deploys your container images and provides features such as auto-scaling and load balancing.
• AWS CloudFormation: This service allows you to define and provision AWS resources as a single, declarative template. With CloudFormation, you can define a composite application as a set of AWS resources, including EC2 instances, load balancers, and databases, and then deploy and manage the composite application as a single unit.
• AWS Service Catalog: This service allows you to create and manage catalogs of IT services that are approved for use in your organization. You can define composite applications as collections of AWS resources, and then make them available to your organization through a self-service portal.

These are just a few examples of AWS products that implement similar technology to the patent claim you provided. AWS offers many other services that can be used to deploy and manage composite applications, depending on your specific use case and requirements.•

Input: Create a two-column mapping of the patent claim against AWS CloudFormation and include hyperlinks of the product as references.

Input: Which Google Maps feature displays geographical map along with embedded images?

Output: The feature you are referring to is called “Google Street View”. It allows users to explore places in a 360-degree panoramic view, which includes embedded images of the surrounding area.Input: Which Google Maps feature displays geographical map along with embedded images?

Input: Don’t you think it displays a complete street view rather than a geographical map?

Output: You are correct, my apologies for the confusion. Google Street View is not a geographical map, but rather a feature that allows users to explore and navigate through panoramic views of locations. It provides a more immersive and detailed experience than a traditional geographical map, which typically shows only the layout and features of an area without photographic detail.

Input: Do you think ground overlay feature of Google Maps does the same?

Output 3: Yes, the “Ground Overlay” feature in Google Maps can be used to embed an image onto a geographical map. This feature allows users to place an image over a specific area on the map, and it will adjust in size and position as the map is zoomed in or out. However, it is important to note that this feature is intended for use by developers who are creating custom Google Maps applications, and it requires some coding knowledge to implement. It is not a feature that is available to general Google Maps users.

Findings:

• Based on the outputs, in its current state the tool cannot be used to identify relevant products for a claim.
• When tasked with summarizing a claim, the tool simply paraphrases the claim language. To be done correctly, product search still requires a subject matter expert to interpret a patent claim effectively and identify the scope of different claim elements.
• The answers provided to a product search query are too broad (such as AWS) and need to be narrowed down with follow-up searches by a human. Someone has to be constantly guiding ChatGPT towards what is actually required as output, defeating the purpose of using AI to gain efficiency.  

Patent Drafting:

There are many ways to test ChatGPT for drafting patent applications, including asking it to generate a description and claims of an invention based on the information provided by the user. The drafting quality would depend greatly on the amount of information shared with tool, including technical field of the problem, inventive steps, etc. However, one of the most alarming concerns with ChatGPT is its privacy policy, which states that the communication is not private and that it collects the contents of the conversation. Patent practitioners should avoid sharing confidential information of proprietary innovations with ChatGPT.

Limitations of ChatGPT:

From the above experiments, it is clear that ChatGPT is in its infancy and is able to provide answers to simple, well-established facts. For questions that require deeper analysis, the performance of the tool is far inferior to even a quick analysis by a subject matter expert.

Patent analysis is closely dependent on interpretation and analysis to establish equivalence. To be done well, it requires a balance between leveraging data, utilizing technology, and understanding the art and context behind an invention. While ChatGPT has been a revelation to the world, at this point it does not have the ability to outperform a patent professional. Below are some of the additional drawbacks in the current version of ChatGPT:

• Lack of updated real-time data: ChatGPT is plagued by incomplete backend data and does not provide real-time data on patents after September 2021. Additionally, it provides answers only based on the data used to train it, rendering its results dubious when considering future technologies that do not have an established or structured literature base.
• No access to patent/non-patent databases: ChatGPT does not have access to patent or non-patent databases. This becomes detrimental to the accuracy of its responses.
• Incorrect patent information: The tool often shows incorrect patent information for a patent number search, incorrect bibliographic information, and returns highly irrelevant patent numbers.
• Incorrect data provided for patent classifications: The ChatGPT tool provides incorrect classification codes and even shows incorrect definitions of patent classifications. Patent classification is a critical aspect of patent search, making ChatGPT useless for classification-based searching.
• Unable to handle complex search queries: Per these experiments, ChatGPT was not able to provide relevant prior arts or products around an input patent, claim, or key feature. It could not identify a knock-out art because it could not support complex search strings.
• Failure to accurately summarize patents: When asked to summarize a patent claim, ChatGPT simply paraphrases the text available in the claim, without any elaboration on the scope of the claim elements. By only recapturing portions of the abstract, prime teaching, and claims, it fails to identify a patent’s essential elements.
• Inept at interpretation and inferences: ChatGPT fails to interpret a patent or draw logical inferences from different features of the prior-art.
• Does not analyze patent prosecution history: ChatGPT does not analyze the prosecution history of a patent to report relevant prior art. This keeps the tool from understanding the novelty of a patent, establishing critical cut-off dates, estoppels, etc.
• Privacy issue: There is no privacy or protection for any information shared with ChatGPT. Sharing any confidential information, especially related to pending applications, could compromise competitive edge.

Conclusion:

After reviewing different use cases for ChatGPT on patent analysis, the technology only shows promise in helping with preliminary research, such as understanding a technology, finding relevant keywords or synonyms, and finding broad level products. In the current state, it has indisputable flaws around even the most basic steps in patent and product searching. To keep results within scope, it requires continual human involvement, which essentially negates the tool’s primary function. This makes it essential to validate and verify all its responses by patent and technology experts.

We, at UnitedLex, provide digitally powered solutions that enable IP leaders to create, monetize, and fully exploit their patent portfolios. Key to these services is Vantage for IP, our AI-assisted technology that produces comprehensive competitive analyses that go far beyond the industry standard. It is built with continual data inputs, so the insights are ready within minutes to give customers a speed and intelligence advantage over rivals. With Vantage for IP corporations and law firms can confidently make strategic patent decisions that enhance ROI, create revenue opportunities and increase efficiencies for unmatched cost savings.