Skip to content

NL to OWL using Prompt Engineering

Jump to bottom
Antonio Zaitoun edited this page Aug 25, 2024 · 1 revision

In this section, we focus on work done in an attempt to translate natural text into OWL statements. The task of axiom extractions is one of the most neglected areas of research when it comes to ontology learning. This is because axioms do not appear in natural language in the form of triples in most cases, but rather as complex statements. For example, consider the following statement: 

Someone who works and studies at a university is a PhD student

We can express this statement using the following OWL statement: 

EquivalentClasses(
    :Phd_student 
    ObjectIntersectionOf(
        ObjectSomeValuesFrom(:studiesAt :University) 
        ObjectSomeValuesFrom(:worksAt :University)
    )
)

While existing work is capable of verbalizing an OWL statement (e.g., Kaljurand, 2007, Androutsopoulos, 2013), the process of translating natural language statements into OWL is more challenging. In this section, we experiment with prompt engineering and zero-shot learning of an LLM in an attempt to generate OWL statements from natural text.

Setup

We used ChatGPT version 3.5 browser version (https://chat.openai.com/ - Last accessed August 2023) and followed the Prompt Engineering Guideline to compose the following prompt, which included four examples from Androutsopoulos, 2013. Ultimately, our prompt exhibited the following structure:

  1. Instruction: Your job is to translate natural language text into OWL 2 Web Ontology Language Functional-Style Syntax.
  2. Context: You are an ontology expert.
  3. Output Indicator: The output shall only contain OWL statements and nothing else + Examples
  4. Input Data: <Provided as follow-up prompts> 
You are an ontology expert. Your job is to translate natural language text into OWL 2 Web Ontology Language Functional-Style Syntax. Translation examples are provided. The output shall only contain OWL statements and nothing else. Acknowledge by responding with "Understood". 

Example 1:
Text: St. Emilion is a kind of Bordeaux from the St. Emilion region. It has red color and strong flavor. It is made from exactly one grape variety: Cabernet Sauvignon grapes.

OWL:
EquivalentClasses(:StEmilion
ObjectIntersectionOf(:Bordeaux
ObjectHasValue(:locatedIn :stEmilionRegion) ObjectHasValue(:hasColor :red)
ObjectHasValue(:hasFlavor :strong) ObjectHasValue(:madeFrom :cabernetSauvignonGrape)
ObjectMaxCardinality(1 :madeFrom)))

Experiments

Example 1: People

We conversed with the model using follow-up prompts that included nothing but natural text we wished to convert into OWL statements. For the sentence - Any man who owns a bike and does not own a car is liked by Mary, the model responded with the following result, which was not valid in OWL: 

SubClassOf(ObjectAllValuesFrom(:likes ObjectIntersectionOf(ObjectSomeValuesFrom(:owns :Bike) ObjectComplementOf(ObjectSomeValuesFrom(:owns :Car))) ObjectSomeValuesFrom(:likesBy :Mary)) :Person)

This issue of having incorrect syntax output has been observed in several other responses.

Example 2: Marine Biology

On the other hand, for the following paragraph, from the domain of oceanography, the model successfully generated a valid response that is partially correct. For simplification, the model response is also presented in Protege in Figure 1. 

Phytoplankton, also known as microalgae, are similar to terrestrial plants in that they contain chlorophyll and require sunlight in order to live and grow. Most phytoplankton are buoyant and float in the upper part of the ocean, where sunlight penetrates the water. Phytoplankton also require inorganic nutrients such as nitrates, phosphates, and sulfur which they convert into proteins, fats, and carbohydrates.
ClassAssertion(:Phytoplankton :phytoplankton)
SubClassOf(:Phytoplankton ObjectIntersectionOf(
    ObjectSomeValuesFrom(:contains :Chlorophyll),
    ObjectSomeValuesFrom(:requires :Sunlight),
    ObjectAllValuesFrom(:buoyantIn :UpperPartOfOcean),
    ObjectAllValuesFrom(:requiresNutrient ObjectUnionOf(:Nitrate :Phosphate :Sulfur)),
    ObjectAllValuesFrom(:convertsInto ObjectUnionOf(:Proteins :Fats :Carbohydrates))
))
Description of Image

The model output was verbalized using owl-to-ace (http://attempto.ifi.uzh.ch/site/docs/owl_to_ace.html) and the below output was produced. 

Every Phytoplankton is something that contains a Chlorophyll and that requires a Sunlight and that buoyantIn nothing but UpperPartOfOcean and that convertsInto nothing but things that are a Carbohydrates or that are a Fats or that are a Proteins and that requiresNutrient nothing but things that are a Nitrate or that are a Phosphate or that are a Sulfur.

When comparing the source text with the verbalized text, it can clearly be seen that (1) Some information is lost. For example, it was expected that Phytoplankton would be a subclass of microalgae. And (2) factually incorrect information was presented. From the generated OWL, it can be understood that all phytoplankton must be buoyant, whereas in the source text, it says Most phytoplankton. An additional mistake that can be seen here is that the model suggests that phytoplankton itself is converted into either Carbohydrates, Fats, or Proteins. But in reality, it is the inorganic nutrients consumed that are converted. A visualization depicts these differences in Figure 2.

Description of Image

Example 3: Cancer-Related Fatigue Treatment Guideline

Another example from the domain of patient care, specifically a medical guideline, the model is able to extract some of the classes but exhibits a major weakness of being too literal in the resulting response. When examining the output, the response can be read in plain English. This, however, comes as no surprise, being that the model is not trained on modeling ontologies, despite providing some examples in the initial prompt. A comparison is presented in Figure 3. 

Mindfulness-based stress reduction: Mindfulness-based clinical interventions combine meditation exercises with psychoeducational elements, cognitive-behavioural interventions and movement exercises. The core practices are sitting meditation with breath awareness and focused attention, awareness of sensations in the body (body scan), yoga exercises (e.g. hatha yoga, mindful movement), walking meditation and insight meditation. The two most used mindfulness-based clinical interventions in oncology are mindfulness-based stress reduction (MBSR) and mindfulness-based cognitive therapy (MBCT). MBSR is a specific multimodal programme focused on improving well-being and health. It has been used as an intervention for cancer patients. Documented benefits after programme participation include improvements in: stress; mood; anxiety; depressive mood; sleep; fatigue; psychological functioning; psychosocial adjustment; stress reduction; enhancements in coping and well-being symptoms; QoL; fear of recurrence.
SubClassOf(:MindfulnessBasedStressReduction ObjectIntersectionOf(
    ObjectSomeValuesFrom(:combinesWith ObjectUnionOf(
        :MeditationExercises,
        :PsychoeducationalElements,
        :CognitiveBehaviouralInterventions,
        :MovementExercises
    )),
    ObjectSomeValuesFrom(:corePractices ObjectIntersectionOf(
        ObjectSomeValuesFrom(:includes :SittingMeditationBreathAwareness),
        ObjectSomeValuesFrom(:includes :AwarenessOfSensationsInBody),
        ObjectSomeValuesFrom(:includes :YogaExercises),
        ObjectSomeValuesFrom(:includes :WalkingMeditation),
        ObjectSomeValuesFrom(:includes :InsightMeditation)
    )),
    ObjectSomeValuesFrom(:benefitsFrom ObjectUnionOf(
        :StressImprovement,
        :MoodImprovement,
        :AnxietyReduction,
        :DepressiveMoodImprovement,
        :SleepImprovement,
        :FatigueReduction,
        :PsychologicalFunctioningEnhancement,
        :PsychosocialAdjustmentEnhancement,
        :StressReduction,
        :EnhancementInCopingAndWellBeingSymptoms,
        :QualityOfLifeImprovement,
        :FearOfRecurrenceReduction
    ))
))
Source Prediction
protege_crf_expert protege_crf_model

Example 4: Chiropractic Care of Special Populations

Finally, another response from the model, based on input from a book named Chiropractic Care of Special Populations that explores tailored approaches in chiropractic treatment for distinct demographic groups, addressing unique needs and considerations for optimal health outcomes. The input describes the potential demographic of children that may have been exposed to lead, and although this was not stated explicitly in the prompt, the model was able to assign the proper class name - ChildsWithPotentialLeadExposure. 

Children who live in or frequently visit housing built before 1950 that is dilapidated or undergoing renovation; who come in contact with other children with known lead toxicity; who live near lead processing plants or whose parents or household members work in a lead-related occupation; or who live near busy highways or hazardous waste sites.

SubClassOf(:ChildsWithPotentialLeadExposure ObjectIntersectionOf(
    ObjectSomeValuesFrom(:liveInOrFrequentlyVisit ObjectUnionOf(
        :HousingBuiltBefore1950,
        ObjectIntersectionOf(
            :HousingBuiltBefore1950,
            ObjectOneOf(:isDilapidated),
            ObjectSomeValuesFrom(:isUndergoingRenovation)
        )
    )),
    ObjectSomeValuesFrom(:comeInContactWith ObjectUnionOf(
        :ChildrenWithKnownLeadToxicity,
        :LiveNearLeadProcessingPlants,
        ObjectIntersectionOf(
            ObjectSomeValuesFrom(:workInOccupation :LeadRelatedOccupation),
            ObjectSomeValuesFrom(:isHouseholdMemberOf :Parent)
        ),
        ObjectSomeValuesFrom(:liveNear :BusyHighways),
        ObjectSomeValuesFrom(:liveNear :HazardousWasteSites)
    ))
))

Despite assigning a correct name to the class, the interpretation of text by the model is semantically wrong. In the paragraph, it is stated that children who live in or frequently visit housing built before 1950 and (is dilapidated or undergoing renovation) and who come in contact with: 1) other children with known lead toxicity or 2) who live near lead processing plants or 3) a household member who is a parent and works in a lead-related occupation or 4) who live near busy highways or hazardous waste sites. The definition was a list of OR (disjunction) and the LLM thought it was AND(conjunction) between the child who lives in an old house and who is in contact with other people. This is true for coming in contact with child friends with known lead toxicity, but the other criteria do not relate to who the child comes in contact with but to what is near the child's location (near a lead processing plant, highway, or waste site), or the child's parents occupation. The expected model is presented in the Figure below.

protege_children_with_lead_exposure

Discussion

Based on our experiments, while prompt engineering can guide LLMs to produce some valid OWL statements, the accuracy of the output is heavily dependent on the complexity of the input text. The LLM often struggles with context-dependent information, particularly when dealing with intricate or nested concepts, and it can produce syntactically incorrect or incomplete OWL statements. Additionally, even when the syntax is correct, the semantics may not always align perfectly with the intended meaning of the input text. Future work should focus on refining prompt engineering techniques and exploring other approaches, such as fine-tuning on specific datasets, to improve the translation of natural language into OWL.

Clone this wiki locally