iPad: Semantic Annotation and Markup of Radiological Images

iPad: Semantic Annotation and Markup of Radiological Images
Daniel L. Rubin, MS, MD,*1,2 Cesar Rodriguez, MD,1 Priyanka Shah,3
and Chris Beaulieu, MD, PhD2
1Center for Biomedical Informatics Research, 2Department of Radiology, and 3Department
of Computer Science, Stanford University, Stanford, CA
Abstract
image information is not directly accessible to
machines, unlike other biomedical data such as
Radiological images contain a wealth of information,
genomics, proteomics, and electronic medical records
such as anatomy and pathology, which is often not
data. If the semantic information in images were
explicit and computationally accessible. Information
recorded in a structured manner, researchers and
schemes are being developed to describe the
clinical practitioners could more efficiently search
semantic content of images, but such schemes can be
and analyze large image repositories according to
unwieldy to operationalize because there are few
particular features of images (e.g., discover
tools to enable users to capture structured
characteristic image appearances of particular
information easily as part of the routine research
diseases, compute the average lesion size, and
workflow. We have created iPad, an open source
summarize types of abnormalities in different
tool enabling researchers and clinicians to create
diseases). In addition, it would be possible to
semantic annotations on radiological images. iPad
combine images with complementary non-image data
hides the complexity of the underlying image
to enhance biomedical discovery.
annotation information model from users, permitting
them to describe images and image regions using a
Schemas to make the semantic contents of images
graphical interface that maps their descriptions to
explicit are being developed,1-3 however, experts who
structured ontologies semi-automatically. Image
interpret images do not generally record their
annotations are saved in a variety of formats,
observations in the highly structured format of such
enabling interoperability among medical records
schemas. Images are viewed and interpreted in
systems, image archives in hospitals, and the
Picture Archiving and Communication Systems
Semantic Web. Tools such as iPad can help reduce
(PACS) or in a variety of image viewers for image
the burden of collecting structured information from
manipulation. While these systems provide tools to
images, and it could ultimately enable researchers
label images (Figure 1), they do not permit users to
and physicians to exploit images on a very large
record the key semantic image content in a standard
scale and glean the biological and physiological
machine-interpretable format. A tool permitting
significance of image content.
researchers to describe the semantic information in
images in a manner that fits within the research
Introduction
workflow could help them to collect the necessary
structured image data.
Images produced by radiology modalities are a
critical data type in biomedicine because they provide
Several groups have previously developed annotation
rich visual information about disease, complementary
tools in biomedicine and radiology, focusing on
to non-image data such as genomics, proteomics,
acquisition of the graphical symbols or on focused
histopathology, and clinical laboratory results. While
application needs,4, 5 however, there are several
the raw images themselves can be useful in some
challenges that have not yet been addressed by the
computer processing applications, much of the critical
prior work on image annotation tools. The first
semantic content in images is not explicit and
challenge is that current annotation tools lack an
available to computers, hindering discovery in large-
information model for semantic image annotations.
volume image data sets.
Such models are important for defining standard data
representations, the types of information that image
The “semantic content” in images is the information
annotations convey, and the types of queries that are
extracted by radiologists who view them: the visual
possible, similar to the Microarray Gene Expression
appearance of anatomic structures and abnormalities
Object Model (MAGE-OM) in the functional
contained in those structures (“image findings” such
genomics community.6
as a mass in the liver or an enlarged heart). The
radiologist-extracted semantic information is rarely
A second challenge for current annotation tools is that
linked or embedded within the images, instead being
the terminology and syntax for describing the
recorded in text reports or in case report forms as part
semantic content in images varies across existing
of clinical trials. Thus, much critical radiological
systems, with no widely-adopted standards, resulting

AMIA 2008 Symposium Proceedings Page - 626

---

A final challenge is that annotations on images
convey semantic information that is not generally
computationally accessible. Many annotation tools
enable users to create graphical symbols on images,
but the semantics is not explicit and difficult to
process computationally (Figure 1).
In this paper, we describe an image Physician
Annotation Device (iPad) that tackles the challenges
detailed above.
Methods
The iPad application comprises three components: (1)
an information model for semantic image annotations,
(2) a user interface for collecting the annotations from
Figure 1. Radiological image with semantic
users, and (3) a storage back-end to save annotations
annotations drawn in an image annotation tool.
as XML, and to serialize the data to other standard
This image is annotated to convey the major semantic
formats. A screenshot of the iPad application is
content, including anatomy (right lobe of liver),
shown in Figure 2. The user selects an image to be
pathology (irregular mass, tumor vascularity), and
annotated and enters a description using a syntax and
imaging observations (2 cm in size, irregular shape,
grammar that is similar to English (e.g., “an
and poorly-defined margins). These image labels are
enhancing, irregular mass in the right lobe of the
human-readable, but not machine-accessible.
liver”). iPad simultaneously provides user feedback
with text and, if desired, voice of the text. Once the
in limited interoperability. Standard terminologies
content of the annotation is valid, iPad stores the
for describing medical image contents—the imaging
annotation as an XML file, subsequently serialized to
observations, the anatomy, and the pathology—are
a variety of standard formats such as DICOM-SR or
generally not incorporated into current image
HL7-CDA. The image itself remains separate from
annotation tools. In terms of syntax, schemes for
the annotation (the “image metadata”), though the
annotating images have been proposed in non-
annotation is linked to the image through the image
medical domains;3, 7, 8 however, no comprehensive
unique identifier (e.g., DICOM UID).
standard appropriate to radiological imaging has yet
1. Information Model for Semantic Image Annotation
been developed. Current standard syntaxes in use
include (1) Digital Imaging and Communications in
We adapted our previously-described information
Medicine (DICOM), (2) HL7, and (3) HTML and
model (“Annotation and Image Markup (AIM)
RDF. DICOM-SR is emerging as a standard for
schema”)2 to describe the minimal information
representing non-image data, but it does not yet
necessary to record an image annotation (Figure 3),
specifically address semantic annotation content.
inspired in concept by the MIAME project to
describe the universe of information in microarray
A third challenge for medical imaging is that
experiments.9 The AIM schema distinguishes image
annotations often have particular information
“annotation” and “markup.” Annotations describe the
requirements: there may be restrictions on
meaning in images, while markup is the visual
annotations in terms of terminology, or a user may
presentation of the annotations. In the AIM schema,
need to specify more specific information for
all annotations are either an Image-Annotation
particular annotations. For example, if a user makes a
(annotation on an image) or an Annotation-of-
statement about the liver, the user should include one
Annotation (annotation on an annotation). Image
or more observations (e.g., the user could state “2cm
annotations include information about the image as
abnormality in right lobe of liver” but should not
well as their semantic contents (anatomy, imaging
simply state “right lobe of liver”). Another example
observations, etc).2 The AIM schema is an XML
of an annotation information requirement is that if a
schema (XSD file) which enables validation of
modifier of an observation is provided, the
instances of AIM XML files.
observation it modifies should also be stated (e.g., the
user could state “irregular mass” but not just
To tackle the challenge of providing checks on
“irregular”). Image annotation tools should
annotation content, the iPad application is linked to
automatically check the image annotations and alert
an ontology of radiology knowledge. Text entered
the user about annotation requirements.
into iPad is parsed into tokens (typically, 1-3 word
phrases) and matched to entities in the ontology. iPad
determines particular annotation requirements by

AMIA 2008 Symposium Proceedings Page - 627

---

provides feedback to users by highlighting relevant
portions of their entries (Figure 4A). In addition,
users can access the RadLex ontology in a tree viewer
and explore the term hierarchy for appropriate
standard descriptive terms (Figure 4A). Users can
browse the structured image metadata being collected
by iPad, both the pixel data in the image region and
the descriptive information the user creates in the
annotation (Figure 4B).
3. Annotation Storage and Serialization
When the user completes an annotation, iPad stores it
as an XML document, an instance of the AIM
Figure 2. The iPad application. iPad is a plug-in to
schema.2 The image being annotated remains
OsiriX. An image is loaded in OsiriX (left screen) and
unchanged; the image annotation metadata is stored
geometric regions in the image are drawn (shown as
in an XML file that is linked to the image via the
rectangles in the image). The user types their ob-
DICOM UID. A code module processes the AIM
servations about anatomy and abnormality in the iPad
instances to serialize them to DICOM, HL7 CDA
window (middle screen) while simultaneously receiving
(XML), or OWL to enable semantic integration and
feedback from iPad about their annotation (e.g.,
to allow agents to access the image annotations across
controlled terms and spelling errors are highlighted).
hospital systems and the Web.2
The annotations are saved as XML files in the AIM
Evaluation
format (right).
We performed a preliminary evaluation of iPad by
asking two radiologists to use it to annotate 10
consulting the ontology. Currently, iPad incorporates
radiological images acquired as part of a research
the RadLex ontology of radiology,10 constraining the
study. They used iPad to describe the features of
image annotation only to terms in the ontology and
abnormalities seen in each image. The radiologists
alerting the user to spelling errors (Figure 2). iPad
were asked to qualitatively evaluate iPad in terms of
also checks to ensure that radiological descriptions
are complete; specifically, (1) completeness: if an
anatomic entity is mentioned, iPad checks to ensure
that one or more observations about that entity are
recorded (e.g., a mass in the liver); and (2) dangling
modifiers: if a modifier of an observation is
mentioned, iPad checks to ensure that an observation
is also mentioned. These checks are implemented
using the RadLex ontology to determine which tokens
are anatomic entities, observations, and modifiers.
2. User Interface for Collecting Annotations
We created iPad as a plugin to OsiriX.11 Osirix is an
open-source imaging platform for multimodality and
multidimensional radiological image display and
visualization. The program provides image access
and display functionality and enables image viewing
workflow similar to commercial PACS workstations.
However, since the program is open source and
provides a plug-in-architecture, its functionality can
be extended through applications such as iPad.
The iPad tool extends the native OsiriX annotation
object. When a user draws a region on an image in
OsiriX, iPad accesses the geometric information in
Figure 3. AIM Schema and Annotation Instance. A portion
the drawn region and the pixels it contains, storing
of the AIM schema (black) and example instance of Image-
that information in the AIM semantic annotation
Annotation (red) are shown. Only is-a and instance-of relations
information model (Figure 3). Users interpreting
are depicted. The figure shows that the annotation describes an
images type their observations into the iPad, and iPad
image (Image 112), which visualizes the liver, and is seen to
contain a mass in the liver measuring 2cm in size.

AMIA 2008 Symposium Proceedings Page - 628

---

A
browse and choose controlled terms from an ontology
viewer (Figure 4), the iPad tokenizing and term
matching feature allows users to directly type in their
descriptions, reducing potential disruption in the
workflow.
iPad provides visual guidance to users as they record
their observations, giving feedback as to the required
semantic annotation content. As they enter
information, iPad highlights terms matching the
RadLex ontology and flags possible misspellings
(Figure 4). iPad also provides visual cues about the
completeness of radiological descriptions (mentions
B
of anatomy also include descriptions of observations,
and mentions of modifiers also include an associated
image observation). For example, if the radiologist
enters “Liver,” iPad flags this as incomplete, while
“Mass in the Liver” would not be flagged. Similarly,
the user would be alerted that “Enlarged” is an
incomplete description, while “Enlarged Liver” is not
incomplete.
The radiologists who evaluated iPad reported that the
tool enabled them to efficiently annotate the study
images in a similar workflow to which they are
currently accustomed. While iPad collects detailed
Figure 4. iPad user interface and structured
structured information, the manner in which it
semantic annotation information. (A) The iPad
accomplishes this task did not qualitatively hinder the
annotation screen (left) is a simple text box into
radiologists in describing the images and recording
which the user types their image observations (middle
their observations. Based on this encouraging
panel). Ontologies such as RadLex linked to iPad
preliminary evaluation, we are enhancing iPad to
provide visual feedback, such as prompting for the
allow recording of one or many annotations by
appropriate controlled terms for the user entry (right)
enabling single-click phrase insertion as well as
or possible misspellings (highlighted terms in iPad
contextual search of the RadLex ontology during
annotation screen). (B) Users can view the structured
typing. These enhancements will enable much larger,
semantic image annotation data in an expandable tree,
formal qualitative or quantitative evaluations of iPad.
organized according to types of image metadata iPad
collects.
Discussion
usability in the image interpretation workflow and
While radiological images contain a wealth of
utility of the feedback iPad provides, compared with
information, much of it is not in the raw pixels
their experience in unassisted collection of image
themselves, but in observations and interpretations
metadata (the current paradigm in radiology
radiologists and researchers make when viewing
research).
them. Ideally, this additional information should be
connected directly to the image (through annotation)
Results
to enable analyses of images with non-image data.
iPad provides a simple interface to adding structured
Most of the image viewing tools used in research and
semantic information to images, simplifying the
clinical practice incorporate graphical annotation
process of associating semantic information with
palettes, but the semantics of image annotation is not
images or image regions. In current practice without
recorded, and annotation information from images is
iPad, radiologists view images in a workstation such
not easily accessed and analyzed. Tools that support
as OsiriX or PACS, they define image regions of
the collection of structured semantic information from
interest, and they describe the features of those
information models generally provide data entry
regions in a text report or in a data collection form.
screens too detailed and complex to be usable in the
The iPad tool enables a similar workflow, with users
imaging research workflow. Thus, there is need for a
accessing images and defining image regions in
tool for collecting semantic annotation information in
OsiriX; however, iPad also enables the user to collect
images in a streamlined manner.
descriptive information about the image in the same
tool used to view it (Figure 2). While users can

AMIA 2008 Symposium Proceedings Page - 629

---

iPad enables researchers to create semantic
The tool has the capability to check the validity of the
annotations on radiological images. The iPad user
annotations and it could enable imaging researchers
interface provides guidance to required annotation
to access and analyze the semantic content of images
content as users enter information, enabling iPad to
in large imaging archives in the future.
accommodate structured image annotation while
being less disruptive to the research workflow than
Acknowledgements
unassisted annotation methods. In addition, iPad is
This work is supported by a grant from the National Cancer
extensible, and we anticipate the tool will have utility
Institute (NCI) through the cancer Biomedical Informatics
in a broad range of biomedical imaging applications
Grid (caBIG) Imaging Workspace, subcontract from Booz-
where collecting structured information from images
Allen & Hamilton, Inc. 0970 370 X277 1390 and the
is needed.
National Institutes of Health, CA72023.
A useful feature of iPad is annotation content checks.
References
The spectrum of image metadata that can be collected
1. Lober WB, Trigg LJ, Bliss D, Brinkley JM. IML: An
can be large, and certain combinations of information
image markup language. J Am Med Inform Assn.
may be appropriate in particular contexts. iPad
2001:403-7.
accesses knowledge in an ontology to provide
2. Rubin DL, Mongkolwat P, Kleper V, Supekar K,
feedback to the user on the annotation content.
Channin DS. Medical Imaging on the Semantic Web:
Admittedly, the current content checks are simplistic,
Annotation and Image Markup. 2008 AAAI Spring
focusing on spelling and matching terms in the
Symposium Series, Semantic Scientific Knowledge
RadLex ontology. However, richer ontological
Integration; 2008 (in press); Stanford University.
knowledge could be accessed to support more
http://stanford.edu/~rubin/pubs/Rubin-AAAI-AIM-
2008.pdf; 2008 (in press).
complex annotation checks in the future. iPad has
been architected as an extensible tool, and we are
3. Troncy R, van Ossenbruggen J, Pan JZ, Stamou G.
planning to support richer annotation checks by
Image Annotation on the Semantic Web. W3C Incubator
Group Report 14 August 2007. 2007 [updated 2007; cited
extending the ontologies iPad accesses.
March 1, 2008]; Available from:
We do not necessarily advocate iPad for routine
http://www.w3.org/2005/Incubator/mmsem/XGR-image-
clinical radiological practice; iPad is targeted
annotation/.
primarily for enabling imaging research, where
4. Armato SG, 3rd. Image annotation for conveying
collection of rich semantic information is needed for
automated lung nodule detection results to radiologists.
radiological discovery and to advance imaging
Academic radiology. 2003 Sep;10(9):1000-7.
science. In clinical practice, few cases require
5. Zheng Y, Wu M, Cole E, Pisano ED. Online annotation
detailed image annotation, though it could be useful
tool for digital mammography. Academic radiology. 2004
for radiologists to selectively annotate clinical images
May;11(5):566-72.
to enable discovery in large clinical image archives.
6. Whetzel PL, Parkinson H, Causton HC, Fan L, Fostel J,
Enhancements to the user interface would be needed
Fragoso G, et al. The MGED Ontology: a resource for
to make iPad practical to use in high-volume image
semantics-based description of microarray experiments.
Bioinformatics. 2006 Apr 1;22(7):866-73.
interpretation paradigms, such as incorporating voice
recognition (obviating the need for the user to type in
7. Halaschek-Wiener C, Golbeck J, Schain A, Grove M,
the findings).
Parsia B, Hendler J. Annotation and provenance tracking in
semantic web photo libraries. Provenance and Annotation
While the focus of iPad is on annotating radiological
of Data. 2006;4145:82-9.
images, we believe this tool could be applicable and
8. Khan L. Standards for image annotation using Semantic
useful to all of biomedical imaging. If the application
Web. Computer Standards & Interfaces. 2007
domain could benefit from describing structured
Feb;29(2):196-204.
semantic information in the image or image regions,
9. Brazma A, Hingamp P, Quackenbush J, Sherlock G,
then iPad may be valuable, provided extensions to the
Spellman P, Stoeckert C, et al. Minimum information about
iPad ontology are made to support the terminology
a microarray experiment (MIAME)-toward standards for
and annotation needs of that domain.
microarray data. Nat Genet. 2001 Dec;29(4):365-71.
10. Langlotz CP. RadLex: a new method for indexing
Conclusion
online educational materials. Radiographics. 2006 Nov-
Dec;26(6):1595-7.
We have developed iPad, a semantic image
11. Rosset A, Spadola L, Ratib O. OsiriX: an open-source
annotation tool for radiology. This tool extends the
software for navigating in multidimensional DICOM
functionality of a currently-available image viewing
images. J Digit Imaging. 2004 Sep;17(3):205-16.
platform, enabling users to add semantic content to

images via a simple user interface and that can be
incorporated into the imaging research workflow.

AMIA 2008 Symposium Proceedings Page - 630

---