International Real Estate Society Conference 1999
Co-sponsors: Pacific Rim Real Estate Society (PRRES)
Asian Real Estate Society (AsRES)
Kuala Lumpur, 26-30 January 1999
ACCURACY ISSUES FOR AUTOMATED AND ARTIFICIAL
INTELLIGENT RESIDENTIAL VALUATION SYSTEMS
PETER ROSSINI
School of International Business,
Centre for Land Economics and Real Estate Research (CLEARER)
University of South Australia
Phone (61-8) 83020649, Fax (61-8) 83020512, E-mail peter.rossini@unisa.edu.au
Home Page http://www.unisanet.unisa.edu.au/staff/Homepage.asp?Name=peter.rossini

Keywords: Artificial Intelligence, Automated Valuation Methods, Valuation Accuracy, Valuation
Variation, Valuation Methodology, Residential Valuation.

Abstract: This paper extends current work in the field of Valuation Accuracy and Valuation Variance.
These issues are examined in respect of Automated and Artificial Intelligent Valuation Systems. The issue
of how accurate an automated system must be to compete with manual valuations is explored. The paper
relies upon existing literature, a survey of practising valuers’ and a previous valuation variation study for
residential properties in Adelaide. The results show that valuers expectations of valuation accuracy greatly
exceed the actual results achieved and that their expectations for automated systems are unrealistic on this
basis.
Introduction
The development of Artificial Intelligent Systems for the valuation of residential property is occurring rapidly.
In South Australia, such a system is under development at the University of South Australia. One of the
difficulties is finding an accuracy level to test this system against. Most valuers’ see even small error terms
as unacceptable. This is based on the premise that their manual valuations are “always right”. This paper
explores recent literature that seriously questions the accuracy and variations of valuations. A valuation
variation study in Adelaide is used to show that a wide range of values is typical even for basic residential
valuations. A small survey of practising valuers is used to gauge opinions of the accuracy of manual
valuations compared to automated systems.
It is hoped this paper with help fill a gap in the literature in this area. In particular the issues of accuracy in
automated systems. This will become a major issue as more automated and artificial intelligent systems are
developed in Australia and Internationally.

---

Accuracy Issues for Automated and Artificial Intelligent Residential Valuation Systems – Rossini
Page 2

Automated and Intelligent Systems
Technological changes are leading to rapid changes in the way that professionals perform their business.
Professionals in the property industry are no exception. Baen et al (1997) discusses a range of affects to
property professionals which are emerging in the US which they believe may lead to very significant
reductions in employment levels. One of the technological changes is the introduction of automated and
intelligent valuation systems. These are most often used in basic residential valuations, particularly for
valuations to support finance.
The foundation of these systems was in the mass appraisal field. Early computer aided systems became
popular in the USA in the early 1980’s. A practical discussion of the methods by Sauter, B. W. (1985)
suggested a logical approach to automate the typical actions of the valuer. Kershaw (1997) used the
general approach suggested in developing the current basic prototype at the University of South Australia.
Other methodologies have been suggested by Jensen (1984) who suggests a variety of different
approaches.
Rossini et al (1992, 1993) suggested the first system that included automated valuation concepts based in
an Australian setting. This proposed system included a complex sales history management system based
on government databases, automated time series system and automated valuation system. The first stage
of the system (data management) was released commercially in 1994 (Kershaw, 1994, 1996) with a tested
prototype of the automated time series being presented in January 1997 (Kershaw et al 1997). Initial work
on a prototype automated valuation system was demonstrated in late 1997 (Kershaw, 1997)
There is evidence that several systems are commercially available in the USA. Systems in Marion County,
Linn County and Benton County, Oregon are described by Detweiler (1996). These are working systems
(Computer Assisted Real Estate Appraisal System, CAREAS) - used by a variety of users but primarily for
mortgage finance purposes. Several Internet sites refer to similar systems. Jensen (1990) reports on the
initial development of a system for Seville in Spain. It is clear that systems vary considerably in each
location, primarily due to differences in purchaser preference (market price allocation) as well as variations in
the type, amount and quality of data that is available.
The current systems are based on relatively straightforward statistical modeling systems with an expert
system element. More complex systems have also been proposed but there is little evidence of any being
applied in at this stage. Eckert et al (1993) proposed a more complex system using econometric modeling.
They stated that a Computer-Assisted Real Estate Appraisal (CARA) would have wide application,
particularly in the risk management of mortgage loan portfolios.
“First, the model can be used to provide an automated, market-based valuation prior to an initial onsite inspection.
CARA’s most important risk management contribution, however, is its ability to provide an automated review
appraisal based on the comparison properties cited in a subject appraisal as well as other subject and comparison
properties. Finally, CARA can automatically update original sale prices to current market levels”.
The latest major change has been the suggestion that systems may use true artificial intelligence rather than
an automated approach. Such systems would normally be based on artificial neural networks. The
distinction between automated systems and artificial intelligent systems is clarified by Rayburn (1995).
While the automated systems use statistical techniques such as regression, intelligent systems based on
neural networks are capable of learning in a more complex and non-linear manner.
The opportunity for their use has been investigated in recent years. For example Borst (1991) reported the
use of ANN to data sets of family residences in New England. Tay and Ho (1992, 1994) examined sets in
Singapore using 833 residential apartment properties for training and tested this against 222 case set of
similar apartment properties. Do and Grudnitiski (1992) used data from a multiple listing service in California
while Evans (1993) worked with residential housing in the United Kingdom. Recent works comes from
Worzala (1995), Borst (1995, 1996), McCluskey (1996a, 1996b) and Rossini (1997a, 1997b, 1997c).
Rossini’s research was based on data from South Australia and demonstrated that the results from artificial
neural networks could potentially produce superior results to more traditional econometric models in certain
circumstances.
What is not clear from the research is the level of accuracy that is expected. This paper will examine various
aspects of valuation accuracy and variations and draw some conclusions about what is required in
automated or intelligent systems.

---

Accuracy Issues for Automated and Artificial Intelligent Residential Valuation Systems – Rossini
Page 3
Valuation Accuracy and Valuation Variation
Before it is possible to assess the accuracy needed from an automated or intelligent valuation system, it is
necessary to consider the broader question of valuation accuracy. The issues of accuracy has always been
of interest to valuers and their clients but only in recent years has there been an active research focus on
this issue. In the UK, Hager & Lord (1985) started the modern debate and quoted some accuracy figures.
This was followed in the UK by Hutchison (1996) and Crosby et al (1998a, 1998b). Newell & Kishore
(1998a, 1998b) and Parker (1998) introduced some accuracy and variation estimates from Australia.
Amongst these papers were many others that discussed methodology. (Lizieri et al, 1991 & 1993;Brown,
1992 & 1998;McAlister, 1995; Wiltshaw, 1996; Boronico, 1997).
There is a clear split between the testing of valuation variance and valuation accuracy. This seems to have
been a point of some confusion in some of the research. Lizieri et al describe these as
“The former (Valuation Variance) occurs where two or more valuers arrive at substantially different values for the
same property…….it is difficult to statistically test the significance of observed variations. The latter (Valuation
Accuracy) concerns the relationship between achieved prices and prior valuations.”
Or as Crosby et al (1998b) succinctly puts it
“…valuation accuracy (valuations against price) and valuation variance (valuations against valuations)”
There is also considerable debate about the analytical methods used in some of the research that may tend
to over estimate the errors. This is a particular argument of Brown et al (1998) who argues that
bootstrapping is a more appropriate statistical method given sample sizes used in most of the research.
Notwithstanding this there is some evidence of the accuracy of valuations from the available research.

The results from Hager & Lord (1985), Hutchison (1996), Morgan, Drivers Jonas/IPD, Matysiak & Wang,
Blundell & Ward, all as quoted in Crosby (1998a), Newell (1998a), Parker (1998) and DeVries (1992) are
summarised in Table 1.

Table 1 - Summary of Valuation Variance and Valuation Accuracy Research

Percentage of properties that fall into each Accuracy Range
Hager & Lord(1985)
Hutchison (1996)
Morgan(1993)* Newell(1998a)
Driver Jonas/IPD* Matyiak & Wang* Blundell & Ward*
Parker (1998) DeVries (1992)

% Absolute Error
Office
Shops
Reversion Rack Rent Commercial Commercial**
Commercial
Commercial
Commercial Commercial
<5%
40
50
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
< 10%
90
80
69
61
66
66
30
30
35
n/a
n/a
< 15%
n/a
n/a
n/a
n/a
80
80
n/a
55
n/a
n/a
n/a
< 20%
100
90
93.5
85
90
90
67
70
80
n/a
n/a
Mean Absolute Error

n/a
n/a
9.53%
10.50%
8.60%
n/a
n/a
n/a
n/a
6.04%
10.78
* As quoted in Crosby (1998a)

** Typical Figure - papers quotes figures over a four year period

The results show very little consistency. Generally the mean absolute error is in the range of 10%. However
anywhere between 30% and 90% of valuations would seem to fall within the plus or minus ten-percent
range. While some of this will be due to different locations and type of property, the range does seem a little
excessive. Significantly there is generally at least 10% of valuations with errors exceeding plus or minus
20% although in some studies there would seem to be over 30% of valuations outside of a +/- 20 % range. It
should be stressed that these relate to commercial manual valuations, but notwithstanding this some of the
accuracy levels would seem to be pathetically low.
On the basis of these studies it would not seem unreasonable to expect an automated or intelligent system
to have a mean absolute error above 10% and 70% of all valuations to have an error of less than +/- 10%.
Valuation Variation Study in Adelaide
The only known study of valuation variance in South Australia is an unpublished work by Daniels (1984). In
this study the author sought to have a large number of licensed valuers, assess the value of two residential
properties on the same day. Two detached houses were selected. The first was a typical property in a
relatively homogenous area while the second was a more difficult assignment being a large house in a
popular high value location, but located within five minutes drive of the first property. Neither had been sold
recently nor were currently listed for sale.

---

Accuracy Issues for Automated and Artificial Intelligent Residential Valuation Systems – Rossini
Page 4
Valuers were randomly selected from the list of licensed valuers in the sate. Some forty valuers responded
that they would complete the valuation however only 18 submitted final results. All these valuers inspected
the two properties on the same day but at different times. Each was given some basic information including
a list of over twenty recent sales.
On completing the valuation, valuers submitted a questionnaire which included details of their age,
education, experience and the nature of valuation work they were involved with. For each property they
were asked for a valuation figure, method of valuation, time taken to perform the valuation and a ranking of
the most comparable sales. This additional data was collected to attempt to establish any significant
patterns in the variance.

For the basic residential valuation, the average absolute error was 5.3% with a standard deviation of 2.3%
this. For the more complex property the average was 8.9% with a standard deviation of 4.8%. The error
levels are shown in Table 2.
Table 2 - Summary of Valuation Variation Study for Adelaide Residential Properties (Daniels, 1984)
% Variation from Basic Property - Colonel
Complex Property -
Mean
Light Gardens
Malvern
<5% 9
50%
7
39%
< 10%
17
95%
9
50%
< 15%
18
100%
17
95%
<100%
18
100%

These results compare more than favourably with those of the other research. However this is to be
expected as this study involves the valuation of freehold residential properties. This is the type of valuation
assignment that is most likely to be estimated using automated or intelligent systems. This study would
suggest that for manual valuation of residential properties in Adelaide, a mean absolute error below 10%
would be expected. But that a mean absolute error of less than 5% would be expected in locations where
there was relatively homogenous housing. In these homogenous locations it would be expected that some
95% of valuations would fall within the 10 % margin of error.
Accuracy of Government Assessed Values in Adelaide
The most likely use of automated and intelligent valuations systems is in government assessments for rating
and taxation as well as for mortgage and financial purposes. The South Australian Government issues
Capital Values for properties and it is generally believed that they have a reasonable degree of accuracy for
residential properties. This is highlighted by the fact that some financial institutions use these as finance
value estimates for properties of lower value and where the loan is for a low percentage of the capital
assessed value. There is a broad range of literature covering the use of assessment ratios and other
indicators to assess these valuations. There is a re-emergence of the idea of using these assessments for
market valuations by utilising modeling. A recent paper by Goolsby (1997) suggests methods for such
analysis. Given the general use of these figures for a variety of purposes and the suggestions of new
methods to adjust these to more realistic values thorough continuous modeling, it is worth considering the
accuracy of the South Australian assessments. It could well be argued that if they are accurate enough that
they might well substitute for any new system. To test the accuracy of assessments for houses in the
Adelaide metropolitan area, sales details were collected over a three-month period (July-September 1998).
Sales of detached houses were chosen and screened to remove all properties that would probably have sold
under non-market conditions. The sales price was then compared to the capital assessed value. The
assessed values were tested against sale price using the same 5%, 10%, 15% and 20% absolute error
margins that were used before. The mean error and mean absolute error were calculated. These are
shown in Table 3. The errors are not significantly different from those measured elsewhere, except that they
show the typical systematic error typical of assessed values i.e. they are generally conservative as indicated
by the mean error of –8.48%. This is nearly always the case in assessed values (Goolsby, 1997) and is one
of the reasons that assessments need to be adjusted before they are used as surrogates for market value.
For this accuracy test the most simplistic adjustment was made. Simply to inflate all CV’s by the average
error of 8.48% to create a new Adjusted CV. This was then compared to the actual sale price as before.
The resultant errors are also shown in Table 3.

---

Accuracy Issues for Automated and Artificial Intelligent Residential Valuation Systems – Rossini
Page 5
After only this simple adjustment (similar to what is often carried out by many users of the assessments) the
accuracy of the assessments is comparable to those from manual valuations and may well be superior to
simple automated or intelligent systems! This may be considered surprising as the assessments are based
on a Computer Assisted Valuation method that is often criticised as being simplistic (Marano 1993).
The accuracy of these assessments is
not inconsistent with the accuracy figures
Table 3 - Accuracy of Government Assessments
reported in the literature. A Mean
absolute error of less than 10% and over
Percentage of properties that fall
60% of cases having an error of less than
into each Accuracy Range
10% is quite acceptable. However it does
% Absolute Error
Capital Value (CV) Adjusted CV
fall well below the accuracy level reported
<5%
25.5%
34.7%
for housing in Adelaide. This is expected
< 10%
50.8%
62.8%
given that these were manual valuations
< 15%
70.8%
82.8%
whereas the government assessments
rely on a simple computer assisted
< 20%
85.3%
90.3%
model. The error from an intelligent or
<100%
100.0%
100.0%
automated system would certainly have
Mean Error
-8.48%
0.00%
to be less than those experienced from
Mean Absolute Error
11.47%
9.53%
the government assessments in order to
compete.
Expected Level of Accuracy
The valuation accuracy and valuation variation studies help to demonstrate how accurate valuers actually
are, but this does not show if this level is acceptable to end-users. The amount of error that is acceptable is
a difficult question. Clearly the users of valuations would prefer there to be no error at all or at least a very
small one. Valuers on the other hand would prefer the “allowable” error to be larger to limit the opportunities
for professional negligence claims. In the UK this has been partially formalised by the recognition in the
courts of a reasonable margin of error. A valuation figure outside of the margin for error is seen as sufficient
evidence of negligence. So that regardless of the actual process or method of valuation, a valuer may be
found negligent by simply being out side the margin for error. (Crosby et al, 1998a, 1998b). This situation is
criticised by Crosby as being inconsistent with the requirements of other professionals.
Regardless of the logical arguments for or against the margin for error concept, the courts do provide some
useful precedent for the setting of an acceptable level of error. From the various precedents, Crosby et al
(1998b) concludes that,
“It was found that no expert had argued for a bracket outside of +/- 20% of the correct valuation and that they
routinely suggested brackets within these margins. In response to this evidence, UK courts had normally
determined that a margin of error should be no more than +/- 10%; in exceptional cases (for example residual
valuations) this had been extended to around 15%. The highest bracket determined by a court has been +/-
17.5% in the UK.”
For residential properties the judges have generally been less lenient expecting smaller margins of error.
The courtroom precedent has again been summarised by Crosby et al (1998a)
“…. The expert witnesses agreed that on a standard estate house the acceptable margin might be no more than
5% ……….. however, it appears that 10% margin of error will be fairly readily accepted, rising towards 15% if the
type of property or the state of the market is such as to present the valuer with a particular difficult challenge.”
Parker (1998) considered that
“in his experience as an institutional investment property operative, a valuation range of 10% around an
acquisition or disposal market price generally appears to have been considered acceptable in Australia.”
This is referring to the valuation of commercial premises. Parker uses a survey of seven property trust
managers to identify an expected range. He concludes that most of them are expecting an accuracy level
within plus or minus 5%.
So there would appear to be a large difference between what end users want and what valuers are actually
providing. The figure of 10% error margin is consistent with the mean error that is typically found in most of
the research. However clearly the expectation of a 10% error by the end users is that this would be the
maximum error probably with a mean of closer to 5%. However even in the best case (the basic residential
house in Adelaide) only 95% of cases fell within the 10% margin and many users would argue that the type
of property involved in this study (basic freehold residential) would warrant an error range of only +/-5%.

---

Accuracy Issues for Automated and Artificial Intelligent Residential Valuation Systems – Rossini
Page 6
Valuers Expectations of Automated and Intelligent Systems
The attitude of valuers towards automated or intelligent systems is an important consideration in their
introduction. Baen (1997) explained how some, valuers might be replaced by automated or intelligent
systems and suggested that valuers would contest their introduction. But not withstanding this, the pressure
from end-users, particularly banks and finance institutions would prevail, and their introduction was
inevitable.
To examine the situation within South Australia a seminar and survey were conducted to gauge the
understanding of valuers about automated and intelligent valuation systems and to measure their attitude
towards the commercial introduction of such systems. The seminar was conducted during an Australian
Property Institute seminar on new technology in valuation. As a part of the seminar a short survey was
conducted. The aim was to answer four basic questions
1. What did valuers know about automated or expert valuation systems prior to the seminar?
2. How they thought that automated or expert valuation systems would impact on their business?
3. What was their attitude to the introduction of automated or expert valuation systems?
4. What level of accuracy would be needed in an automated or expert valuation systems and how did this
compare to their perception of the accuracy of manual valuations?
A short questionnaire had open questions and sought to answer the research questions listed above as well
as the type of work and experience undertaken by the respondent. The accuracy questions required the
respondent to consider what percentage of valuations would fit in to four accuracy levels; less than 2% error,
less than 5% error, less than 10% error and greater than a 10% error. Respondents were asked (from their
experience) what percentage of valuations would usually fall into the accuracy ranges for a blind valuation of
a typical residential property. This was defined as the valuation of a property where the asking, offer and
sale prices are all unknowns prior to the valuation.
Some 45 people attended the seminar however only 18 surveys were completed. While this was less than
anticipated, it is sufficient for preliminary analysis, given the qualitative nature of the questions. No
quantitative analysis was possible from the survey except for simple tabulations. Individual comments
however provide some useful insight to valuer’s perception of the introduction of these systems.

Question: What did you know about using Expert Systems for Residential Valuation - prior
to this seminar?
To summarise the comments
• 5 of the 18 people knew nothing of automated systems
• 6 of the 18 knew that they existed but not how they worked or what they could really do
• 5 of the 18 had some idea from the literature of what they were capable off and had some idea of how
they worked
• 2 of the 18 had a good idea about automated and expert systems, had detailed knowledge of how they
work and what to expect from them and had seen them in operation or examples of their in and outputs.

Question: Do you think that Expert Systems will impact on your business. If so how?
Of the 18 people who responded, 16 were involved directly in the provision of valuation services. Of these
valuers, 15 thought that these systems would impact directly on their work.
Some direct quotes from their comments
“Potentially, it could have a significant impact. We need to be proactive with our existing clientele, becoming a
'value added' product in their lending process.”
“Yes, I do and I believe that we will have to adapt to the use of these systems, possibly by jumping the fence and
becoming involved with the application side or by using the system as part of our valuation process.”
“Yes. People will choose the cheapest option and that will ultimately be the computerised one – (this will)
therefore result in a loss of business. Although you may be able to introduce the system into business and it could
be a value added tool.”
“Yes - hugely. With Government trying to reduce costs, salaries are our major costs. Expert Systems would allow
fewer people to perform our role and possibly concentrate on data collection and maintenance, which may lead to
increase revenue-raising potential.”

---

Accuracy Issues for Automated and Artificial Intelligent Residential Valuation Systems – Rossini
Page 7
Question: What is your attitude to the potential introduction of Expert Systems to value residential
properties?
The attitudes were split fairly evenly between positive and negative attitudes. However there was an almost
universal attitude that these systems would soon be an inevitable fact of life for residential valuation. The
positive comments were based on a perception of the time return trade off. It was also seen as an
advantage for younger less experienced valuers.
The following comments are all considered to be a positive endorsement of automated and expert systems
“Is a great tool which I would love to have access to myself. I think that valuers should be pro-active
“I welcome it because, whilst it is scary that my job will change it gives me a chance, with my 3 years experience
to leap frog valuers with 10+ years experience, and become a more sought after professional.”
“I am positive it will allow the 'mundane' work to be done and allow valuers to concentrate on the more complex
properties.”
“It is inevitable because valuers have not maintained the provision of service that is required by clients.”
“Hopefully an increase in speed, so that the valuation will be made available quickly.”
“Depends on the application - like most new systems they will have their place.”
“Will drive some valuers using traditional methods out of the market – (As these valuers) will not be able to
compete on price or service time.”
The clear implication here is that some valuers are keen to move in the automated or intelligent direction,
particularly for mortgage valuations and for rating and taxation purposes.
The negative comments do provide some useful points for concentration, however some may be dismissed
as unsound arguments. The first relates to accuracy of the data used for model building. The following
comments suggest that this would be a major problem.
“Sceptical, but - the area of weakness in the models - properties falling outside of the norm - reliability of original
data e.g. inaccurate: building details, site area, rooms, condition rating, etc.”
“I believe that the data base for comparable sales are too shallow for any meaning full use, except in areas such
as Elizabeth, Salisbury, Marion where properties are very similar and therefore quick and easy to value.”
While these arguments are superficially sound, the reality of residential valuation is that some valuers make
no reference to current sales and those that do infer value from current sales, invariably use the data
provided through the sales history files. This is the same information used in the automated or intelligent
systems. On this basis it may well be argued that the automated or intelligent system will at least use a
sufficient volume of data to reduce the impact of errors for individual properties.
One valuer with a more pragmatic view, said simply:
“Don't like it (automated or intelligent systems)- but bankers will”
This again is an acceptance that while it is not a preferred direction for valuers, that banks will certainly see
some advantages in their use and will probably support the introduction of suitable systems.
The following two comments highlight valuers understandable concerns about the reliability of systems and
their impact on the profession and professionalism.
“It should be given a long time 'on test' before it is relied upon. A few mistakes and a couple of major court actions
will dice the lot. No P/I Insurance for this.”
“If it is inevitable then we must plan to cope with the impact. Main Concerns as follows: (1) Professional habits of
supplier of valuations, (2) Compliance with AIVLE standards if properties are not inspected.(3) Degree of accuracy
for properties that do not have full information. The 10% outscale of range could vary significantly. (4) No
identification if potential builders faults - future risk identification.”
Question: This question relates to the accuracy of a blind valuation of a typical
residential property. This is a valuation of a property where the asking price, offer price and
sale price are all unknown.
In respect of a Manual Blind Valuation, From your experience what percentage of valuations
would usually fall into the following accuracy ranges?
In respect of a Blind Valuation using an Expert System with a minimal inspection of the
subject property only, in your opinion what percentage of valuations would need to fall into
the following accuracy ranges, in order to be acceptable.
The opinions of valuers about current manual valuations are quite consistent with the literature.

---

Accuracy Issues for Automated and Artificial Intelligent Residential Valuation Systems – Rossini
Page 8
Although individual expectations did vary the overall average was that roughly 17% of valuations would be
with in plus or minus 2% error, 43% of all valuations would be within plus or minus 5% and 90% would be
within plus or minus 10% error. Surprisingly valuers generally thought that expert systems would have to
perform better than manual valuations in order to compete on average the expectations were that nearly
20% would have to be within 2% error, nearly 50% within 5% error and 90% within 10% error. The last
figure is being the same for both manual and expert system valuations.
Given that the valuation for the expert system would include only a cursory inspection of the subject
property, and no inspection of comparable sales, it is expected that it could be delivered at a lower cost and
over a shorter time frame. However valuers didn’t accept that these advantages would warrant any
decrease in accuracy.
Accuracy in Automated and Intelligent Systems
The accuracy levels of current automated and intelligent systems are unclear. While some papers such as
Eckert et al (1993) and Detweiler et al (1996) report selected results and some coefficient of variation
figures, it is unclear how these systems rate using the type of accuracy estimates mentioned above.
Independent accuracy research does not seem to be performed on any of the commercial systems.
Various papers have compared neural network results to MRA results in research that could be considered
to give an indication of the accuracy level that might be expected. The studies by Do and Grudnitski (1992),
McCluskey et al (1996b), Lenk et al (1997) and Rossini (1997b) all provide some clue as to expected
accuracy levels. The indicated levels are shown in Table 4.
Table 4 - Accuracy Levels Quoted from Literature

Percentage of properties that fall into each Accuracy Range


Do & Grudnitski (1992)
McCluskey et al (1996b) Lenk et al (1997)
Rossini (1997b)
% Absolute Error
MRA Neural
Net MRA Neural
Net
MRA
Neural Net
MRA
Neural Net
<5% 25
47
39
40
31
25
< 10%
53
81
55
60
44
53
< 15%
79
100
78
79
64
83
< 20%
93
100
n/a
n/a
80
89
<100% 100
100
100
97*
100
100
Mean Absolute Error
11.26% 6.90% 1.59%
0.81%
9.70%
10.20% 10.42%
15.27%
* This Figure may simply be the result of a typographical error in the paper

The accuracy estimates from McCluskey (1996b) seem to be unbelievably high although in an earlier paper
(McCluskey, 1996a) quotes a MAE of 7.75% for an NN model based on different data which seems to be in
line with the other results. The figure of .81% error seems to suggest that the valuation accuracy would be
significantly greater than what might be expected to be a normal negation range in a typical transaction.
The work by Do and Grudnitski (1992) and Lenk (1997) show some consistency of results. Each indicates
that around a 10% MAE could be expected and that around 80% of valuations would be expected to fall
within a 15% margin of error. Also only some 50 to 60% of valuations would fall within the magic 10%
margin of error range. The Rossini (1997b) results are less encouraging. While they suggest similar results
at the 10% and 15% margin for error, the average error (particularly for the neural network model) is
significantly greater at over 15%. The results from this study include a number of valuations with extremely
high errors particularly from the neural network model.
These results suggest that if a working system can deliver similar levels of accuracy, that such systems
would be acceptable on the basis of average absolute error but unacceptable in terms of the numbers of
valuations with less than a 10% error.

---

Accuracy Issues for Automated and Artificial Intelligent Residential Valuation Systems – Rossini
Page 9
Conclusions
As a result of this research it would seems reasonable to conclude:
• End users of valuations see an error margin of plus or minus 10% as acceptable in most cases, but that
for basic residential properties they would expect errors to be within +/- 5%. This error range would be
the largest accepted error, not the average.
• Valuers in South Australia would consider that for a typical residential valuation that at least 90% of
valuations should have an error of less than +/- 10% and that an automated or intelligent system would
have to deliver the same degree of accuracy to be viable.
• This expected level of accuracy for manual valuation is reasonable considering that the only known
valuation variation study in Adelaide, showed that for a basic house, 95% of valuations were within a +/-
5% error with a mean absolute error of less than 5%. For a significantly difficult residential valuation,
95% of the values were within +/- 15% with a mean absolute error of less than 10%.
• International studies of valuation accuracy are based mainly on commercial valuations. Although there
is a high degree of variation within the results, a mean absolute error of around 10% is typical. Similarly
(in general) about 70% of valuations are found to fall within the +/- 10% error margin with about 80%
falling within the +/- 15% margin.
• The accuracy level shown in these international studies is all that could probably be expected from a
residential expert system based on the evidence from the literature. Generally the systems would
expect to deliver a mean absolute error in the vicinity of 10% with around 80% of valuations within +/-
10% error
• Government assessment for residential properties in South Australia are a little less accurate than
manual valuations with a mean absolute error of nearly 10% but with only 63% of properties within the
+/- 10% error margin and some 83% within the +/- 15% margin.
• Most valuers know little about automated or intelligent systems, but when shown how they work most
would see their introduction as inevitable, if regrettable. A reasonable number of valuers would see the
introduction of suitable systems as being positive.
On the basis of this research it would seem that for an automated or intelligent residential valuation system
to be viable and acceptable in South Australia that it would have to perform to the following accuracy levels.
• A mean absolute error below 10% across all housing with an error of less than 5% for relatively
homogenous suburban areas.
• At least 90% of valuations to be within +/- 10% of sale price (on an accuracy test)
• Within relatively homogenous suburban areas, at least 95% of valuations to be within +/- 10% of sale
price (on an accuracy test)
A system that could produce these accuracy levels would be expected to be viable and acceptable in the
market place for residential valuations.

---

Accuracy Issues for Automated and Artificial Intelligent Residential Valuation Systems – Rossini
Page 10
References
Baen, J. & Guttery R. (1997) The Present and Potential Effects of Technology on the Property Professions in America,
presented at the 3rd Pacific Rim Real Estate Society Conference, Massey University, Jan 1997
Boronico, J. & Moliver, D. (1997) Appraisal Reliability and the Sales Comparison Approach. The Appraisal Journal,
October 1997
Borst, R. (1991) Artificial Neural Networks: The Next Modelling/Calibration Technology for the Assessment Community?
Property Tax Journal, IAAO, 10(1): 69-94
Borst, R. (1995) Artificial neural networks in mass appraisal, Journal of Property Tax Assessment & Administration, 1(2):
5-15
Borst, R. and McCluskey, W. (1996) The Role of Artificial Neural Networks in the Mass Appraisal of Real Estate, paper
presented to the Third European Real Estate Society Conference, Belfast, June 26-28
Brown, G (1992) Valuation accuracy: developing the economic issues. Journal of Property Research, 1992, 9, 199-207
Brown, G. Matysiak, G. and Shepherd, M. (1998) Valuation uncertainty and the Mallison Report, Journal of Property
Research, 1998, 15, 1-13
Crosby, N., Lavers, A, Murdoch J. (1998a), Property Valuation Variation and the “Margin of Error”, Proceeding of the 4th
Pacific Rim Real Estate Society Conference, Perth Australia, 1998
Crosby, N., Lavers, A, Murdoch J. (1998b) Margins Of Error In Property Valuation Negligence In Australia And The UK:
Should Valuers Be Judged By Their Results?, Australian Land Economics Review, Vol. 4, No.1, 1988
Daniels, M. (1984) Test of Variance in Valuation Figures Placed on Individual Properties by Different Valuers
Unpublished Student Project, South Australian Institute of Technology, Department of Property Resource
Management, School of Accountancy
Detweiler J. and Radigan R. (1996) Computer-Assisted Real Estate Appraisal: A Tool for the Practising Appraiser, The
Appraisal Journal, January 1996, 91-101
DeVries, B, Miles, M. & Wolgin, S. (1992) Prices and Appraisals: Where is the Truth, Real Estate Issues, Fall / Winter
1992.
Do, A. and Grudnitski, G. (1992), A Neural Network Approach to Residential Property Appraisal, The Real Estate
Appraiser, Dec. 1992:38-45
Eckert J., O’Connor P. and Chamberlain C (1993) “Computer-Assisted Real Estate Appraisal: A California Savings and
Loan Case Study” The Appraisal Journal, October 1993, 524-532
Evans, A. James, H. And Collins, A. (1993), Artificial Neural Networks: an Application to Residential Valuation in the UK,
Journal of Property Valuation & Investment: 11:195-204
Goolsby, W. (1997) Assessment Error in the Valuation of Owner-Occupied Housing, Journal of Real Estate Research,
Vol. 13, No 1, 1997
Hager, D. & Lord, D. (1985) The Property Market, Property Valuations and Property Performance Measurement. Institute
of Actuaries, London, UK
Hutchison, N. (1996) How do valuers vary in their valuations of commercial property? RICS Research Findings No 2.
RICS
James, H. And Lam, E, (1996) The Reliability of Artificial Neural Networks for Property Data Analysis, paper presented to
the Third European Real Estate Society Conference, Belfast, June 26-28
Jensen, D. (1984) Alternative Modeling Techniques in Computer-Assisted Mass Appraisal, Appraisal Journal
Jensen, D. (1990) Artificial Intelligence in Computer-Assisted Mass Appraisal, Property Tax Journal, Vol. 9, 5-26
Kershaw, P. (1997) Demonstrating Work on the School’s Expert Valuation System, Working Links Seminar September
1997, University of South Australia
Kershaw, P.J., Rossini, P.A. & Kooymans, R.R (1996) “Developing Specialised Software for Real Estate Applications A
Case Study of the Development of UPmarket” Pacific Rim Real Estate Society Conference, Brisbane, 1996
Kershaw, P.J., Rossini, P.A. and Kooymans, R.R. (1994) UPmarket™ Software package REI South Australia
Lenk, M., Worzala, E., and Silva, (1997) High-tech valuation: should artificial neural networks bypass the human valuer?,
Journal of Property Valuation and Investment, Vol 15, No1, 1997, 8-26
Lizieri, C. & Venmore-Rowland, P. (1991) Valuation accuracy: a contribution to the debate. Journal of Property Research,
1991, 8, 115-122

---

Document Outline

• ÿ
• ÿ
• ÿ
• ÿ
  • • ÿ
    • ÿ
    • ÿ
• ÿ

---