Technological Forecasting & Social Change 73 (2006) 966 – 980

Innovations in meta-analysis and social impact analysis relevant

for tech mining
Henk A. Becker a,⁎, Karin Sanders b
a
Utrecht Center for Applied Sociology, Woestduinlaan 65, 3941 XC Doorn, The Netherlands,
retired from Utrecht University, The Netherlands
b
Cubicus Building, P.O. Box 217, University of Twente, 7500 Enschede, The Netherlands
Received 22 September 2005; received in revised form 17 October 2005; accepted 11 January 2006

Abstract

In the social sciences, non-utilization of knowledge is a major problem. Many publications stored in libraries or
available on the Internet should be used more than they are now. Conventional approaches like providing abstracts
and lists of keywords have proven to be insufficient. For more than thirty years already, meta-analysis is available
for the accumulation and dissemination of scientific knowledge. In the social sciences, meta-analysis has been used
on a limited scale only, mainly because there still remains a gap between the knowledge available and its
application in policymaking. Recently, value transfer has been introduced as an additional method to bridge the
gap between available knowledge and the demands for knowledge in new problem areas. Not only in the social
sciences but also in the information sciences non-utilization of information is a major problem. It is the mission of
tech mining to contribute to a mitigation of this non-utilization. In this article, we will show how tech mining could
profit from innovations in meta-analysis and social impact assessment. Special attention will be paid to research on
technology generations, research on social change in cohesive social systems showing solidarity at work, and tech
mining in support of the Lisbon Strategy of the European Commission.
© 2006 Elsevier Inc. All rights reserved.

Keywords: Extended meta-analysis; Information extraction; Data mining; Data warehouse; Meta-analysis; Non-utilization of
knowledge; Scenario-to-strategy workshop; Social impact assessment; Solidarity at work; Technology generations; Think tanks;
Value transfer

⁎ Corresponding author.
E-mail address: h.becker@hetnet.nl (H.A. Becker).

0040-1625/$ - see front matter © 2006 Elsevier Inc. All rights reserved.
doi:10.1016/j.techfore.2006.01.008
 H.A. Becker, K. Sanders / Technological Forecasting & Social Change 73 (2006) 966–980 967

1. Introduction: mitigating the non-utilization of social science knowledge

In 1976, Professor Sir Karl Popper was awarded the Lippincott Award of the American Political
Science Association for his book The Open Society and Its Enemies, [first published in 1945]. This award
is given for the best book in political theory or philosophy, of exceptional quality, that is still considered
significant after a time span of at least fifteen years since the original date of publication. The motivation
for giving this award should make all less brilliant scientists very modest, because practically all their
books will be forgotten after fifteen years or even less. We also have to take into consideration that as a
rule articles in scientific journals are not cited any more after about five years. On a limited scale only they
get summarized in some sort of consolidation papers or books.
In many cases, it is not to be regretted if publications are forgotten because their relevance has faded
away. There are a lot of publications that should have an impact for a much longer period than they
have nowadays, however. Now we are talking about books and articles that get lost in the quickly
growing stream of publications. As an illustration of the size of the problem we take the thirty million
books and eighty million articles and brochures stored in the Library of Congress in Washington.
Sophisticated data analysis mining methodology is required to curb non-utilization in data sets of this
kind.
In general, authors of publications try to facilitate the diffusion of their ideas by providing abstracts,
keywords and full papers on the Internet, for instance. In most fields of expertise, overviews of the
literature are produced. Not only on the supply side but also on the demand side attempts are made to
bridge the gap. Potential readers apply information extraction and similar types of data mining to gather
the information they are looking for. Data warehouses are used to store the information in a systematic
way. Nevertheless, too much relevant knowledge in the behavioural sciences, social sciences, health
sciences etc. still remains unutilized. This is to be regretted, because much research is duplicated in an
inefficient way and much research funds are squandered.
Since a number of years, new approaches have been introduced to bridge the gap between producers
and consumers of knowledge. Firstly, meta-analysis has become available to facilitate the synthesis and
dissemination of scientific information. Secondly, as we will show later, value transfer facilitates putting
knowledge to use in other areas than in which it has been produced. Thirdly, impact assessment provides
methodology to integrate scientific knowledge into policy formation and governance, in the public and
private sector. Each of these approaches raises concern about reliability and precision, however. We will
call the combination of the three methodologies ‘extended meta-analysis’.
In this article, we will focus on recent advances in extended meta-analysis in the behavioural and social
sciences. Following this, we will explore opportunities to apply extended meta-analysis in tech mining.
Porter and Cunningham define tech mining as: ‘the application of text mining tools to science and
technology information, informed by understanding of technological innovation processes’ [1]. Porter
and Cunningham distinguished two basic models of the mining. First, the ‘chain link’ models, looking at
flows between technology and the market place. This first class of models is basically singular in nature,
because the models look at one organization generating new technology and taking it to market. Second,
the class of the ‘policy network’ models. This second class of models recognizes the interplay among
institutions in generating and acting on science and technology.
This second class acknowledges that institutions exist in a framework of competitive and collaborative
relationships. In both classes of models, individual and collective social actors can be distinguished.
Information about these actors and their activities has been stored in data archives on a large scale already
968 H.A. Becker, K. Sanders / Technological Forecasting & Social Change 73 (2006) 966–980

and these data sets are enlarged continuously. The second class is vital in respect to the application of
extended meta-analysis in tech mining.
In this article, we will try to shed light on three questions. First: what are the main characteristics of
extended meta-analysis, consisting of meta-analysis, value transfer, and social impact assessment, in the
behavioural and social sciences? Second: how can the reliability and precision of these approaches be
assessed and further enhanced? Third: what can these approaches contribute to tech mining?

2. Meta-analysis

According to Bal and Nijkamp, meta-analysis can be defined as: ‘a scientific investigation of a well-
defined set of previously published individual studies concerning a certain subject, with the aim to apply
qualitative as well as quantitative review techniques in order to make an integrated cross-sectional or
comparative study of the available material, leading to more insights on the chosen subject of research and
ending up with a systematic synthesis’ [2–4].
The emergence of meta-analysis can be traced back to a heated debate in clinical psychology [5]. In
1952, Hans Eysenck argued that psychotherapy had no beneficial effect on patients. By the mid-1970s,
hundreds of studies of psychotherapy were available and they showed a wide array of positive, null and
negative results. Reviews of those studies had failed to solve the contradictions. To evaluate Eysenck's
claim, Gene V. Glass statistically standardized and averaged treatment-control differences for 375
psychotherapy studies. He called his method ‘meta-analysis’. He and his colleague, Mary Lee Smith,
concluded that psychotherapy was indeed effective. Eysenck was unconvinced and attempted to discredit
the method by calling it in 1978 an exercise in mega-silliness. Despite the criticism of Eysenck and other
scholars, meta-analysis became widely accepted as a method for summarizing the results of empirical
studies within the behavioural, social, and health sciences, however. Since the pioneering work in the
1970s, literally thousands of meta-analyses have been conducted and great improvements have been made
in meta-analysis methodology [5].
According to Lipsey and Wilson, there are basically four reasons for using meta-analysis to analyse and
summarize a body of research studies rather than applying conventional research reviewing techniques [5].
First, meta-analysis procedures enhance the discipline in summarizing research findings. Also, meta-
analysis requires that each step be documented and open to scrutiny. Furthermore, it leads to specification
of the criteria that define the population of study findings to be analysed. It requires search strategies that
assist to identify and retrieve eligible studies, formal coding of study characteristics and findings. Finally,
it also requires data analysis to support the conclusions that are drawn [5].
Second, meta-analysis represents study findings in a more differentiated and sophisticated way than
conventional review procedures that only supply qualitative summaries. ‘By encoding the magnitude and
direction of each relevant statistical relationship in a collection of studies, meta-analysis effect sizes
constitute a variable sensitive to different strengths across studies’ [5].
Third, meta-analysis is capable of finding effects or relationships that are not made explicit in other
approaches to summarizing research. Meaningful effects and relationships upon which studies agree, and
differentiated effects related to study differences, are more likely to be discovered by meta-analysis than
by less systematic and analytic approaches [5].
Fourth, meta-analysis provides an organized way of handling information from a large number of study
findings. A meta-analysis conducted by Lipsey in 1992 for instance resulted in a database of more than
 H.A. Becker, K. Sanders / Technological Forecasting & Social Change 73 (2006) 966–980 969

150 items of information for each of nearly 500 studies [5]. Meta-analysis does not require a large number
of studies, however. In some circumstances, it can be usefully applied to not more than two or three
studies.
In the characterization of meta-analysis by Lipsey and Wilson, to which our overview of the method
owes a lot, the authors emphasize that there is no reason in principle why both quantitative and qualitative
reviews cannot be done on the same body of research findings, with overall conclusions drawn from both.
We find this argument also in the definition of meta-analysis by Bal and Nijkamp cited at the beginning of
this section.
The key to meta-analysis is defining an ‘effect size statistic’ capable of representing the quantitative
findings of a set of research studies in a standardized form that permits meaningful numerical comparisons
and analysis across the studies [5,6]. Once a set of quantitative research findings representing a specified
relationship has been identified, those findings will have to be encoded into values of an appropriate effect
size statistic and carried forward for analysis. Which effect size would be appropriate? This depends upon
the nature of the research finding, the statistical forms in which they are reported, and the hypotheses
being tested by the meta-analyst [5].
Another key concept in meta-analysis is the ‘moderator variable’ [6]. A moderator variable
moderates or alters the magnitude of a relationship. Moderator variables can be applied to determine
the factors that were associated with variation in the magnitude of the relationships between two
variables. As an example, Rosenthal [6] takes a research program on experimenter effects. Eight studies
had been summarized in each of which the performance of experimenters at a given task could be
correlated with the average performance of those experimenters' subjects on the same task. Rosenthal
was interested in learning the degree to which these correlations changed from the earlier to the later
studies. Rosenthal found a significant and substantial (r = .81) effect of ‘when’ a study was done:
studies conducted earlier obtained significantly more positive correlations (relative to later-conducted
studies) while later-conducted studies obtained more negative correlations (relative to earlier-conducted
studies) [6].
Meta-analysis can be used in ‘theory formation.’ In their paper on ‘A New Scientific Challenge in
Economics: Theory Building via Synthesis’, Bal and Nijkamp [3] advocate the use of meta-analysis as a
tool for more accurate theory building in the field of economics. They conclude: ‘ In fact, by means of
meta-analysis inconsistencies among analytical research findings can be explained and the impact of the
study conditions on the research findings can be analysed, by a quantitative analysis’ [3,7].
A major problem in meta-analysis is the identification of relevant studies in large data sets. Many
methods are available for selecting studies suited for inclusion of the meta-analysis. One of them is
‘information extraction’. [8–11]. Another approach is ‘computer-based qualitative data analysis’ [12].
Both approaches consist of scanning text by applying keywords.
Meta-analysis can be seen as a form of survey research in which research reports, rather than people,
are surveyed. A coding form [survey protocol] is developed, a sample or population of research reports is
gathered, and each study is ‘interviewed’ by one or two coders who read it carefully and code the
appropriate information about its characteristics and quantitative findings. The resulting data are then
analysed using special adaptations of conventional statistical techniques to investigate and describe the
pattern of findings in the selected set of studies. Examples of coding forms and SPSS programs for meta-
analysis are available, as Lipsey and Wilson demonstrate [5].
As an example of meta-analysis relevant for tech mining we now will summarize a multi-country study
of the agricultural sector by Nijkamp and Vindigni [13]. The project aimed at an explanation of
970 H.A. Becker, K. Sanders / Technological Forecasting & Social Change 73 (2006) 966–980

productivity differences in OECD countries. The empirical basis consists of 50 case studies. The
independent variables included: changes in efficiency and effectiveness of agricultural policies. As
independent variables, among others country, sector coverage, method applied in the meta-analysis and
the starting point in time has served. The problem to be dealt with was first a reduction of knowledge;
second a check of the consistency of this knowledge. Next, agricultural policies were evaluated with
regard to their efficiency and effectiveness. The final step in the data analysis was the identification of a
classification algorithm, which permits to make conditional predictions (‘if then’) with regard to the
outcomes of agricultural policies.
Our second example deals with income attainment during transformation processes. It focuses on a
meta-analysis of the market transition theory. The changing communist regime in China and the
‘velvet’ revolutions in Central and Eastern Europe [CEE] have provided an opportunity to investigate
trends in income attainment during processes of transformation. In 1989, the market transition theory
was launched. It indicates the main determinants of changing income attainment during the
transformation process from a state-regulated, centrally planned economy to a market economy. An
extensive literature, with studies testing the predictions derived from this theory on China and CEE, has
emerged since this seminal theory was published. However, there are theoretical and empirical
inconsistencies in these studies. This calls for a systematic comparison of the empirical results and
evaluation of the theory. The paper by Verhoeven, Jansen and Dessens [14] reports a meta-analysis
performed on 64 publications to determine to what extent there is consistent empirical evidence for the
hypothesized relationship between marketization processes and the changing effect of income
determinants. Verhoeven, Jansen and Dessens found that political capital remains important during
transformation in urban China, the gender gap in income increases in urban China as well as in CEE
and although human capital and market-related resources are important determinants of income, it is
not evident that their importance increases during the transformation process. Verhoeven, Jansen and
Dessens found some support in favour of the market transition theory, but it needs revision and
elaboration.
As a third example, a current research project by Sanders and Becker is presented. The project focuses
on sustainability with regard to economic, cultural, social and political activities by individual and
collective social actors in work organizations and in urban systems in The Netherlands. The meta-analysis
will be based on 300 survey data sets, permitting an analysis of the developments in The Netherlands and
a comparative meta-analysis covering 40 countries. With regard to urban systems, the relative and
absolute increase of the number of workers in professions requiring creative activities based on
information and communication technology will be researched.
In this research project, a number of innovations in meta-analysis methodology have been
developed and will be tested. Firstly, multi-level meta-analysis will be applied. The higher level of
abstraction will consist of a model of sustainable development in The Netherlands with regard to
economic, cultural, social and political participation of individual and collective social actors. Data
sets available in Dutch data warehouses, covering as a rule three decades, will be used. The lower
level of abstraction will bring meta-analyses in a number of areas covered by a limited number of case
studies.
Secondly, post-diction will be applied. Because the available data covers a relatively long period of
time, the researchers will present post-dictions with regard to the last five years of the time covered by
their data. In this way, the researchers will test the hypothesis that their data permits reliable and precise
predictions, within the limits of their explanatory model.
 H.A. Becker, K. Sanders / Technological Forecasting & Social Change 73 (2006) 966–980 971

Thirdly, the researchers will explore opportunities for narrowing the gap between the ‘suppliers’ and
‘consumers’ of data for meta-analysis. At present, data sets selected for meta-analysis are coded according
to specific procedures [5]. Many of the rules for coding could be applied in advance by ‘suppliers’ of data
for meta-analysis, at the time of publication of the research reports. As examples, the researchers mention
systematic, standardized descriptions of characteristics like type of data (behaviour of individual and/or
collective social actors), periods covered, variables used, statistical tests used to mitigate bias and errors.
Sanders and Becker expect, that researchers will be prepared to provide information on these
characteristics if their contribution would lead to citation in the publications of the ‘consumers’ of their
contributions [15,16].
In many respects already, in the behavioural and social sciences meta-analysis can be carried out on
huge sets of highly reliable and precise data on an almost unlimited scale. It is to be expected, that meta-
analysis will permit large parts of the behavioural and social sciences to acquire a scientific rigour
comparable to that of many natural sciences. Hunter and Schmidt [17] state:
In fact, meta-analysis has even produced evidence that cumulativeness of research findings in the
behavioural sciences is probably as great as that in the physical sciences. We have long assumed that
our research studies are less replicable than those in the physical sciences. Hedges in 1987 used
meta-analysis methods to examine variability of findings across studies in 13 research areas in
particle physics and 13 research areas in psychology. Contrary to common belief, his findings show
that there was as much variability across studies in physics as there was in psychology.
Furthermore, he found that the physical sciences used methods to combine findings across studies
that were ‘essential identical’ to meta-analysis. The research literature in both areas – psychology
and physics – yielded cumulative knowledge when meta-analysis was properly applied. Hedge's
major finding is that the frequency of conflicting research findings is probably no greater in the
behavioural and social sciences than in the physical sciences. The fact that this finding has been so
surprising to many social scientists points up to the fact that we have long overestimated the
consistency of research findings in the physical sciences. In the physical sciences also, no research
question can be answered by a single study, and physical scientists must use meta-analysis to make
sense of their research literature, just as we do (and, as noted earlier, the physical sciences do not
use significance tests).

3. Value transfer

Bal and Nijkamp [18] define value transfer as: ‘a scientific analysis of a subject under study, which
aims to use cumulated knowledge generated via previously undertaken similar types of research
endeavours in order to draw inferences on hitherto unexplored cases’. It serves to meet the formulated
study objectives of a repeated study against the least possible research cost. A survey and meta-analysis
can be considered as complementary to each other [19].
Value transfer, also called benefit transfer, leads to the acquisition of the required scientific insights at
low costs due to the transfer of relevant information from previously undertaken studies to a new similar
type of case study. It has in the past become common practice in the social sciences that even for almost
similar problems new studies are initiated in order to derive the insights needed. All such new projects
require often substantial research budgets. Especially for large research projects these costs are often
972 H.A. Becker, K. Sanders / Technological Forecasting & Social Change 73 (2006) 966–980

prohibitive. Consequently it seems rational to investigate whether the application and transfer of
previously undertaken studies – when necessary, with slight changes – to a new, similar subject of study is
– in economic terms – worthwhile [18].
The name ‘value transfer’ stems from the attempt of transferring (parameter-) values from one case
study toward another. We will call the site that serves as the origin of the estimated values the ‘source
system’ and the site that will benefit from the transfer the ‘target system’. Both systems are also known
under the names ‘study site’ and ‘policy site’. We will avoid these two terms because in discipline oriented
research there is no policy side.
Basically, three modes of transfer can be distinguished. First of all, collected data and other relevant
information can be transferred. Secondly, the methods used in the study of decision-making process can
be considered for re-use. And thirdly, the institutional or procedural framework in which the study or
decision-making process has taken place can act as a guideline for the structure to be used for a new study
process or research procedure [18].
In the second mode of transfer, there exists a form of value transfer in which the type of method is
solely applicable within a particular field of science. This type of transfer can be called ‘intra-disciplinary
value transfer’. In scientific research also a form of ‘inter-disciplinary value transfer’’ can be found [18].
An example of inter-disciplinary value transfers is game theory, which is used in economics, sociology
and evolutionary biology, to name a few disciplines.
As a case, we present ‘Winners and Losers in Spatial Duopoly Markets; the Relevance of a Value
Transfer Approach’, by Bal and Nijkamp [18]. The effect of economic behaviour of different actors on the
size of a market area has been studied in regional economics many times already. This study focuses on
the key forces determining the size of a market area and its implications and possibilities for value transfer.
In particular the combination of price competition and transport costs is addressed. The reasons for the
emergence of regional duopoly markets are investigated by using the willingness-to-pay of households. A
modified duopolistic model to analyse the effects of choice behaviour under conditions of congestion is
placed in the context of earlier research on these types of models, while its implications for value transfer
are traced.
Since the 1950s, in various types of simulation studies transfer of values has been practiced, among
others between data on past and future social structures. and social processes [20–22]. Besides similarities
there are also substantial differences with present value transfers. Since the 1970s, the data archives have
been expanded enormously and opportunities for value transfer and are much more favourable than in the
past.
In the project by Sanders and Becker, value transfer takes place from the meta-analyses on social
cohesion as described before. As a target system, sustainable urban systems in The Netherlands will be
taken. They will represent economic, cultural, social and political participation by individual and
collective social actors.
As an innovation, in the project by Sanders and Becker, multi-level value transfer will be applied. The
highest level will be the Dutch urban system. At a lower level of abstraction we will deal with look at local
sub-systems. As an example we mention ‘solidarity’, interpreted as genuine altruism and quasi-altruism
(self-seeking participation in dependency relationships).
A second innovation will consist of using value transfer in the course of theory formation. In the project
by Sanders and Becker, generation theory will be applied to explain and interpret urban social change
[20–24]. The researchers we will concentrate in particular on the move towards the ‘creative class’,
described by Florida [25].
 H.A. Becker, K. Sanders / Technological Forecasting & Social Change 73 (2006) 966–980 973

As a third innovation, in the project by Sanders and Becker value transfer provides an input to a
monitoring system of urban social change and the effects of urban social policy.
Modern urban systems substantially depend on ICT. As a consequence, value transfer in the project by
Sanders and Becker will consist substantially on tech mining.

4. Integration of scientific knowledge into social impact assesment

If the campaign against non-utilization of knowledge would be restricted to meta-analysis and value
transfer, the outcomes of the campaign would be too limited. We have to hunt down non-utilizers not only
among scientists but also among consultants, decision-makers and members of the public in general. Let
us start with a close look at each of the four social roles involved.

4.1. The role of the scientist

Meta-analysis and value transfer are the responsibility of scientists engaged in discipline-oriented or
policy oriented empirical research. Both types of scientists are responsible for enhancing the body of
knowledge of science. In the empirical disciplines they have to come forward with descriptions,
theoretical explanations and interpretations [21].

4.2. The role of the consultant

Consultants are not responsible for enhancing scientific knowledge. They have to apply scientific
knowledge. The ‘application’ of scientific knowledge takes place according to a specific methodology.
Consultants are members of a traditional profession, like the traditional profession of the physicians
[21,26].

4.3. The role of the decision-maker

Decision-makers [also called policymakers] have to use scientific knowledge. The ‘utilization’ of
scientific knowledge by this type of practitioners also takes place according to a specific methodology
[21]. Policymakers are members of a ‘neo-profession’ [21]. They are responsible for ‘management’, also
called the art–science of ‘governance’. Scientific management in a strict sense is impossible, because
governance has to deal with too much ‘systems noise’. Policymakers have to base their activities on ‘least
regret strategies’ [21]. A policymaker designing a least regret strategy ought to have at his or her disposal
the best knowledge science has to offer. If necessary, consultants can act as intermediaries between
scientists and policymakers.

4.4. The role of the citizen in general

In everyday life, citizens are free to use scientific knowledge or to refrain from the utilization of this
kind of knowledge. As soon as they adopt substantial responsibilities like participation in social
movements, they have a moral obligation to take account of what science could do for them. If, for
instance, they decide to attack government for squandering public money, they have a moral obligation
974 H.A. Becker, K. Sanders / Technological Forecasting & Social Change 73 (2006) 966–980

to use data mining to discover violations of rules for legality, efficiency and effectiveness. If they
decide to protect the natural environment, they are morally obliged to use tech mining. If they have
insufficient knowledge about tech mining, they ought to co-operate with scientists or a consultants
[27].

4.5. The methodology for social impact assessment

In this article we are discussing ‘tech mining’, as a component of ‘impact assessment’. Technology
assessment is often combined with social impact assessment. The methodology to be applied in social
impact assessment projects shows two phases [21]:

A. The initial phase in a social impact assessment project

[1] problem analysis and design of a communication strategy;
[2] systems analysis;
[3] baseline analysis;
[4] trend analysis and monitoring design;
[5] project design.
B. The main phase in a social impact assessment project
[1] scenario design;
[2] design of strategies;
[3] assessment of impacts;
[4] ranking of strategies;
[5] mitigation of negative impacts;
[6] reporting;
[7] stimulation of implementation;
[8] auditing and ex-post evaluation.

How does meta-analysis fit into this scheme? In systems analysis and baseline analysis, meta-analysis
can play an important role. How does value transfer fit into this scheme? Now trend analysis could profit
from value transfer, especially as soon as value transfer is incorporated in a monitoring system. As an
example of a monitoring system, we would like to mention monitoring developments in the labour market
of a country. If every six months a survey is launched, the monitoring system requires meta-analysis and
value transfer, applying moderating variables.
The methodology sketched above has often been used in huge, long-term large activities like building
large dams and reservoirs [21]. On the other hand, ‘instant assessment’ and other short versions of the
methodology are frequently used in short-term decision-making, for instance in co-ordination of disaster
relief [28].
In the project by Sanders and Becker, the methodology described above will be applied in three ways.
First, the researchers will study how practitioners like consultants and decision-makers interact with
regard to economic, cultural, social and political participation in urban areas. Do they use scientific
knowledge available to them? In this sub-project social impact assessment methodology will be used as a
frame of reference. This sub-project will be run as an in-basket game. In an in-basket game, participants
have to provide answers to problems described in a large set of letters deposited in an in-tray. Participants
have to answer the letters as quickly as possible.
 H.A. Becker, K. Sanders / Technological Forecasting & Social Change 73 (2006) 966–980 975

Second, the researchers will create a ‘learning environment’’ for consultants and decision-makers in
urban areas and we will stimulate them to use the outcomes of meta-analysis and value transfers,
including the use of the monitoring system. This sub-project will be run as a scenario-to-strategy
workshop [21].
Third, the researchers will create a learning environment for consultants, decision-makers, and citizens
of urban areas. We will use this learning environment in an experiment with action research.

5. Extended meta-analyses in research programs

5.1. Can implies ought

The role of extended meta-analyses in integrated research programs will increase, because these
programs will more and more have access to substantial databases relevant for their research
questions. In these situations, ‘can implies ought’. It would be inefficient not to use these databases.
In these research programs, extended meta-analyses will be applied at the beginning, during and at
the end of the program. If applied research is involved, impact assessment will not always be
required.
Research programs have to become more transparent and more accountable, because of the
requirements of financing agents. This applies in particular to research financed out of the taxpayer's
money. Governments have to organize an open tendering for the projects that constitute the research
program. As an example, we will discuss now a research program launched by a national science
foundation that will have to contribute to policy-oriented and discipline-oriented knowledge. The national
science foundation will be able to finance four projects and it is confronted with forty reactions to its
tender. It is evident that a multi criteria analysis will have to be applied for evaluating and ranking the forty
potential projects. This criteria will have to be used:

[1] contribution to answering the central research question;

[2] adequacy of the research methodology;
[3] availability of data;
[4] contribution to discipline-oriented knowledge;
[5] contribution to policy-oriented knowledge;
[6] quality of the research co-ordination;
[7] quality of the research personnel;
[8] relevance to policymaking;
[9] cost-effectiveness.

It is also evident that it would be difficult, if not impossible, to engage two independent, capable
referees for each potential project. This problem could be solved by organizing two rounds. In the
first round, a selection committee of scientists evaluates and ranks the proposals, selecting four
projects. In the second round, eventually, requests for a revision of the decisions in the first round
will be dealt with. Sometimes, the strongest of the rejected projects in the ranking will be evaluated
again, comparing them with the already selected projects. In the second round, both the national
science foundation and the co-ordinators of the projects to be evaluated again will be invited to
976 H.A. Becker, K. Sanders / Technological Forecasting & Social Change 73 (2006) 966–980

enlist the support of one or two experts. An independent jury will judge the arguments of the parties
involved.
We predict, that in integrated research programs one meta-analysis will be selected in the first round. If
this does not happen, we predict that a meta-analysis will be selected in the second round. The argument
behind these predictions is, that meta-analysis will provide a crucial contribution to the research program
as a whole.
These predictions are based on the following arguments. In the first phase of the research program, the
meta-analysis will summarize relevant knowledge that is already available. The meta-analysis will
provide a frame of reference for the projects that deals with aspects of the central research problem. In the
second phase of the program, the meta-analysis could be used as a basis for a monitoring system,
supporting policymaking. In the third and last phase, the meta-analysis will provide an international
positioning of the outcomes of the research. This chain of arguments also applies [mutatis mutandum] to
an extended meta-analysis.
A number of methodological problems with meta-analysis can be foreseen. First, reliable effect
size data are scarce. Second, policymakers will try to hide data on their activities as soon as they
realize that auditing by enlarged meta-analysis tends to unmask illegal, inefficient and ineffective
actions.

5.2. Reduction of non-utilization of scientific knowledge

Non-utilization of scientific knowledge can be an effect of three factors: ignorance, controversies, and a
‘cultural gap’. We already discussed in brief the factor of ignorance. If millions of items are involved,
researchers need auxiliaries to spot the knowledge that is relevant for them. Meta-analysis is a powerful
tool in the search for already available knowledge.
A second factor is controversies between scientific disciplines and sub-disciplines. Often it is in the
interest of scientists to ignore the work of others, at least in the short run. Research outcomes look more
important if they are presented as completely new. In the long run, however, co-operation is the best
solution [29]. Meta-analysis could ultimately lead to summarizing all relevant knowledge in an area of
scientific research and theorizing. In the ‘auditing society’ we get a high degree of transparency and
accountability. Auditing stimulates the reduction of inefficiency and ineffectiveness.
In the third place, we are confronted with the enigma of the ‘two cultures’ [21]. Scientists,
policymakers and members of the public in general often do not understand each other, at least in the short
run. In the policy sciences, methods have been developed and tested to bridge this type of cultural gaps. In
extended meta-analysis, the third component represents the application of man and man-machine
simulations, ‘scenario-to-strategy workshops’, ‘decision support rooms’ etc. [21].

6. Relevance of extended meta-analysis to tech mining

We have already discussed a number of applications of meta-analysis in tech mining. In this

section, we will elaborate on these applications. We will discuss [a] the effects of discontinuous
macro-change on ICT-related behaviour of individual actors in urban areas, [b] technology assessment
related to this behaviour, [c] ex post evaluation of this TA, and [d] the effects of changes in ICT-
networks on social participation in urban areas. The analysis of these issues could be accomplished by
 H.A. Becker, K. Sanders / Technological Forecasting & Social Change 73 (2006) 966–980 977

using information available in data archives in The Netherlands and a couple of further West-European
countries. ICT means information and communication technology, TA stands for technology
assessment.

6.1. Information extraction

Van Bakel has applied a linguistic approach to automatic information extraction [8]. The central goal of
his project was to create a Chemical Information Extraction System, that is a system for automatic
extraction of chemically relevant information from descriptions (titles and abstracts) of English texts on
chemical processes. He extracted information that has been stored in a knowledge base and can be used in,
for example, an expert system.

6.2. Ex post evaluations

The impact of technological change on the behaviour of individual and collective social actors has
been evaluated a number of times already. As examples we mention research on technology generations
[23,24]. The data archive of the Netherlands Institute for Scientific Information [NIWI] provides data on
cohort membership, gender, place of residence, level of education, values and utilization of ICT-driven
appliances. What have been the effects of stimulating this utilization? The two research projects
involved, commissioned by Philips Design, show the same research design. Independent variables have
been discontinuous social macro-change, cohort membership of individual social actors, and data on the
characteristics of these individuals in representative samples. Dependent variables deal with the
handling of ICT-driven appliances. Both data sets witness to the hypothesis that handling of appliances
learned in the formative period in the life course, that is between the age of ten and the age of twenty-
five, has a lasting impact on the handling of appliances later in life. The two datasets have been
submitted to meta-analysis. In a current project by Becker, value transfer is taking place towards
generational differences in proficiency in the use of the English language, and the handling of
translation technology. The data for this analysis have been provided by the Netherlands Institute for
Scientific Information at Amsterdam. The data deal with cohort membership, gender, place of residence
level of education, proficiency in English and the utilization of ICT-driven appliances. The main
research question reads: what have been the effects of generational differences on the utilization of
English and the ICT-driven appliances? Two kinds of generations have been distinguished. First, cohort
generations, consisting of individuals in cohorts, showing effects of discontinuous macro-change in their
formative period [between the age of ten and the age of twenty-five] on values, attitudes and
expectations, and on life course characteristics. Second, typology generations. A typologies of cohort
generations is a model, providing a quick overview of the typical aspects of each generation. As an
example of a typology of generations we mention ideal types of the silent generation, the protest
generation and the lost generation. Details about research on generations have been published by the
first author of this article elsewhere [24].

6.3. Ex ante evaluations

In the late 1960's environmental impact assessment, technology assessment and social impact
assessment have been institutionalized [30,31]. Information on the outcomes of ex ante evaluations of the
978 H.A. Becker, K. Sanders / Technological Forecasting & Social Change 73 (2006) 966–980

use of ICT-driven appliances is available, for instance because information about the use by the elderly is
relevant to policymakers.

6.4. Ex post evaluations of ex ante evaluations

Ex ante evaluations are a kind of future research. What are the reliability and the precision of these
predictions, futures explorations and future speculations? [21]. The data stored in data warehouses in The
Netherlands and a number of further West-European countries could be used to evaluate the reliability and
precision of ex ante evaluations regarding the use of ICT-driven appliances in urban areas.

6.5. Analyses of complex social networks

For tech mining, analysing information not only on individual behaviour but also on complex systems
is relevant. Modern countries like The Netherlands illustrate ‘The Rise of the Network Society’ [32]. In
countries with a widespread use of ICT-driven appliances, cohorts born in 1946 and later show non-
bourgeois values [33]. In urban areas, ‘The Rise of the Creative Class’ [32] takes place primarily in the
baby boom cohorts now living in urban areas.
In The Netherlands, a research program on urban citizenship has been launched in 2005. The data
archives permit an analysis of information on the use of ICT-driven appliances by citizens of urban areas,
specified with regard to cohort membership, gender, level of education, and values. This constitutes a
meta-analysis. In another research program, cohort membership, gender, level of education and social
cohesion have already been analysed. Sanders and Becker are engaged now will [a] present a meta-
analysis on Social Cohesion, [b] a value transfer from Social Cohesion as a source system towards Urban
Citizenship as a target system, [c] a meta-analysis on Urban Citizenship, and [d] a ‘scenario-to-strategy’
workshop based on this information, inviting policymakers and concerned citizens to try their hand at
strategy formation, social impact assessment and technology assessment. One of the target variables could
be the use of ICT-driven appliances.
Closely related to this project, Becker is engaged in an analysis of ‘hidden resources’ in the European
Union. In this project, data on the Internet deposited by the European Commission are explored, taking
legal requirements like legitimacy, efficiency and effectiveness as a frame of reference. SPSS software
like Clementine and Lexiquest will be applied. The project requires data mining, in particular tech mining.
Hidden resources consist among others of unutilized potentials of human capital. Methods for
multilingual communication are applied, in particular Sociolinguafranca®. This method facilitates the
bridging of language barriers. This project aims at empowering the European Union to meet the targets of
the ‘Lisbon Strategy’, launched in 2000 and revised in 2005. The strategy tries to transform the European
Union into an advanced knowledge society by 2010 or later.

7. Conclusions and discussion

7.1. Conclusions

Our first research question focused on the main characteristics of extended meta-analysis and its
components. We have been able to show, that in many cases adequate databases are already available
 H.A. Becker, K. Sanders / Technological Forecasting & Social Change 73 (2006) 966–980 979

and that they are further improved. We have also been able to demonstrate that for each of the
components sophisticated methodology is available. The integration of the components is still in its
infancy, however.
Our second research question dealt with the improvement of reliability and precision. The available
data permit rigorous ex post evaluations. In-basket simulations could assess strategy formation.
The third research question was related to the contribution of extended meta-analysis in tech mining.
We have been able to present a blueprint for these applications and an example of an application.

7.2. Discussion

As a pioneering example of meta-analysis we presented evidence-based medicine. Since the 1950's,

medical research has applied meta-analysis and contributed to the improvement of meta-analysis
methodology. In the 1990's, economic research has adopted meta-analysis and value transfer, based on
the growth of the data archives. In 2005, substantial areas in the behavioural and social sciences are ready
for extended meta-analysis. Because this type of analysis is able to produce reliable and precise
information at relatively low costs, it has to be applied. In this case, ‘can implies ought’.
Tech mining with regard to information dealing with economic, cultural, social and political
participation is relevant in particular for strategies to strengthen the transformation of countries like the
member-states of the European Union into ‘knowledge societies’ and strong participants in the global
digital development. The European Union has pledged to strengthen its position in the global knowledge
society in the ‘Lisbon Strategy’ launched in 2000.

References

[1] A.L. Porter, S.W. Cunningham, Tech Mining, Exploiting New Technologies for Competitive Advantage, Wiley, London,
2005.
[2] F. Bal, P. Nijkamp, In Search of Valid Results in a Complex Economic Environment: The Potential of Meta-analysis and
Value Transfer, 1997. published on the Internet, see DSpace at VU.
[3] F. Bal, P. Nijkamp, A New Scientific Challenge to Economics: Theory Building via Synthesis, 1998. published on the
Internet, see DSpace at VU.
[4] R. Brouwer, F.A. Spaninks, The validity of environmental benefits transfer: further empirical testing, Environ. Resour.
Econ. 14 (1999) 95–117.
[5] M.W. Lipsey, D.B. Wilson, Practical Meta-Analysis, Sage, London, 2001.
[6] R. Rosenthal, Meta-Analytic Procedures for Social Research, Sage, London, 1991.
[7] H.-P. Blossfeld, G. Prein (Eds.), Rational Choice Theory and Large-Scale Data Analysis, Westview Press, Boulder, 1998.
[8] B.J. van Bakel, A Linguistic Approach to Automatic Information Extraction: Technolinguistics and Information Extraction.
Nijmegen, 1996. doctoral dissertation.
[9] G. Cybenko, D.P. O'Leary, J. Rissanen (Eds.), The Mathematics of Information Coding, Extraction, and Distribution,
Springer, New York, 1999.
[10] R. Grishman, Information extraction: techniques and challenges, in: M.T. Pazienza (Ed.), Information Extraction, A
Multidisciplinary Approach to an Emerging Information Technology, Springer, Berlin, 1997.
[11] M.T. Pazienza (Ed.), Information Extraction in the Web Era: Natural Language Communication for Knowledge Acquisition
and Intelligent Information Agents, Springer, Berlin, 2003.
[12] G. Fitzgerald, Computer-based qualitative data methods, in: H.A. Becker, F. Vanclay (Eds.), Social Impact Assessment,
Edward Elgar, London, 2003.
[13] P. Nijkamp, G. Vindigni, A Multidimensional Comparative Assessment Methodology for Policy Analysis: A Multi-Country
Study of the Agricultural Sector, 1999. published on the Internet, see DSpace at VU.
980 H.A. Becker, K. Sanders / Technological Forecasting & Social Change 73 (2006) 966–980

[14] W.-J. Verhoeven, W. Jansen, J. Dessens, Income attainment during transformation processes, a meta-analysis of the market
transition theory, Eur. Sociol. Rev. 3 (21) (July 2005) 201–226 (available online atwww.est.oxfordjournals.org).
[15] J. Iedema, H.A. Becker, K. Sanders, Transition into independence: a comparison of cohorts born since 1930 in The
Netherlands, Eur. Sociol. Rev. 2 (13) (1997) 117–137.
[16] K. Sanders, H.A. Becker, The Transition from education to work and social independence: a comparison between the
United States, The Netherlands, West Germany, and the United Kingdom, Eur. Sociol. Rev. 2 (10) (1994) 135–154.
[17] J.E. Hunter, F.L. Schmidt, Methods for Meta-Analysis Correcting Error and Bias in Research Findings, Sage, New York,
2004.
[18] F. Bal, P. Nijkamp, Winners and Losers in Spacial Duopoly Markets: the Relevance of a Value Transfer Approach, 1997.
published on the Internet, see DSpace at VU.
[19] E.T. Higgins, L.C. Idson, A.L. Freitas, S. Spiegel, D.C. Molden, Transfer of value from fit, J. Pers. Soc. Psychol. 6 (84)
(2003) 1140–1153.
[20] H.A. Becker, Assessing the impact of introducing new technologies and interventions for the elderly, in: R.L. Kane, J.G.
Evans, D. Macfadyen (Eds.), Improving the Health of Older People: a World View, Oxford University Press, Oxford, 1990.
[21] H.A. Becker, Social Impact Assessment, Routledge, London, 1997.
[22] H.A. Becker, G. Dewulf (Eds.), Terugkijken op Toekomstonderzoek, ISOR, Utrecht, 1989 (in Dutch only).
[23] M. Docampo Rama, Technology Generations: Handling Complex User Interfaces, Shaker, Maastricht, 2001.
[24] A.-G. van Goor, H.A. Becker, Technology Generations in The Netherlands: a Sociological Approach, Shaker, Maastricht,
2000.
[25] R. Florida, The Rise of the Creative Class, and How it's Transforming Work, Leisure, Community and Everyday Life, Basic
Books, New York, 2002.
[26] J.A. Smith, The Idea Brokers: Think Tanks and the New Policy Elite, Free Press, New York, 1991.
[27] L. Halman, R. Luijkx, M. van Zundert (Eds.), Atlas of European Values, Tilburg University, Tilburg, 2005.
[28] H.A. Becker, F. Vanclay (Eds.), International Handbook of Social Impact Assessment, Edgar Elgar, London, 2003.
[29] K. Sanders, It Takes Two to Tango, Twente Technical University, Enschede, 2005.
[30] H.A. Becker, A.L. Porter (Eds.), Methods and Experiences in Impact Assessment, Reidel Publishing Company, Dordrecht,
1986.
[31] H.A. Becker, A.L. Porter (Eds.), Impact Assessment Today, Van Arkel, Utrecht, 1986.
[32] M. Castells, The Rise of the Network Society, Blackwell, Oxford, 1996.
[33] R. Inglehart, Modernization and Postmodernization: Cultural, Economic, and Political Change in 43 Societies, Princeton
University Press, Princeton, 1997.

Henk Becker is an emeritus professor of Sociology, and Methodology of Social Research, at Utrecht University, The
Netherlands. His main interests are general sociology and research methodology, applied sociology and research methodology,
and the state-of-the-art in sociology. He has published in a number of journals, like the European Review of Sociology, Quality
and Quantity, International Studies of Management and Organization, Impact Assessment, European Journal of Operational
Research, and International Journal of Contemporary Sociology.

Karin Sanders is a professor of Work and Organisational Psychology at the University of Twente, the Netherlands. Her main
interests are the determinants and effects of cooperative behaviour among employees, and between employees and supervisors
within organizations; and the formal and informal governance structures [human resource management], which influence
cooperative behaviour. She published on this topic in a number of journals like Journal of Governance and Management, Small
Group Research, and Journal of Managerial Psychology.