Expert Systems with Applications 38 (2011) 53365345

Contents lists available at ScienceDirect

Expert Systems with Applications

journal homepage: www.elsevier.com/locate/eswa

Evaluating probability of default: Intelligent agents in managing a multi-model system

Igor Perko a,, Miro Gradisar b, Samo Bobek a
a b

Faculty of Economics and Business, University of Maribor, Slovenia Faculty of Economics, University of Ljubljana, Slovenia

a r t i c l e

i n f o

a b s t r a c t
For the successful probability of default (PD) evaluation with the application of multiple prediction models two issues should be addressed: the accuracy of the analytic models which decreases over time and the evaluation of results which must be presented in a uniform shape. To deal with these two issues, a multi-agent system (MAS) and knowledge management systems (KMS) based process management system is proposed. The proposed system has two goals: to prevent the PD information quality deterioration by active management of analytical processes and to provide a universal access point allowing the simultaneous use of multiple prediction models. 2010 Elsevier Ltd. All rights reserved.

Keywords: Probability of default Predictive analytics Business intelligence Multi-agent systems Ontologies Knowledge management systems

1. Introduction The PD evaluation is part of business intelligence (BI)1 and customer relation management systems in the nancial institutions. It denes the likelihood that a loan will not be repaid and will fall into default. The use of predictive analytics (PA) in the PD evaluation is discussed in the academic research (Crouhy, Galai, & Mark, 2000; Fuertes & Kalotychou, 2006; Yeh & Lien, 2009). The PD evaluation is particularly interesting since although the PDs denition is consistent, the methods for acquiring it are not specied, therefore a multitude of models and algorithms are used to evaluate the PD. The heterogeneity of PD evaluation systems is examined by Fuertes and Kalotychou (2006), while some models are compared by Crouhy et al. (2000). The most consistent denition of PD and the system requirements for the PD evaluation is delivered by the BASEL II accord (Basel Committee, 2006) and Capital Requirements Directive (European Parliament, 2006). A selected range of requests, important for the structure of the information systems is presented (European Parliament, 2006):  the IS must include all methods, processes, controls, data collecting processes and subordinated information systems leading to credit risk evaluation, ranking credit risk into classes and risk parameter setting;
Corresponding author. Tel.: +386 2 229 0 119.
E-mail address: Igor.perko@uni-mb.si (I. Perko). Abbreviations: (PD) probability of default, (MAS) multi-agent system, (BI) business intelligence, (PA) predictive analytics, (DF) directory facilitator, (BDI) belief desire intention.
1

 if multiple models are used, the customer and exposure ranking rules and process must be transparent for all the models;  the models and processes must be constantly observed and managed to ensure their subsequent performance in the actual exposure structure and business environment;  evaluations of risk parameters should be gathered directly from the model for every obligator or exposure. In this case, the evaluation is used as a credit risk class on a continuous risk scale;  stress testing (evaluating the effects of changes in global attributes) must be regularly executed and checked by controllers. Stress testing involves a model response to business events or business climate changes which could negatively affect the exposure of the bank and an evaluation of the banks capacity to overcome such events. The PA models for the PD evaluation evolve over time and the nal state of the research process in this led is not in sight (Crouhy et al., 2000; Yeh & Lien, 2009). Du (2004) for instance explores structural credit risk models, while Huang and Van Gestel (Huang, Chen, & Wang, 2007; Van Gestel et al., 2006) explore support vector machines algorithms for PD evaluation. While most of these approaches focus on building a single model, capable of PD evaluation, some authors focus on the integration of multiple mod , Rupnik, Bajec, & Krisper, 2007; Oracle, els (Hsieh, 2004; Lavbic 2006). In the nancial institutions the PA models compete with the statistical models and manual PD evaluation, conducted by analytics. The main advantage of the manual PD evaluation is the ability of the evaluator to use informal information, the expertise, intuition, and use all the background information he can access. The advantage of PA and statistical models lays in the evaluation speed and the thoroughness in complying with the model rules.

0957-4174/$ - see front matter 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.eswa.2010.10.023

I. Perko et al. / Expert Systems with Applications 38 (2011) 53365345

5337

We assess that two objectives must be accomplished for the successful use of multiple models in the PD evaluation: PD must be evaluated for all the customers in the portfolio, and the accuracy of the PD evaluation must be maintained over time. The question is how to achieve these objectives. According to our knowledge the literature does not offer an approach where the models used in the evaluation process would be actively managed and aligned according to the changes in the environment. The purpose of this article is therefore to propose such a solution. To optimize evaluation accuracy for all the customers in the portfolio we propose the use of multiple models, including the manual evaluation. Specic models can have better characteristics for some groups of customers furthermore goals of the evaluation can differ between customers. For instance: the manual evaluation can give the most reliable results since the informal data is used in the evaluation process, but it is also the most expensive method. Therefore it is not preferred to use it in the PD evaluation of small or easy predictable customers. When using a system of multiple models, multiple PD evaluations from diverse models are available. To enable the selection of the most appropriate PD evaluation, the results of different models must be comparable. That means that a unied platform for the model results is required. While some claim the use of multiple models in the decision process creates a fuzzy system (Yu, Wang, & Lai, 2009), we believe the multi model system can be dened as a complex deterministic system and solved as one. 1.1. Predictive analytics in business intelligence systems The BI main purpose is to continuously deliver quality business information. The users trust is attained by keeping or even leveraging the information quality over time and delivering full scope of required information, allowing the user to rely to the system (Microstrategy, 2006; Moss & Atre, 2003; Shim et al., 2002). Data stored in BI systems is transformed into information by executing analytical processes. Analytical processes range from simple summarization to complex processes used in PA algorithms. The PA, application strategy usually consists of building a single model, based on available data and algorithms of a single PA tool. The model results are usually applied only once or on limited occurrences (Crouhy et al., 2000). This application strategy is in conict with the BI concepts and hinders the successful use of PA in BI systems. To build or to apply a PA model multiple analytic processes are executed, transforming the input data to the PD information. The PA model execution process typically consists of several steps: data preparation, model building, model testing, and the application of results. These steps consist of several sub-steps. For instance data preparation can involve: outlier treatment, normalization, handling of missing values, discretization and others. The analytic processes involved in implementing a single model are dependent on the tool and algorithm used. The complexity of the analytical processes in the system grows signicantly when instead of a single model multiple models, created by multiple analytic tolls, are used. To enable the use of PA tools for the PD evaluation, a new strategy of rebuilding and reapplying multiple existing evaluation models should be involved. To successfully maintain the PD evaluation accuracy over time, the complex PA analytical processes must be actively managed. To deliver unied results representation from multiple sources, a exible platform, with the high representation and reasoning capabilities should be used. The combined use of intelligent agents and reasoning capable ontologies is proposed to solve these issues. Some research is done in applying the PA in the BI systems by using metadata structures (Bhrammanee & Wuwongse, 2008;

Cheung & Babin, 2006) and intelligent agents (Kishore, Zhang, & et al., 2007). The model presenting the comRamesh, 2006; Lavbic bined use of IA using local KMS is offered by Perko and Bobek (2008a, 2008b), but the unied application results of the PD evaluation still were not presented. 1.2. Intelligent agents and multi-agent systems Intelligent agents possess the capabilities to reason about their environment and actively execute the tasks, required to achieve their goals (Mohammadian, 2004), while the multi-agent systems enable the data and tasks interchange by offering the standard communication platform (Bellifemine, Poggi, & Rimassa, 2001). Many IA models are based on the belief desire intention (BDI) model, developed by Bratman (1987) and enhanced by Rao and Georgeff (1995). They rely on the presumption that the agents architecture and the reasoning capabilities of the agents are based on recognizable behavior patterns and that all the capabilities are stored inside the agent (Rao & Georgeff, 1995). A MAS combines groups of intelligent agents and provides common services as for instance a standard communication layer and board of MAS agents services in a form of directory facilitator (DF), as dened in FIPA (2006a). The IA communication and collaboration using MAS services is described by Bellifemine et al. (2001). MAS standard structure enables IA communication and services sharing inside the agency, but also with agents from different MAS. The agents role in MAS is to offer its services for achieving the goals set to the MAS. The role sharing between IA and MAS is further explained by DInverno, Luck, Georgeff, Kinny, and Wooldridge (2004). For the successful implementation of IA in the BI environment the dynamic IA should be used. The IA reasoning should be upgraded to successfully use the knowledge already stored in the knowledge structures and reasoning tools that already exist in the BI environment. Some attempts of connecting IA with the external KMS are made by creating static agent structures, resembling the KMS from their environment (Novak & Dix, 2006), although such agents lack the ability to adapt to the structural changes in the environment. Others tried to embody the KMS dynamic structure into the agents memory (Kishore et al., 2006; Laclavik, 2006; Perko & Bobek, 2007); although the approach is promising the results of using dynamic IA were not presented in the literature so far. 1.3. Knowledge reasoning The knowledge management R&D community has been active for the last 20 years and has shown great potential in knowledge representation and reasoning technologies. Some of the KM concepts are presented by Alavi and Leidner (2001), while R&D directions are presented by Frappaolo (2006). The usage of KMS is constrained with the capability of storing complex data structures and the reasoning capabilities of their contents. The R&D in KMS led to the design of ontologies in complex standard reasoning capable structures. The ontologies present the implementation of KMS and are capable of storing the widest array of data and providing reasoning capabilities, based on the stored data. One of the promising ontology languages is the OWL-DL (ontology web languages description logic) (Sirin, Parsia, Grau, Kalyanpur, & Katz, 2007). OWL-DL is a universal ontology language enabling machine reasoning based on the data structure and content. The reasoning algorithms, used by reasoning engines as for example Sirin et al. (2007) are optimized to extract knowledge stored in the ontologies. The reasoning and query languages such as semantic web rule language (SWRL) and simple protocol RDF

5338

I. Perko et al. / Expert Systems with Applications 38 (2011) 53365345

query language (SPARQL) (W3C, 2007) represent the state of the art in the reasoning algorithms. By using the ontologies and reasoning algorithms, the standard form of knowledge representation is provided, enabling knowledge extraction and reasoning capabilities by the intelligent agents. The solution resolving the problem of continuously employing multiple prediction models in the PD evaluation using a MAS and reasoning capable ontologies is not presented in the literature so far. Thus the purpose of this paper is to propose a solution for multiple prediction models management and a uniform result representation. The proposed solution provides the structure for accessing to the knowledge about multiple prediction models and the means to execute the analytic processes. It is adaptive, allowing the modications and upgrading of the PD evaluation system. The paper proceeds as follows: in Section 2 the multi-model PD evaluation management system is presented. In Section 3 the PD evaluation results are presented. In the conclusion, the research results are summarized and further research directions are discussed. 2. Managing the PD evaluation processes The PD evaluation management main task is to provide quality PD evaluation results in the dynamic environment, using multiple predictive models. Its main goals are to:  maintain optimal results in time in a dynamic environment,  unify the results, provided by diverse analytical models. A system, capable of maintaining PD information quality and providing unied results is presented in Fig. 1. It is constructed of structures, existing in the BI system, upgraded with a MAS and a integrated common ontology. The unied results representation is achieved by recognizing the analytical models data structure and delivering their contents in a uniform way. To reproduce optimal results in time in a dynamic environment, the active management of the analytical processes is used. To create a PD evaluation in the complete credit portfolio, a system of evaluation models is presumed. In a complex system PD evaluation results are provided by applying evaluation models, produced by predictive analytic tools, legacy analytical tools, manual evaluation and by external evaluation. The models, data, and local metadata are mainly stored in the data warehouse. The

BIMAS-PD task is to manage multiple PD evaluation models, considering changes in the data sources, used models and analytical tools and to provide unied PD evaluation results. 2.1. Data warehouse The data warehouse consists of all structures and processes needed for the PD evaluation: the source data (the problem of acquiring the source data is not the scope of this paper), metadata holding the general data warehouse information and multiple analytic tools. Local analytic tools typically contain analytic models, processes, and local metadata structures. The PD evaluation results are also stored in the data warehouse. PD evaluation tools consist of analytic processes and data. Both are described in form of local metadata storages. A prediction model is an implication of analytic processes on underlying data. The model partial and nal results are stored in data warehouses. Multiple PD evaluation sources are detected:  predictive analytics tools: using data mining and statistical algorithms for providing PD evaluation models. The models are created by learning from existing data, therefore when the underlying data is modied; the same algorithms create modied models. Metadata is stored in local KMS, while source data and PD evaluation data is stored in a data warehouse.  Legacy analytic tools: These tools mostly provide static PD evaluation models. These models are created using expert knowledge. On data changes models structure remains unchanged, only the PD evaluations results are modied. Metadata is stored in local KMS, while source data and PD evaluation data are stored in a data warehouse.  Manual evaluation: If the PD is evaluated by experts, some models may be used. This evaluation technique is the most wasteful on resources, but implies the use of informal data in the evaluation. The existing metadata is stored in local KMS, while PD evaluation data is stored in a data warehouse.  External evaluation: The PD is evaluated by external experts. The metadata used in evaluation is not available; data is not stored in data warehouse. The need for multiple models in PD evaluation is presumed since not all of the customers in the credit portfolio can be evaluated using the same model; the dynamics of the system is presumed since the data values and structure are modied in time, requesting the adaptation of the models in the system. 2.2. Dynamic intelligent agents Dynamic intelligent agent opposed to standard IA posses two levels of reasoning: internal and dynamic reasoning. The internal reasoning is used by the agent to identify its capabilities and to locate the means on how to asses them. The capabilities of internal reasoning are usually hard coded. For instance: if one of the agents capabilities is the query (Q) capability, then the reasoning on connected protocols, the location and the content of the credentials is internal. The dynamic reasoning is used when the execution of the agents capabilities depend on the external KMS structures. For instance the selection of the analytic processes to be executed depends on: the existing data; analytical process information, and execution rules and strategies, stored in BIMAS-PD ontology. The real advantage of dynamic IA is the capability to use external reasoning engines, performing OWL-DL optimized querying in the BIMAS-PD ontology. The intelligent agents manage the PD evaluation processes. All IA share the common MAS services and the BIMAS-PD ontology access, while every agent possesses special capabilities enabling

Fig. 1. The BIMAS-PD structure in PD evaluation.

I. Perko et al. / Expert Systems with Applications 38 (2011) 53365345

5339

them to provide services to other agents and external users by registering these in the agent services directory. By offering special IA services among agents the redundancy of the IA capabilities can be largely reduced. In the Fig. 1 the individual IA capabilities are presented with the capability abbreviation, to represent the common capabilities a dotted arrow line is used. Common IA capabilities are to:  Communicate: all agents are capable of using the advanced FIPA (FIPA, 2006b) communication standards, enabling them to send and receive simple and complex. rdf messages.  Use (agent services): using agent services, stored in agent services directory agents can use services offered by IA in their MAS instance or by agents in other MAS.  Q (BIMAS-PD ontology): query BIMAS-PD ontology enables the IA to access the content of the BIMAS-PD ontology. The following simple capabilities are required: connect, execute query, parse query results. Since all IA in the BIMAS-PD share the common ontology, the volume of the messages can be largely reduced. Instead of complete instruction, a reference in the ontology can be sent to inform the receiving agent. In Fig. 1 six intelligent agents are involved, each possessing special capabilities and offering services:  Applicator takes care of the user interface (UI). It receives the users requests, sends requests to other agents, receives their responses, and reports the results to the user.  UI: user communication using standard user interfaces, needed for sending requests and delivering reports. This capability, used by all IA can provide a unied representation of heterogeneous tools used in BI system.  Process manager manages the execution of the analytic processes. It recognizes the processes required to be executed. The reasoning about BI processes execution is performed using a BIMAS-PD built in reasoner. Requests for the process execution are sent to the execution agents.  Reason (BIMAS-PD ontology): this capability enables the IA to use complex decision strategies and decides upon the content of the common BIMAS PD ontology. The following simple capabilities - in addition to Q (BIMAS-PD ontology) are required: store and retrieve complex query structure. By externally storing query structure the change in agents reasoning can be applied by only modifying the query structure.  PA executor manages the execution of PA analytical processes. It executes the selected processes using external methods, stored in the data warehouse environment. Since models in PA applications can be rebuilt it posses the capability to execute the rebuilding of the used models.  Execute (processes): the Execute capability enables the IA to execute existing processes in an analytic tool. Since processes are bound to data analysis tools, a connect capability is required.  Legacy executor manages the execution of legacy analytical processes. It also posses the Execute (processes) capability. Since the models in legacy analytic tools cannot be rebuilt, when new data emerges, the applying of the existing models to new data can be executed.  Integrator integrates the PD evaluation results stored in multiple sources in data warehouse using the rule set, stored in the BIMAS-PD ontology, and providing unied explainable PD evaluation results for all the customers in the credit portfolio.  Reason (BIMAS-PD ontology)  Base manager manages MAS central ontology by updating the ontology content with the environmental data.

 Q (local metadata): query local structures containing knowledge about BI system is a basic capability required by every active agent. It enables the use of dynamic capabilities and external execution methods.  Q (data sources): query external data sources. Agents can connect to and query external data sources. They dont have the update capability.  Q/U (BIMAS-PD ontology): the query and update (Q/U) BIMAS-PD ontology represents the IA capabilities to connect, advanced query, reasoning, and update the BIMAS-PD ontology content. BIMAS-PD delivers the denition of the required IA capabilities for the successful PD system management. The complexity of the management system is reduced by determining the required capabilities and appointing them to single agents. These basic agents capabilities can be offered as services, although the real power of the MAS services is in creating composite services, based on the sequence of simple IA capabilities. Agents presenting composite services use own capabilities and services offered by other MAS agents or agents from other MAS instances. Lets provide an example of a composite service Manage analytic processes. This service provides optimal data analysis results after a change in data, in metadata or the analytical processes occurs. To create a correct state of the system, all the analytical processes affected by the change should be executed. For instance: if important customer data is added, the customer PD evaluation models are rebuilt, reapplied, tested and the results reintegrated. To offer a service Manage analytic processes, a set of basic capabilities is required: reading data Q (data); reading the corresponding metadata Q (BIMAS-PD ontology); executing the analytical processes Execute (analytical processes); integrating the results Integrate (data); and presenting the data UI (User). The steps are described in the form of available services (1), intelligent agents (2) and required capabilities (3). A set of IA services, used in the complex service Manage analytic processes, is described in 1

RepeatCoordinate BIMAS PD ontology > Select Processes > Execute Processes until all required processes are executed > integrate the PD ev aluation data > send the PD ev aluation report to the user :
A set in 1 of services is executed by agents 2

Repeat Base manager > Process manager > PA executor ^ Legacy executor until all processes are executed > Integrator > Applicator:
To execute the services, agents use the required capabilities 3:

Repeat Q Data ^ Q metadata ^ U BIMAS PD ontology > Q Data ^ Q BIMAS PD ontology > Execute processes until all processes are executed > Q Data ^ Q BIMAS PD ontology > U Data > UI User : 3

The execution of the Manage analytic processes is processed as follows: 1. the data and BIMAS-PD ontology content is updated to represent the current state of the system, 2. the required local analytical processes are executed. The selection and the sequence of the analytical processes to be executed depends on the data content, execution strategy, stored in the

5340

I. Perko et al. / Expert Systems with Applications 38 (2011) 53365345

BIMAS-PD ontology and the available analytical processes denition, stored in the local metadata, if changes were applied, go to step 1, 3. the PD evaluation are integrated according to the rules, set in the BIMAS-PD ontology and delivered to the user. The execution of the analytic processes often results in data changes, triggering the execution of analytic processes, using the modied data as the input, therefore the steps in process is repeated until all the processes in the chain of execution are executed. The Manage analytic processes is executed in cooperation of multiple agents, each contributing its services. The presented design is useable in complex environments, where multiple instances of analytical processes can occur. This is a case when the PD evaluation results are created by multiple models or even tools, and data and metadata are stored in multiple locations. In such cases changes in source data can affect the results of multiple analytical models. The BIMAS-PD involves the concept of dynamic agents using external KMS in the reasoning process and provides a MAS framework, supporting the use of common ontology and communication means. This allows agents to be resistant to changes in data content, structure or the reasoning strategies. The capabilities needed to fulll the MAS goals are distributed among agents, creating a simple hierarchy with the MAS collective means serving as a standard communication platform. This allows the unied and standard representation of up-to-date PD evaluations. 2.3. The knowledge management structures In the data warehouse environment multiple existing local metadata structures, supporting the local analytical tools are presumed: the local metadata structures involve data on the location and availability of data sources, data content and structure information, the status of analytic processes etc. The local metadata structure and content is dedicated to support the single instance of analytical tool, therefore local metadata differ from tool to tool and usually dont apply to common standards. The BIMAS-PD ontology is used to support two IA capabilities needed to operate in dynamic environment:  the dynamic reasoning is enabled by partially exporting the internal IA reasoning process to the BIMAS-PD ontology structures. The detail IA structure, enabling external reasoning is elaborated in Perko and Bobek (2008b);  the communication support between the IA is enabled by combination of FIPA (2006a) communication and direct access to the common ontology by all the MAS agents. This allows agents to communicates using short messages, combining the received information by reasoning the common ontology. The BIMAS-PD ontology presented in Fig. 2 complies to OWL-DL (W3C, 2006) standards. It consists of classes and subclasses, containing properties and asserted conditions for expressing knowledge about the available tools, PA functions, algorithms, and current status information. It is database stored, using indexing services to speed up the reasoning process. The BIMAS-PD contains hierarchical classes with asserted conditional structure.2 The main class PDEvaluation has several subclasses:

 Functions holds the data about 4 available PA functions. The individual function properties are stored in 4 subclasses.  Algorithms presents the properties of 11 available algorithms in the PD evaluation process. Algorithms and its subclasses are inverse functionally constrained with Functions and its subclasses using supportsAlgorithm and usesFunction constraints. For instance, function Classication can use 5 out of 11 algorithms.  ExplainPower denes the algorithm capacity to explain the results. Algorithms and ExplainPower are related using a hasExplainPower constraint.  Tools has four subclasses: these hold information about the tools used in the data mining process and are related within Algorithms using the usesAlgorithm constraint.  BI_PROCESSES contains all the information on data structures and processes involved in the active instance of PD evaluation. Structures and processes are stored in the form of triplets: data sources, processes, and data results.  BI_V_MODELS contains information about the implemented models; the four subclasses elaborate in detail the data structure, models tests results and applied data. These classes are particularly important since they contain the information for all the models used in the PD evaluation and allow direct comparison of the models results. Algorithms and Tools serve are related to BI_V_MODELS via the isCreatedBy and usesProcess constraints.  IntegrationRules contains rules used in the models results implementation. Some of these rules are xed, other are dependable on the model testing results or the PD results. The BIMAS-PD ontology is actively reasoned by the agents in two occurrences:  The PA processes execution selection: the selection strategy is based on a simple set of rules to determine the analytic processes to be executed: a process is to be executed after one of its data sources is changed or when the analytical process is altered. The process modies one or more results. Because of the sequential predictive analytics processes, some results of previous processes are the sources for the succeeding processes.  The selection of the appropriate PD evaluation result: the selection rules consist of xed rules, model test results, and a customer PD evaluation results. For every customer the PD evaluation is scored. The score is a result of xed rules, model test results, and PD evaluation results. The PD evaluations with the highest score are presented to the user. In the presented example basic reasoning strategies are used, although to change the reasoning strategy, only a query needs to be modied, while the structure of the reasoning agents remains unmodied. The BIMAS-PD provides a consistent environment containing the information means needed for the successful management of heterogeneous analytic processes and their results integration; thereby the level of information quality can be set on a new level. By raising the information quality on acceptable level the successful inclusion of predictive analytics in BI systems is enabled. The BIMAS-PD concepts enable the successful process management in dynamic heterogeneous BI systems, such as the PD evaluation:  a limited list of agent capabilities, required to complete their tasks and their unied representation;  an abstract notion of processes and structures;  encapsulation of complexity in the heterogeneous system and its sub-systems;

2 For additional reference on asserted and inferred classes and individuals see Sirin et al. (2007).

I. Perko et al. / Expert Systems with Applications 38 (2011) 53365345

5341

Fig. 2. The BIMAS-PD ontology class schema.

 a single platform for the management of multiple subsystems;  unied communication standards in complex systems. The analytical processes and structures used for the PD evaluation are only a part of the extensive BI environment. The future research could be focused in using multiple instances of BIMAS structures, providing a number of services to manage the complex BI environment. 3. The results The BIMAS-PD success is evaluated on maintaining the information quality level represented to the user. Its task is to react to the changes in source data, user requests and predictive models. All required PA processes should be executed and the results should be integrated, minimizing the effects of changes on information quality decrease. The BIMAS-PD is put in a realistic controlled environment, evaluating the PD of commercial bank customers. The data includes a time sequence of 17 half year data for a portfolio of 3482 bank business customers (in steps 3a and 3b the number of customers increases to 3763). The evaluation data includes 880 numeric and text attributes, explaining the nancial and business condition and the industry situation. As presented in Table 1 for the PD evaluation models are built using diverse data sources and classication algorithms. Model 3 represents manually evaluated customers, all other models are created by Oracle Data Miner (ODM). In models 1, 2 and 4 all the customers in the portfolio are evaluated, in model 3 only part of the customers is evaluated, while using models 5 and 6 the portfolio is divided to the well described and customers missing some attributes. The customers with high quality input data are evaluated using model 6, while others are evaluated using model 5. The experiment starts with an existing system of four PA models for the PD evaluation models 14 in Table 1. The dynamics of

the system is simulated implying changes to the system in a series of 3 steps. In each step the evaluation system is modied, and then the results are observed: to evaluate the process management the test results for the involved PD evaluation models and the modications in PD evaluation are examined; to evaluate the success in data integration, the dynamics in number of customers evaluated using involved models and the integrated PD evaluation results are examined. Using the experiment, the success in PA processes management and data integration is evaluated. The expected results should prove the robustness of the system to the implied changes and the ability to provide unied PD evaluation results. Step 1: two new models5 and 6are added. Model 5 is used to evaluate the customers with low quality input data, while with model 6 the customers with high quality input data are evaluated. The two models results are expected to be included in the resulting dataset and not to interfere with the existing models results. Step 2: changes in the models are applied to the models 2 and 4. The changes in the models require rebuilding a model and alter the models results. We expect that the rebuilding of the model does not interfere with the other models results. Step 3: the data content on which the models are built is modied. The new data are added, allowing the PD evaluation, based on data on December 31, 2008. Major change in data requires the rebuilding of all PA models. To evaluate the effect of the active management, the step 3a is introduced. In step 3a the BIMAS-PD is hindered to rebuild the models, instead the existing models are applied on new data. In step 3b the BIMAS-PD is allowed to process all the models. We expect a sharp decrease in the model test result in step 3a and a mayor modication in the unication process. In step 3b better

Table 1 Models used in the experiment. Model ID Model Model Model Model Model Model 1 2 3 4 5 6 Tool ODM ODM Manual ODM ODM ODM Algorithm SUPPORT_VECTOR_MACHINES SUPPORT_VECTOR_MACHINES MANUAL DECISION_TREES SUPPORT_VECTOR_MACHINES SUPPORT_VECTOR_MACHINES Data source All customers All customers Evaluated customers All customers customers with low quality input data customers with high quality input data No. attributes 880 50 All 50 880 880

5342 Table 2 Model test results after each step. Step 0 1 2 3a 3b Values OVERALL ACCURACY AREA UNDER ROC OVERALL ACCURACY AREA UNDER ROC OVERALL ACCURACY AREA UNDER ROC OVERALL ACCURACY AREA UNDER ROC OVERALL ACCURACY AREA UNDER ROC

I. Perko et al. / Expert Systems with Applications 38 (2011) 53365345

Model 1 96.616% 98.218% 96.616% 98.218% 96.616% 98.218% 88.580% 97.505% 98.213% 99.278%

Model 2 96.472% 97.093% 96.472% 97.093% 96.616% 96.650% 97.319% 97.505% 98.014% 98.163%

Model 3 91.954% 91.954% 91.954% 90.866% 90.866%

Model 4 96.400% 96.428% 96.400% 96.428% 96.328% 95.205% 98.113% 96.279% 98.361% 97.539%

Model 5

Model 6

95.402% 98.442% 95.402% 98.442% 83.412% 90.277% 97.630% 98.642%

96.774 98.464 96.774% 98.464% 96.339% 97.475% 98.447% 99.228%

test results for the rebuilt models and an increase in number of customers evaluated using these models are expected. In Table 2 the overall accuracy3 and area under ROC4 curve are presented for the models involved in the experiment. All the models were successful in providing results after every step and none of the changes seriously affected the models results:  Step 0: the overall accuracy in all models is above 90%, The most accurate is the model 1, with 96.6% accuracy, the lowest results are in the model 3, 91.9%. The area under ROC for all models is also above 90%, where the measure for the manual model 3 could not be obtained.  Step 1: all models test results remain unchanged, with 2 new models being added to the models test class. As expected the model 5 overall accuracy is slightly lower than the average of other ODM models, while the model 6 test results are better than the result gained by other models.  Step 2: the model 2 and model 4 test results are almost unaffected by the change (under 0.2%), while other models test results remain unaffected. Again, only processes in modied models are executed. The relatively small change in the models test results displays the resistance to changes in the modied models.  Step 3a: changes in source data, result in the considerable decline of test results for some models, the decline is expected, as existing models are built on old data set. Model 1 overall accuracy declines over 8% to 88.58%, while the model 5 overall accuracy declines to 83.412%. Other models are not that affected by the change; model 2 and model 4 test results are even better than before step 3a. The processes of applying and testing are, as expected, executed for all the models. The interesting part is the raising of the model test results for models 2 and 4.  Step 3b: the rebuilding of the models produces positive results: All the models tests results outperform the test results, acquired in step 3a. The models 1 and 5 regain the lost overall accuracy test results. The only exception is model 3 because the manual evaluation type does not let the rebuilding of the model to be performed.
3 Overall Accuracy: the simple accuracy of the model. Maximum Overall Accuracy will provide no costs/weights, allowing the model to achieve a higher overall accuracysometimes at the cost of missing predictions on rare target values (Oracle, 2006). 4 The area under the ROC curve (Receiver Operating Characteristics) measures the discriminating ability of a binary classication model. The larger the area under the curve, the higher the likelihood, that an actual positive case will be assigned a higher probability of being positive than an actual negative case. The area under the curve measure is especially useful for data sets with unbalanced target distribution (one target class dominates the other) (Oracle, 2006).

The models test results demonstrate the BIMAS-PD ability to keep the quality of PD evaluation models. Table 3 represents the PD evaluation results for all the models in the BIMAS-PD after each step. All models are applied on the customer data. To improve the results transparency, customers are distributed in 10 buckets, according to the PD evaluation results.5 The bucket upper limits are described in the top row of Table 3. An example: customers with the PD smaller than 0.3% are placed in the bucket 1, while customers with the PD greater than 50% are in the bucket 10. The model is performing well when the customer PDs are evaluated with a high level of certainty, therefore the buckets 14 and 10 are preferred. Customers evaluated with a low level of certainty are placed in buckets 59. Using models 1, 2 and 4 the whole population of customers is evaluated. Manual model 3 is used to evaluate 1333 customers in steps 02 and only 1062 customers in last two steps. The models 5 and 6 together are also used to evaluate the whole population, although in steps 02 a number of 2448 customers are evaluated using model 6, and residing 1034 are evaluated using model 5. In steps 3a and 3b the number of customers evaluated using model 6 raises to 2.817, while the number of customers evaluated using model 5 declines to 946. This signals the quality improvement in the input data. Model 1 shows uctuation in steps 3a and 3b. In step 3a the number of customers in buckets 7, 8, 9 and 10 increases by 801, while the number of customers in buckets 16 is reduced by 1,082. In step 3b number of customers in buckets 14 increases slightly, with the majority of the customers (1568) placed in bucket 8. Model 2 shows uctuation in steps 2, 3a and 3b. Model 2 is very inconclusive, since number of customers evaluated with the high certainty is minimal in all steps. After step 2, no customers are in baskets 17, while most of the customers are in the basket 8. In step 3b some customers, previously in basket 8 are evaluated to the baskets 9, and baskets 47. The manual evaluation in model 3 is not affected by the changes in the system, although the number of manually evaluated customers decreases in the newly acquired data. Using model 4 customers are evaluated with great certainty, since a great portion of customers resides in buckets 14 and 10 in all steps of the experiment. In step 2 model 4 is altered. As a result, customers previously in buckets 2 and 3 are evaluated with the higher certainty and are placed in bucket 1 (174 changes). In step 3a number of customers in bucket 5 rises, while in step 3b the customers previously in bucket 5 are largely reevaluated in bucket 3.

5 The adopted customer distribution was researched by Krink, Paterlini, and Resti, 2008.

I. Perko et al. / Expert Systems with Applications 38 (2011) 53365345 Table 3 The PD evaluation distribution after each step. Model Step Bucket/PD upper limit 1 0.0% Model 1 0 1 2 3a 3b 0 1 2 3a 3b 0 1 2 3a 3b 0 1 2 3a 3b 1 2 3a 3b 1 2 3a 3b 95 95 81 34 16 16 16 13 13 681 681 855 540 605 78 78 78 43 38 2 0.3% 41 41 41 19 25 1 1 3 0.5% 123 123 123 47 68 2 2 4 1.0% 278 278 278 89 138 12 12 5 2.0% 361 361 361 137 134 16 16 6 3.0% 575 575 575 320 249 53 53 7 5.0% 1103 1,103 1,103 838 760 212 212 8 10.0% 477 477 477 584 1,568 2,722 2,722 3,239 3,375 3,006 67 67 67 62 62 92 92 114 66 9 660 660 418 451 536 536 705 670 9 20.0% 180 180 180 933 314 186 186 13 14 193 107 107 107 80 80 53 53 35 2 28 150 150 366 49 301 301 704 151 10 50.0% 266 266 266 753 469 278 278 230 374 424 10 10 10 11 11 788 788 792 943 795 15 15 68 67 272 272 414 352

5343

Total 3482 3482 3482 3763 3763 3482 3482 3482 3763 3763 l333 1333 1333 1062 1062 3482 3482 3482 3,763 3763 1034 1034 946 946 2448 2448 2817 2817

Model 2

1 41 41 41 27 27 90 90 288 288 288 193 193 149 149 1 1.719 112 112 112 89 89 124 124 160 44 52 2 2 2 6 137 137 64 62 203 203 98 172

5 154 154 154 127 127 1,363 l,363 1,350 1,121 95 9 9 1 15 171 171 95 156

23 308 308 308 280 280 69 69 92 32 33 33 7 58 256 256 150 284

111 230 230 230 180 180 73 73 83 15 261 164 164 84 300 423 423 479 905

Model 3

Model 4

199 1 1

Model 5

Model 6

54 54 27 31

Using model 5 only customers with low quality data are evaluated, therefore, as expected, only a small number of customers is evaluated with a high degree of certainty. Model 6 evaluation results show a relative high certainty. Some customers are placed in buckets 14 and 10. As in previous models step 3a has a negative effect on the PE evaluation certainty, while in step 3b the number of customers in buckets 14 increases. The model test results and PD evaluation distribution prove that the reliability and accuracy of the models is mainly preserved, though to the changes. The two information quality properties: the accuracy and the timeliness are preserved from the changes in the environment. To support the decision process, the most appropriate PD evaluation result should be delivered to the user. Because the unied results representation and their explanation are available by the BIMAS-PD, an automatic result selection can be involved. The reasoning in the selection process involves explicit business rules, model type properties, model testing results and PD evaluation results. In Table 4 the number of customers, evaluated by the models in PD evaluation system is presented. For every customer only the most appropriate PD evaluation is selected. In each step changes in the model selection are observed.

Table 4 Model selection in customer PD evaluation after each step. Step Customers Model 0 1 2 3a 3b 960 541 810 432 474 Model 2 1215 674 44 35 104 Model 3 1234 1191 1192 997 998 Model 4 73 48 227 1320 688 Model 5 710 725 334 718 Model 6 318 484 645 781 Total 3482 3482 3482 3763 3763

In the start, model 2 and model 3, used in over 1200 cases, are the most used models, followed by the model 1. In step 1 the number of customers evaluated by existing models declines, due to newly introduced models 5 and 6. In step 2 a decrease of customers evaluated using the model 2 is observed. The model 2 evaluation is selected in only 44 cases, probably due to the changes in the model conguration. In step 3a the number of customers, evaluated using model 4 raises dramatically, while the number of customers, evaluated with models 1 and 5 is reduced. The redeployment is correlated with the decrease in models 1 and 5 testing results. In step 3b the models usage is distributed evenly, using all the models. The ability to select the most appropriate model results in a system where models, that do not produce optimal results are not being used. The typical example is the exclusion of model 2 after second step and exclusion of models 1 and 5 in step 3a and their inclusion in step 3b. In Table 5 the unied customer PD evaluation results, using multiple models are presented. In step 0 for the 59 customers PD evaluation certainty is lower than 0.3%, with the average PD of 0.07%. A total of 698 customers reside in buckets 14, having a PD evaluation certainty under 2%. The majority of customers, 2551, reside in buckets 49, with the PD evaluation between 2% to 50%. 233 customers reside in bucket 10, with the PD evaluation above 50%. The average PD in bucket 10 is 93.99%, implicating that many of these customers are evaluated PD with great certainty. In step 1 the involvement of two new models does not have a great effect on the customer distribution. Although a rise in buckets 14 to a total of 796 customers is observed. In step 2 the rise of customers in buckets 14 is probably connected with using model 4 in more evaluation cases. The number of customers in buckets 14 rose to 878 and number of customers in bucket 10 is 263.

5344

I. Perko et al. / Expert Systems with Applications 38 (2011) 53365345

549 6.31%

655 6.43%

7 5.0%

1,022 6.95%

536 6.06%

427 6.20%

In step 3a the number of customers rises from 3482 to 3763, due to newly added customers in portfolio. The number of customers in buckets 14 is reduced to 563. The reduction can be connected with the decline of models accuracy, due to data change.6 A rise in bucket 5 and bucket 10 is also observed. In step 3b the number of customers in buckets 14 is 985, although the gravity is now around the bucket 3, with the average PD 0.65%. The gravity in buckets 59 also shifts from bucket 5 to bucket 7. The PD evaluation results show the system ability to adapt the use of the available models to the changes in the environment. The BIMAS-PD results performed accordingly to the expectations: the PD evaluation results quality is preserved over time due to the successful execution of the predictive analytics processes. The unied representation of the results is accomplished by combining the content stored in local KMSs, published with the means of MAS. The use of multiple models in PD evaluation is proven to be possible, even more, the automated model selection, based on existing knowledge, enables the replacement of models with inferior test results by better performing models. We believe that by automating the management processes and using KMS for the means of presentation, the option of using multiple models in the decision process became relevant.

3482

3482

3763 699 97.24% 70 29.19% 350 12.87%

Total

233 93.99%

3482

227 95.13%

263 97.71%

10 50.0%

50 20.86%

47 20.54%

34 20.51%

9 20.0%

977 13.78%

996 14.27%

639 14.69%

8 10.0%

482 11.95%

43 21.09%

465 96.19%

3763

4. Discussion and conclusions

611 3.44% 519 3.39% 555 3.43% 498 3.41% 516 3.48%

Table 5 The PD evaluation results after each step.

Step

The information quality deterioration, the complexity of analytic processes, and the inability of providing the unied representation of multiple analytical models are identied as the most important issues hindering the use of multiple predictive models for the PD evaluation. These issues prevent the exploring of positive effects in using a managed multiple models system over using a single static predictive model. The issues are tackled using a combination of active process management and unied representation of results, stored in distributed locations. Two technologies are involved: dynamic intelligent agents and reasoning enabled KMS with a mission of creating an adaptable transparent system, capable of recognizing the environment dynamics and acting upon relevant changes. The BIMAS-PD utilizing the KMS, MAS and IA capabilities to successfully manage the analytic processes and provide unied PD evaluation results representations is presented and tested. The combination of IA, using the external ontologies in the decision process, and the MAS communication means results in accessible complex MAS services, provided by relatively unsophisticated intelligent agents. The active process management and unied result representation using IA and reasoning KMS is tested in a system of multiple analytic models for the PD evaluation. The dynamics in a BI system is simulated with a sequence of changes in system of models, individual models conguration, and data applied to the system. The BIMAS-PD is evaluated on its success in managing the predictive analytics processes and providing unied PD evaluation results. The test results reveal the success in active process management and the unied result representation. All the required analytical processes are executed, ensuring the models delivered optimal results in time and producing consistent results in every step of the experiment. The information quality is preserved and the model test results and the PD evaluations are presented in a uniform fashion. Possessing the multiple models results, the problem of selecting the best one emerges. The unied model representation allows detailed individual result explanation for all the models involved. The comparable
6

6 3.0%

377 2.27%

348 2.25%

458 2.27%

1,156 2.26% 170 1.24% 238 0.54% 46 0.33% 109 0.05% Customers Average PD 3a

5 2.0%

299 1.31%

310 1.31%

4 1.0%

336 0.54%

374 0.56%

384 0.57%

3 0.5%

55 0.33%

60 0.34%

69 0.35%

Class/PD upper limit

2 0.3%

59 0.07%

63 0.10%

115 0.07%

1 0.0%

Customers Average PD

Customers Average PD

Customers Average PD

3b

Customers Average PD

Values

64 0.08%

100 0.36%

633 0.65%

188 1.24%

248 1.30%

250 2.25%

The data on 31.12.2008 already contain more defaults, due to the nancial crisis.

I. Perko et al. / Expert Systems with Applications 38 (2011) 53365345

5345

results and the reasoning capable metadata enable the dynamic determination of the most appropriate model for every customer in the portfolio. Therefore the BIMAS-PD results are used to reason about the most appropriate model. For the reasoning about the most appropriate model, the institutional business rules, the model test results, the customer properties and the results are involved. Selecting the most appropriate model the risk of model failure is greatly evaded. Model results, shoving inferior performance are not selected for the PD evaluation. This raises the reliability of the system of predictive models to a much higher level than the reliability of a single model. The acquired results document the success and prove the capability of BIMAS-PD in managing the analytical processes of multiple models. A solution resolving the problem of continuously employing multiple prediction models in the PD evaluation using a MAS and reasoning capable ontologies is presented in this paper. Since this solution is not presented in the literature so far, we believe this paper represents a valid contribution to the research community. The BIMAS-PD is focused to the management of the PD evaluation analytic processes. It would be interesting to expand the use of MAS and ontologies to manage other PA processes and processes involved in data acquisition and information distribution processes in the BI systems. References
Alavi, M., & Leidner, D. E. (2001). Review: Knowledge management and knowledge management systems: Conceptual foundations and research issues. Mis Quarterly, 25, 107136. Basel Committee. (2006). Basel II: International convergence of capital measurement and capital standards: A revised framework. <http:// www.bis.org/publ/bcbs118.htm>. Bellifemine, F., Poggi, A., & Rimassa, G. (2001). Developing multi-agent systems with a FIPA-compliant agent framework. Software-Practice & Experience, 31, 103128. Bhrammanee, T., & Wuwongse, V. (2008). ODDM: A framework for modelbases. Decision Support Systems, 44, 689709. Bratman, M. (1987). Intention, plans, and practical reason. Cambridge: Harvard University Press. Cheung, W., & Babin, G. (2006). A metadatabase-enabled executive information system (Part A): A exible and adaptable architecture. Decision Support Systems, 42, 15891598. Crouhy, M., Galai, D., & Mark, R. (2000). A comparative analysis of current credit risk models. Journal of Banking & Finance, 24, 59117. DInverno, M., Luck, M., Georgeff, M., Kinny, D., & Wooldridge, M. (2004). The dMARS Architecture: A specication of the distributed multi-agent reasoning system. Autonomous Agents and Multi-Agent Systems, 9, 553. Du, Y. (2004). Credit rating, default probability and structural credit risk models. Ph. D. Queens University at Kingston, Canada. European Parliament, C. (2006). Directive 2006/48/EC of the European Parliament and of the Council of 14 June 2006 relating to the taking up and pursuit of the

business of credit institutions (recast) (Text with EEA relevance). Ofcial Journal of the European Union (Vol. 48). FIPA. (2006a). FIPA abstract architecture specication. <http://www.pa.org/>. FIPA. (2006b). Foundation of intelligent physical agents (FIPA). <http:// www.pa.org/>. Frappaolo, C. (2006). Knowledge management. Oxford: Wiley. Fuertes, A. M., & Kalotychou, E. (2006). Early warning systems for sovereign debt crises: The role of heterogeneity. Computational Statistics & Data Analysis, 51, 14201441. Hsieh, N. C. (2004). An integrated data mining and behavioral scoring model for analyzing bank customers. Expert Systems with Applications, 27, 623633. Huang, C. L., Chen, M. C., & Wang, C. J. (2007). Credit scoring with a data mining approach based on support vector machines. Expert Systems with Applications, 33, 847856. Kishore, R., Zhang, H., & Ramesh, R. (2006). Enterprise integration using the agent paradigm: Foundations of multi-agent-based integrative business information systems. Decision Support Systems, 42, 4878. Krink, T., Paterlini, S., & Resti, A. (2008). The optimal structure of PD buckets. Journal of Banking & Finance, 32, 22752286. Laclavik, M. (2006). Ontology and agent based approach for knowledge management. Ph. D. Univerzita Pavla Jozefa afarika v Koicicah. , D., Rupnik, R., Bajec, M., & Krisper, M. (2007). Agent oriented approach for Lavbic integration of BI systems. In Information technology interfaces, Cavtat, Croatia. Microstrategy. (2006). The 5 Styles of Business Intelligence: Industrial strength business intelligence. <http://www.microstrategy.com/>. Mohammadian, M. (2004). Intelligent agents for data mining and information retrieval. London: Idea Group Publishing. Moss, L. T., & Atre, S. (2003). Business intelligence roadmap: The complete project lifecycle for decision-support applications. Boston: Addison Wesley. Novak, P. & Dix, J. (2006). Modular BDI Architecture. In AAMAS, Hakodate, Japan. Oracle. (2006). Oracle data mining Concepts. <http://www.oracle.com/ technology/documentation/datamining.html>. Perko, I. & Bobek, S. (2007). An agent model in BI knowledge intensive environment. In Cavtat: ITI 2007. Perko, I. & Bobek, S. (2008a). Supporting business intelligence in a knowledge intensive environment: BIMAS, a multi agent system. In Wienna: EMCSR 2008. Perko, I., & Bobek, S. (2008b). Intelligent agents in managing data mining processes: Predicting probability of default of bank customers. Journal of International Scientic Publications: Economy & Business, 2, 544562. Rao, A., & Georgeff, M. (1995). BDI agents: From theory to practice. In ICMAS95 (pp. 312319). Cambridge: MIT Press. Shim, J. P., Warkentin, M., Courtney, J. F., Power, D. J., Sharda, R., & Carlsson, C. (2002). Past, present, and future of decision support technology. Decision Support Systems, 33, 111126. Sirin, E., Parsia, B., Grau, B. C., Kalyanpur, A., & Katz, Y. (2007). Pellet: A practical OWL-DL reasoner. Web Semantics: Science, Services and Agents on the World Wide Web, 5, 5153. Van Gestel, T., Baesens, B., Van Dijcke, P., Garcia, J., Suykens, J. A. K., & Vanthienen, J. (2006). A process model to develop an internal rating system: Sovereign credit ratings. Decision Support Systems, 42, 11311151. W3C. (2006). OWL web ontology language overview. <http://www.w3.org/TR/owlfeatures/>. W3C. (2007). SPARQL. <http://www.w3.org/TR/rdf-sparql-query/>. Yeh, I. C., & Lien, C. H. (2009). The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36, 24732480. Yu, L., Wang, S. Y., & Lai, K. K. (2009). An intelligent-agent-based fuzzy group decision making model for nancial multicriteria decision support: The case of credit scoring. European Journal of Operational Research, 195, 942959.