What is Set Processing?

Set Processing uses SQL to process groups, or sets, of rows at one time rather than processing each row individually. With row by row processing you following a repetitive loop that selects a row, determines if it meets a given criteria, if so, apply rule x to row Update row, commit. With set processing, you only select those rows that meet the filtering criteria and then run the rule once again all the affected rows. Advantages of Set Processing? Improved Performance: - Our internal testing has revealed that, in an overwhelming majority of cases, set processing performs significantly better than it is -by-row counterpart for reasonable batch processing volumes. Minimized SQL Overhead: - It tends to use fewer Application Engine SQL tatements that each processed more data than the statements executed in row-by-row processing. Easy Maintenance: - if need to make a fix or add an enhancement to SQL, its just a matter of modifying the SQL or inserting the new Chunk. Leveraging the RDBMS: - With Set based processing, you take advantage of the SQL processing engine on the database rather than placing the processing burden and overhead on the application executable.

The word Set in Set Processing comes from the mathematical definition meaning a group of elements. In the case of database programming, a row of data is a single element and a set would be a group of rows. Processing means to change data in some way such as a calculation. Multiplying Hours Worked times Pay Rate to get the Gross Pay of an employee is an example of processing. Thus, Set Processing means to process data as a group of rows rather than one row at a time. To get a good grasp of what Set Processing can do for you requires good definitions for both Set Processing and Row-by-row Processing and then to compare the two types. Row by Row Processing When the data in your application requires some type of changes, there needs to be some method to get at this information and modify it. In most of the traditional languages the programmer codes a set of statements to accomplish this. Here is an example of how the programmer might code this: Begin a loop Get the next row of data to process If no row available, then exit the loop Do the needed update (calculation, modification, deletion, etc.) to that one row Continue at the top of the loop This set of statements is called Row-by-row Processing because it processes data one row at a time. Use

of a Do Select Action is an example of row-by-row processing. Set Processing is built on the concept of handling groups of rows at one time. By contrast to Row-by-row Processing, the coding of Set Processing goes like this: Do the needed update to all the rows to process From the programmers perspective this is definitely simpler than the row-by-row procedure. Set Processing is called non-procedural because the programmer tells the program what to do, but not how to do it within the database. The programmer does not specify the procedure used to do the update. Instead this procedure is built into the Relational Database Management System (RDBMS) and it is out of the hands of the programmer. The details of how to do it are left up to the RDBMS, thus various types of built-in optimizations are applied to the update that the programmer need not program or worry about. The biggest advantage Set Processing has over Row-by-Row is performance. One of the important aspects of computers and computer programs is that they be sufficiently fast for the tasks at hand. Using Set Processing correctly in your Application Engine programs can reap performance benefits of 10, 20, 30 or sometimes even more than 50 times the speed of row-byrow processing. When you take a batch update Application Engine program that runs in a half hour and change it to run in a minute is of no small consequence. It also will free up needed server resources.