Sie sind auf Seite 1von 22

A PlantPAx Virtual Batch Upgrade

Insert Photo Here Sean Nichols


Controls Engineering Lead The Bama Companies
Rockwell Automation Process Solutions User Group (PSUG) November 14-15, 2011 Chicago, IL McCormick Place West
Copyright 2011 Rockwell Automation, Inc. All rights reserved.

The Bama Companies


History: Founded in the early 20th century Privately held All US production based in Tulsa, Oklahoma Inclusive environment (all stakeholders at every level involved) Products made include:
Biscuits Pizza dough Hand held pastries Pie crusts Breadsticks

Copyright 2011 Rockwell Automation, Inc. All rights reserved.

Sean Nichols Bio


Educational
Nuclear power (Navy) Electrical Engineering from OSU Extensive Rockwell training

Occupational
5 years with the Naval NPTU 7 years with Bama Companies 5 years maintenance 2 years as Corporate Controls Engineer Current responsibilities include plant floor controls and visualization for all Tulsa based plant locations

Personal
Married, 3 kids, one on the way; live in Tulsa, Oklahoma Interest include: Family activities, computers

Process Description
Ingredients Sheeting Line

Freezer Mixing

Packaging

Goal: Address Challenges with Frozen Dough


Pre Improvement State:
System could get out of sequence Had difficulty recovering from power outages Legacy technology (PLC-5 and RSView32 based platform) Maintenance and troubleshooting had to occur at a Ladder Logic level Changed programing on average once a day (Bama-tized) Constantly tweaking to try and make process run better or make recipe changes Environmental and UPS considerations to make system more reliable Improvements for recipe security and operator tracking S88 based programming (modular equipment/phase coding and flexible recipes) Complete documentation sets around functional operations VMware and thin client architecture ( Jointly driven by IT and Engineering ) Electronic hand adds and other SOPs (goal to eliminate batch sheets) Electronic lot tracking to meet regulatory compliance Whatever chosen would be planned at other facilities

Desired Improved State:


Solutions Evaluated
All technology was to remain Allen-Bradley based
Installed base and familiarity with products

Three independent implementations reviewed:


1) PC based batch manager integrated with ControlLogix; delivered by a System Integrator - Recipe configuration, Campaign management, Batch reporting ConrolLogix and RSView32 system custom created for Bama; delivered by System Integrator - Lowest HW/SW investment - Already familiar with the process - Concerns around bugs; (Bama only knowledge) PlantPAx (FTBatch, FTView, ControlLogix) system delivered by Rockwell Automation - Recognized Industry leader (Price Premium) - Dedicated Food and Beverage group - Off the shelf application
Copyright 2011 Rockwell Automation, Inc. All rights reserved.

2)

3)

Vendor Selected
PlantPAx (FTBatch, FTView, ControlLogix) system delivered by Rockwell Automation Only one to formalize/organize a User Requirements Spec
(Was a billable activity but of value)

System interruption was designed to go into a Hold state


Power outages and brownouts are common at our facilities Other systems would end up scrapping batches

Flexible recipe management True S88 compliant offering

Copyright 2011 Rockwell Automation, Inc. All rights reserved.

Delivery Team Bama Foods


Sean Nichols:
Project Manager & Engineering Lead

Rockwell Automation Solutions


Lynn Hecathorn:
Project Manager

Dwayne McAlexander:
Lead Engineer

Nick Huddleston
Process Improvement Mngr. Operations Lead

David Keen:
HMI Implementation

John Blaylock
Networking and SW Installs

Jed Hepfner:
Field Support, Training, Testing, Startup support

Additional Bama Staff


SQL integration team 4-5 Automation Techs 4-5 Operator Blackhats Executive Sponsorship (Budget

Industrial Controls of Ok. Troy Bridenstine:


rework)

Electrical Foreman

Copyright 2011 Rockwell Automation, Inc. All rights reserved.

Project Execution: Foundation Work


User Requirements Specification and Proposal
Interview and captured voice/concerns from IT, Operations (Plant and Production), Maintenance, and Quality Allowed for project to be on fast track Knowledge of third party expert (and implementer) was trusted in design Justified a network audit prior to startup

Contract Negotiations (Good Faith Start)


Mutual investment in creating a sustainable relationship Required the same engineering lead to design and implement (to the best extent possible)

Functional Specification
Mapped out entire process (Blueprint) Completed and approved before design/coding started 9 revisions by Bama lead operators, engineering, maintenance (inclusivity) Contractually binds the deliverables and acceptance criteria
Copyright 2011 Rockwell Automation, Inc. All rights reserved.

Project Execution: System Topology


19 Logical Racks
500+ Control Modules 9 Weigh Scales

22 Batch Units
105 Phases 25 Active Recipes

1 ESX Server
FTView SE VM FTBatch VM SQL VM Citrix Published Apps 10 Thin Clients Redundant HardDrives
Failover Scheme
Copyright 2011 Rockwell Automation, Inc. All rights reserved.

Project Execution: Hardware Design


ControlLogix L74 Controller
Supports existing Control Modules plus room for spare w/ redundancy Primarily Ladder Logic (standard programming language for Bama) Limited Function Block used for certain capabilities

Reuse as much RIO as possible


(2)1756 Remote I/O Scanners
1771 Racks Weigh scales Local PLC (rack 0) replaced with ASB

Project Execution: Hardware Design


Added new FLEX I/O to Ethernet
Multiverter (hardware that was not previously automated)

Added Operator Interfaces for Hand Adds


Designed to be PanelView; Switched to thin client to be compatible with Windows Domain Servers (required for tracking scheme)

Project Execution: Software Design


FTView SE
250 Screen License RA standard GEMS Library version C2 (~6 module classes) Event logs from each client and alarm logs are dumped to a SQL database through an ODBC connection

FTBatch
22 Units
Incorporated all Hand Adds, Equipment Checks, and SOPs through operator prompting Batch journal is dumped to a SQL database through an ODBC connection

All recipes created by Bama


Found to be much more flexible then previous system Plans to talk with Product group about ideas for future enhancements Ultimate recipe and procedural edits will be owned by Quality and Process Improvement

SQL Database
Existing System prior to upgrade Uses Business objects (our ERP) Data table work required but ultimately working with our existing reports
Copyright 2011 Rockwell Automation, Inc. All rights reserved.

Project Execution: Virtual Configuration & Deployment


Hardware:
Dual ESX servers; Hard drive SAN controllers connected by dual fibers (14 HDs partitioned as 2 arrays) One HD in each array is a hot spare (Standard RAID technology) Each ESX server has a 1GB switch with dual connections Redundant fiber to core switch Off site backup process for all images

Software
System originally built for existing IT/Business infrastructure Process system added with minimal changes
New VLAN and a few switches standard growth

Not completely new concept (running RSView32 Virtually for 3 years) Ten (10) Thin clients being managed by single terminal server image
Install once, use multiple, replace easily Only ini file needs to be updated with MAC ID of new hardware

Three (3) virtual servers dedicated to the PlantPAx system Fourth virtual server (SQL) offsite Three (3) Citrix Applications

Project Execution: Control Code Implementation


RA standard GEMS Control Modules version C2 Phase Manager using a sequencer that can be controlled internally or externally (operators can walk through an entire batch manually if need be) Supports strutt (scheduled checks/test of equipment) Needed additional units to insure appropriate arbitration Comments can be added at any point to the batch record Broke recipes into a wet mix and mixer class (3 mixers)
Batch ID transfers automatically Does a material check to make sure product compatibility Can run multiple products in unison

Copyright 2011 Rockwell Automation, Inc. All rights reserved.

Control Module Checkout


Purpose to verify that all GEMS control modules used were compatible with the 1771 I/O and that the ControlLogix controller can communicate with all the racks This process requires downtime as it temporarily puts the new hardware topography in place and test the system to manage the risk of a swap over During our specific checkout:
4 racks visible to the PLC5 but invisible to the ControlLogix until we fixed one dip switch setting and repaired a damaged wire Found the weigh scales were wired for DH+ instead of RIO

This was a lifesaver for startup because it eliminated hurdles that would have existed otherwise

Copyright 2011 Rockwell Automation, Inc. All rights reserved.

Factory Acceptance
Performed At Bama :
Brought to Bama at our request; standalone simulated process so we could have a large audience and increased exposure Difficult to segment between familiarization, training, and, testing Left active in the off hours so others could interface with system Eventually used the virtual test rig from FAT for startup and cut install time (just redirected communications) Remote access for Rockwell Automation allowed so that FAT punch list and initial installation steps could be done before install

Test Plan: developed against functional spec


Delivered at start of FAT, could have been streamlined if delivered earlier (similar to Function Spec process) Executed against test plan and captured punch list items
Copyright 2011 Rockwell Automation, Inc. All rights reserved.

Startup and Commissioning


Application Training: Valuable but could have been better
No one was previously at formal batch training (informal application training) Plan was to train first and then do FAT Was a lecture/watch format with electronic content provided by Rockwell instead of do format (pros and cons) 7 day shutdown
Standard Bama SOPs included Rockwell (and all other contractors, over 40 total) in planning meetings to create master schedule and resolve conflicts up front Used a field resource (Jed) to accompany Dwayne of the 24/7 coverage. Jed had worked on the project and come onsite with Dwayne for a few days beforehand to insure a smooth support structure. Monday startup gave plenty of run time to debug/familiarize and allowed for 24/7 coverage to go to remote support by the weekend

Copyright 2011 Rockwell Automation, Inc. All rights reserved.

Lessons Learned
Not all Units are Created Equal:
Typical Units = mixers, hoppers, slurry tanks, etc We found others such as a Salt Chute.

Remote ESX Images:


Installations and operations from VPNs and firewalls can be a challenge. Using vSphere as a Citrix App seems to remedy the situation.

Local RA Field Support:


Do your best to have Rockwell Automation use a local resource with the appropriate skill set so that when the delivery team is complete you have expertise around the corner. (We are also attempting to do this through ICO)

Dont use Microsoft auto updates:


IT and Eng have an agreement to work on required patches together and determine how to deploy

FAT:
Was short as we were trying to train at the same time we were checking out; should not have tried to address issues all at once instead of working on a fix at the time.

Virtual Servers:
Use Snapshots to keep fresh rollback points.

Copyright 2011 Rockwell Automation, Inc. All rights reserved.

Benefits Seen
Still in the evaluation processes but
Looking to see about improvements around power outages Flexible recipes trials (no salt) completed without interrupting production Found a lot of hardware and material deficiencies as part of the process i.e. RPM sensor preventing packing flour sifter at least once since install Double signoff now needed which improves/enforces QA involvement on signoff Eliminated paperwork for hand add tracking Improved manual equipment control capabilities that did not exist before Improved alarming and troubleshooting capabilities from the plant floor Inclusivity of system selection and deployment improved plant adoption

Plans Moving Forward


Offsite data replication to further improve disaster recovery More in-depth training on batch and phase manager Pursuing continuing the rollout of the new standards across the organization Review new data being collected to see how they may be incorporated into reporting/process improvements

Copyright 2011 Rockwell Automation, Inc. All rights reserved.

Insert Photo Here

Rockwell Automation Process Solutions User Group (PSUG) November 14-15, 2011 Chicago, IL McCormick Place West
Copyright 2011 Rockwell Automation, Inc. All rights reserved.

Das könnte Ihnen auch gefallen