Author: ABBdriveX

Home  ›  Posted by ABBdriveX  ›  Page 23

AQ: How/where do we as engineers need to change?

System Design – A well designed system should provide clear and concise system status indications. Back in the 70’s (yes, I am that old), Alarm and indicator panels provided this information in the control room. Device level indicators further guided the technician to solving the problem. Today, these functions are implemented in a control room and machine HMI interface. Through the use of input sensor and output actuator feedback, correct system operation can be verified on every scan.

Program (software) Design – It has been estimated that a well written program is 40% algorithm and 60% error checking and parameter verification. “Ladder” in not an issue. Process and machine control systems today are programmed in ladder, structured text, function block, etc. The control program is typically considered intellectual property (IP) and in many cases “hidden” from view. This makes digging through the code impractical.

How/where do we as engineers need to change? – The industry as a whole needs to enforce better system design and performance. This initiative will come from the clients, and implemented by the developers. The cost/benefit trade-off will always be present. Developers trying to improve their margins (reduce cost – raise price) and customers raising functionality and willing to pay less. “We as engineers” are caught in the middle, trying to find better ways to achieve the seemingly impossible.

AQ: Friendly system without technicians diagnose

How to make our systems so friendly that they do not need technicians to help diagnose problems? Most of the more obvious answers have been well documented here but to emphasize these points it is normally the case that diagnostics and alarms can dominate the amount of code constructed for an application. That is the amount of code required to fully diagnose an application may be as much if not more than the code required to make the application work in the first place!

I have seen HMIs with diagnostic screens showing an animated version of the Cause & Effects code that allows users to see where the trip condition is. I have also seen screens depicting prestart checks, Operator guides, etc all animated to help the user. Allen-Bradley even have a program viewer that can be embedded on an HMI screen but you would probably need a technician to understand the code.

From a control system problem perspective it is inevitable that you would need to use the vendor based diagnostics to troubleshoot your system. Alarms may indicate that there is a system related problem but it is unlikely that you could build a true diagnostics application in your code to indicate the full spectrum of problems you may encounter. For example if the CPU dies or the memory fails there is nothing left to tell you what the problem is 🙂

From an application/process problem perspective if you have to resort to a technician to go into the code to determine your problem then your alarm and diagnostics code is not comprehensive enough! Remember your alarming guidelines an Operator must be able to deal with an abnormal situation within a given time frame, if the Operator has to rely on a Technician to wade through code then you really do need to perform an alarm review and build a better diagnostics system.

AQ: Techniques contribute in control system

1. Any successful methodology is not a simple thing to come by and typically requires a huge commitment in time and money and resources to develop. It will take several generations to hone the methods and supporting tools.

2. Once you get the methods and tools in place, you then face a whole separate challenge of indoctrinating the engineers in the methods.

3. Unique HMI text involves a lot of design effort, implementation, and testing.

Many of the techniques contributed by others in the discussion address faults, but how do you address the “normal” things that can hold up an action such as waiting for a process condition to occur, such as waiting for a level/pressure/temperature to rise above/fall below a threshold or waiting for a part to reach a limit switch?

Some methods allow for a text message that describes each step. When developing these text messages, I focus on what the step’s transition is waiting for, not the actions that take place during the specific step. This helps both the operator to learn the process as well as diagnose what is preventing the machine from advancing to its next step.

I have seen sequencing engines that incorporate a “normal” step time that can be configured for each step and if the timer expires before the normal transition occurs, then you have “hold” condition. While effective, this involves a lot of up-front development time to understand the process and this does not come cheaply (with another nod to John’s big check!).

(Side note on sequential operations: I have used Sequential Function Charts (SFCs/GRAFCET) for over 20 years and find them to be exceptionally well-suited for step-wise operations, both from a development perspective as well as a troubleshooting perspective.)

I have seen these techniques pushed by end users (typically larger companies who have a vested interest in standardization across many sites) as well as OEMs and System Integrators who see these as business advantages in shortening development, startup, and support cycles. Again, these are long-term business investments that require a major commitment to achieve.

AQ: Systems Development Life-Cycle

Step 1. Initiation
Step 2. System Concept Development
Step 3. Planning
Step 4. Requirements Analysis
Step 5. Design
Step 6. Development
Step 7. Integration and Test
Step 8. Implementation
Step 9. Operation and Maintenance
Step 10.Disposition

There are three major players present in this model; Customer (client), System Integrator, and Machine or device manufacturer.

In many instances, the result of step 4 (Requirements Analysis), is an RFQ for the system implementation has been issued to one or more systems integrators. Upon selecting the system integrator, step 5 (Design) begins. Upon completing step 5 (Design), the system or process flow is defined. One of the major outputs from step 5 are the RFQs for the major functional components of the finished system. Based on the RFQ responses (bids), the Machine or device manufacturers are chosen.

Steps 6, 7, and 8 are where all the individual functional components are integrated. This is where the system integrator makes sure the outputs and feedback between to machines or devices is defined and implemented. Step 8 ends with a full systems functional test in a real manufacturing situation is demonstrated to the customer. This test includes demonstrating all error conditions defined by the requirements document and the systems requirements document. If a specific device or machine fails its respective function it is corrected (programming, wiring, or design) by the manufacturer and the test begins anew.

Each of the scenarios presented is correct. The technician role being presented (customer, integrator, or manufacturer) is not clear. System diagnostics are mandatory and need to be well defined, even in small simple machines. There should be very few and extreme conditions under which the customer’s technician should ever have to dig into a machine’s code to troubleshoot a problem. This condition usually indicates a design or integration oversight.

(You can find a complete description here, http://en.wikipedia.org/wiki/Systems_development_life-cycle)

AQ: Why we need Engineers?

Even the humble motor car runs diagnostics that the garage read to see the problems with your car. This doesn’t involve technicians looking at the code that controls the car but is 100% driven by the faults flagged by the car’s management system programs. These could even be displayed to the users, the drivers like me and you but the manufacturers don’t want amateurs hacking around their management systems and you know that is exactly what we would do.

Do we ask for this functionality from our car manufacturer? Do we complain about it and ask for them not to fit it? Would we like to go back to the “golden age” of motoring where we spent as much time under the hood as we did on the road?

You do?…. Yeah right and neither do I nor do I want a plant where I need a guy with a laptop to diagnose a blown fuse, sticking valve, overload trip, etc .

We need to change as Engineers by selling systems to customers that fulfill their needs, that are safe and reliable, that follow industry and international best practices and are user friendly. The notion of having to wait for a blank cheque from the customer to fulfill these goals is really a cop out, you either do what is right or just walk away because at the end of all this it is you who are under scrutiny when things go wrong not the customer who will plead ignorance.

AQ: Design and Implementation

The owner of the system should provide clear requirements of what the system should do and should define what constitutes “maintainability” of the system. This places a burden on the owner of the system to consider the full life-cycle of the system.

1. You need good design documentation.

2. All source code should be well-documented.

3. Coders should be trained on the techniques used and mentored,

4. The use of “templates” helps ensure that coders and maintenance alike are familiar with routine functions.

5. The HMI should provide clear indication of faults and interlocks.

6. The HMI should provide clear indication of equipment statuses.

7. Any code that is hidden must “work as advertised”. This means that it must be completely and unambiguously documented for all inputs, outputs, statuses, and configurations. It must be thoroughly tested and warranted by the vendor,

8. All code should be well-tested. (I have found that the first line of defense is to simply read the code!)

Post-Startup
1. The owner should have a change-control procedure to manage modifications.

2. All users and maintenance support personnel should have adequate training. Training needs to be periodically refreshed as it can become stale through lack of use.

AQ: “critical” operation with a double-action cylinder, hydraulic or pneumatic

If I had a “critical” operation with a double-action cylinder, hydraulic or pneumatic, I’d put proximity sensors on both ends of travel, typically with small metal “marker” on the shaft. Each input “in series” with the “output” to each coil, time delayed to give the cylinder a chance to reach its destination. The “timer” feeds the “alarm.” If you want to spend the money for a pressure switch (or transducer) on each solenoid output, that’s a plus.

Now you can tell if there was an output to the solenoid from internal programming, if not another interlock prevented it from actuating. If there is an output to the solenoid and no pressure, then the signal did not reach the coil (loose wire somewhere), if it did the coil may be bad, if the coil is good and no pressure, the solenoid may be stuck or no pressure to it from another supervised failure or interlock. If there was sufficient pressure and the cylinder travel not reached, then the cylinder is stuck.

As a technician crawling over all kinds of other people’s equipment since 1975, I could figure out a lot of this from an old relay logic or TTL control system. A VOM confirms whether there is an output to the correct solenoid at the control panel terminals. This lets you now which direction to head next. If there is no power, it’s “upstream” of there, another interlock input that needs to be confirmed, time to dig into the “program.”

If there is power and the cylinder does not move it’s a problem outside of those terminals and the control system. I’d remove the wiring and check for coil resistance, confirming the coil and field wiring integrity while still at the panel. If everything checks out then go to the cylinder and see if a pressure gauge shows pressure on the line with the coil energized – presuming there is pressure to the valve. No pressure would be another “input alarm” from another pressure switch. If there is pressure and power to the valve and no pressure, the valve is bad. If there is pressure on the output side and the cylinder does not move – the cylinder is stuck or mechanically overloaded.

I&E “technicians” may know a lot about programming and code, but if they don’t know how a piece of equipment operates I/O wise then they don’t have a clue where to start looking. Then I guess you need all the sensors and step by step programmed sequences to “spell it out” for them on a screen. A device sequence “flow chart” may help run I/Os out for something like above. I/O status lights on the terminals like PLCs can easily confirm at a glance if you have the proper inputs for a sequence to complete, then you should have the proper outputs. Most output failures are a result of correct missing inputs. The more sensors you’re willing to install, the more the sequence can be monitored and spelled out on an HMI.

From a factory tech support in another location, being able to access the equipment remotely is a huge plus, whether directly through modem, or similar, or indirectly through the local technician’s computer to yours i.e. REMOTE ASSISTANCE. A tablet PC is a huge plus with IOMs, schematics and all kinds of info you can hold in one hand while trouble-shooting.

AQ: Floor programmer and office programmer

The biggest differences between the floor programmer and the office programmer is often a piece of paper (knowledge and experience do not replace a piece of paper in the mind of HR person that has no understanding of the position they are seeking to fill) and that the floor programmer must produce a working machine. Also many an excellent programmer will never put up with the office politics seen in many companies. To appear right for me is worthless when being right is the goal. In a physical world it can be shown that a program is right or wrong because the machine works or does not. In the theory driven world of the office that can not happen, so appearing correct as well as being correct is necessary.

If you are the best programmer in your company or the worse. If you are the worse one then maybe you are correct. But if you are the best then please take a close look at the worse programmer’s work and tell us if there is not a need for some improvement.

I have cursed out more than one officer programmer for missing logic which on the floor is easy to see is necessary. The office programmer was more than once, myself. Making logic to control machine in theory is far more difficult a task than modifying that logic on a real running machine. Maybe your imagination and intelligence can create a theoretical image that matches the physical one.

Many office programmers are not up to that level. They lack the intelligence, imagination, experience or time to take an offline program that can be loaded and run a machine without help. But no fear, most start-up techs cannot debug a machine after the build is complete and remove all issues that will surface when the machine enters a customer’s plant and full production.

A good program will grow as time passes. To fill in the gaps in the software, to change the design from what design intended to what production requires and to cover the design changes as product models evolve. Static is not the floor condition of a good company, products and machines evolve and grow. More reliable, durable, quicker tool changes or device swaps, lower cycle times or more part types. There are examples of logic once written it never changes but that is not the whole of the world just one part of it.

AQ: Machine tool

Ahh I see the words machine tool and shop floor; now I can see where you guys are coming from. The type of machines that you talk about were controlled by relay logic and then when technology arrived the electrical drawings were probably “converted” into ladder logic. The techs had lots to do because you cannot translate relay based systems into ladder logic 100% successfully as they behave differently.

The guys doing this work are just that programmers. They are probably NOT software engineers and are closer to the shop floor techs who are fiddling about with your machines.

I can and have designed many control systems for automotive type machines such as hobbing machines, milling and borers. Very easy code to write if you do not translate the relay logic directly but use the existing documentation as a reference. All of the systems that I did work really well. I did some similar type of machines in a pharmaceutical plant but that was after another company was kicked out after failing to make the machines work. I had to redesign the whole control philosophy as the machine tool world methods used were really a bad fit for the intended application.

But that is only one facet of the work that we Industrial Automation Engineers do. I work in many different industries where the demands for quality deigned, controlled and maintained systems is paramount. We go through proper project life cycles and we deal with the project from inception through design, build, test and commissioning. We even do the maintenance of the systems. We do not sit in Ivory Towers but do the work at the customer site no matter where that is on the planet.

Electrical engineers are tasked with doing all things electrical and we are tasked with all things control. Programming, that is writing the actual code is only one part of what we do and not necessarily the most time consuming part.

I am here in Kazakhstan at the sharp end of a multi-billion dollar project a long way from any ivory tower. I fix other engineers software too, why? Because the vendor may use offshore resources to code much of the systems that are installed at site. Kazakhstan has extreme Summers (up to 60degC) and Winters (down to -50degC), most of the people are friendly but English is not so prevalent. A long way from your shop floor environment. Far more dangerous too as the plant processes H2S or will when first Oil & Gas comes onshore.

Here I have supported technicians performing loop checks and other engineers doing logic tests. I can diagnose many loop problems without even looking in the code but just by looking at what is happening. I have found that if a loop doesn’t work then the techs approach us first as a one stop shop to give them an answer rather than actually trouble shooting the loop themselves.

I said to you guys before you need to get out and look at other industries and see what is going on in the rest of the world. Much of what I have seen would go a long way to improving your world too! Engineers like myself are far away from the “programmers” you have.

AQ: How to improve troubleshooting techniques?

The guy asked for suggestions on how to improve troubleshooting techniques. I mentioned this earlier as a “suggestion” for starters but the idea got lost in all the complaining and totally irrelevant responses like the one above.

Proper lay out of inputs and outputs and a “Troubleshooting guide” or flow chart. I have an Aris cablem modem and Netgear wireless router for internet If loose Internet service I can do three things.

A. Pick up the phone, call tech support and wait two days for someone to show up

B. Take them apart and ‘DIG INTO THE PROGRAMMING”

C. Read the instructions someone took the time to write. Before I can get an output identified by the LEDs, I have to have the correct inputs identified by the LEDs. It’s a waste of time tearing in the “programming” over a loose cable connection somewhere. Same for the wireless router and a bad LAN cable connection or network service issue on the computer. I’m already familiar with the proper LEDs for normal operation. When one goes out it gives me an idea where to start looking before even opening up the instructions which I’ve downloaded in PDFs for quick access to their “troubleshooting” guides. Maybe the service is off line – there is an LED for that. No TVs either, no service or common upstream cable connection problem, no-brainier. The first thing a Xfinity service tech does is go outside and look for a signal at the house customer jack. It’s either in his cable or my house. Once their cable had to be replaced. It mysteriously got damaged right after AT&T dug a big hole in my backyard to upgrade their Uverse service – go figure.

In order to get something to operate output wise, you need a certain amount of inputs to get it. If you don’t have a particular output, then look at the trouble shooing guide and see what inputs are required for it. If there are four direct sensor inputs required for a particular output, group them together.

Grouping internal interlocks together helps also when digging into a program like ladder logic instead of hopping through pages of diagrams or text to find everything it takes to get one output. It’s a common program development issues to throw in ideas as you program depending on where you are sequentially.