CAT Leader Forum - EBT Workshop Report

Change in aviation is slow, and for good reasons. Before you leave a well-proven system behind and embark on a new, potentially better, but unknown path, there must be thorough reflection.

EBT is a typical example. The first idea of EBT, initially referred to as Risk-Based Training, surfaced in 2005. Aeroplane manufacturers, major airlines, pilot groups and international organisations started an International Working Group in 2007, and in 2010 Emirates began the first trials. More than 10 years later, we are still discussing implementation issues, and only a minority of airlines worldwide are now using this new methodology.

It is difficult to identify precisely how many airlines are now training with EBT. For the moment, it seems that Europe leads with the Middle East following and then some in Asia-Pacific. But as there are various stages in EBT implementation, there are also misconceptions about it, including the definition of a (healthy) EBT programme. At the end of this report, you will find the link to an extensive study done by the NLR to learn more about EBT implementation around the world.

But 10 years after the first trials, there is still an enormous amount of work ahead.

Where Do We Stand?

EBT is based on the concept that there was a lot of training going on based on old data, and pilots felt that 60% of what they trained was not relevant to line operations. So, based on aircraft types, operating theatre, etc., the training could become more relevant and efficient by looking at the risks and threats.

Thus, the first question to those airlines that are advanced in this field was: do you feel we are progressing?

The answer was a firm yes! EBT-trained pilots feel much more prepared to deal with the various threats in operation than before. Heads of training, instructors and pilots agree that training has become more relevant and efficient.

So, an airline implementing EBT can achieve at least part of the initial targets.

The whole process is data-driven. Training data from instructor assessments are used to enhance the training programme to individualise the training as much as possible. This requires standardisation (instructor concordance), effective software, and trust from the pilots; this requires time and patience but is achievable.

Data from airline operations and the entire Industry must also be used and followed. Such a follow up is a massive task that takes time and resources and is quite complicated. EBT operators should initially focus on improving the reliability and use of internal assessment data, and subsequently integrate other data sources (safety, OEM, industry, etc) in subsequent years, as experience with EBT grows.

An average airline with a good team would have a functional baseline EBT programme set up and running in about 18 months. It all depends on the investment of time… and motivation.

EBT is competency-based training. That means that we use a different approach to training with a different taxonomy and a different language. Foremost, this is a challenge for instructors as the framework and assessment methods change dramatically. Good instructor training is critical for effective competency training.

Other departments, such as safety, are still to catch up with this new framework. According to airlines which have implemented EBT for a few years, this gets better, and after a while, everybody speaks the same "competency" language. We see some authorities using this language: for example, the French Accident Investigation Board (Bureau d'Enquêtes et d'Analyses - BEA) has used the competency model for the first time in an investigation report.

Both airlines which have gone through implementation and training consulting organisations that help these airlines call EBT a long journey that needs good planning, resources, and good communication, including all stakeholders. Underestimating the task is a mistake that will prolong the whole process.

Let us have a look at roadblocks and enablers across different key areas that affect and are affected by EBT implementation.

Image credit: CEFA Aviation.

Regulations

A few months back, the Head of Training meeting expressed the fear that there was a trend toward too much regulation. But is this true?

Authorities' prime responsibility is to protect the travelling public. Therefore, all regulations will have a "policing" aspect. But EASA understands the importance of innovation toward even better and safer aviation and publishes guidance material to help the industry move forward. There are, of course, political issues and legal limitations. Still, the present regulations give quite some leeway to the airlines, and there are certainly improvements that will follow as the whole subject matures.

The main thing is setting the standards right, define the desired outcome. This is the basic premise of performance-based regulation: regulate performance, not process. Then work from there. Per definition, the airline owns the process. They describe the process that takes the data from internal sources and the data from external sources and creates an adapted training that evolves as the threats change.

The regulators, mainly the local authorities, must supervise how operators develop a solid process of continuous improvement within the training operators and follow up on the results. We do not need more stringent rules; rather it is essential that authorities understand the core philosophies, principles, best practices and processes. And this seems to be an issue in many countries worldwide, probably by a lack of adequate staffing and training for the local regulators. Maybe EASA could step more into this field to ensure that their regulations are applied in the same manner by local authorities. Many advisers to operators also support regulators to understand the EBT concepts and practices, such that they can be more effective in overseeing a performance-based training system.

Overall, one can say that having too much regulation might kill the essence of EBT, but having too little could be dangerous as not all airlines are capable or willing to do what is right. It is important to realize that the different phases of EBT (principles à mixed- à baseline à enhanced) are there to grow confidence in a performance-based training system and associated performance-based oversight; this is also continuous improvement at an oversight level.

Regulations must also be such that there is a level playing field. If the rules are too complicated or complex, they could be barriers for smaller airlines to enter.

In this context, it is essential to note that there is a misconception concerning the name. Many people think that the first thing they need to do is start assembling "evidence" before building a programme.

That is not correct. There exists a baseline programme off the shelf to start, building on the ICAO EBT data report (Doc 9995, Manual of Evidence-Based Training). Using this baseline has the added advantage that the supervising regulator also has something solid to reference!

For local regulators, it can be a tricky transition. Airlines deciding to implement EBT must take their authorities with them on the journey. Both the airline and the authorities must first understand, and it is best that both go through this educational process together. This way, the regulator has the knowledge to do the oversight of the programme.

Culture and Organisational Issues

With the publication of the ICAO and IATA manuals and the EASA guidance material, we have introduced a new training method. A training methodology that is "pilot centred". A methodology where we try to individualise, where the pilot must reflect on his performance, and where the instructor is no longer telling but facilitating. But is the world ready for this?

Culture, whether national culture and organisational culture, could be a blocking point or, at least, a hindrance. In countries where recall is the primary learning style and where instructors are undisputed and teach "ex-cathedra", the new concept may be hard to accept, both by pilots and instructors. So does this mean these areas should not adopt EBT? Certainly not. It means that the airlines will need to prepare more, communicate more effectively, and understand the cultural difference and work from another culture that can override the national culture when operating. That is the "pilot" culture.

The same is also true for airlines with different nationalities in the cockpit. A major Middle East airline, employing more than a hundred nationalities, made a study on the subject and concluded that the solid professional culture, typical for pilots, allows the pilots to first identify as a pilot and then as a member of the airline and only then as a nationality.

The airline reported that the change of training methodology did not create a problem. Still, it is essential to include this in the reflection before implementing and to accompany the pilots in this journey.

There is no need even to try to change a person's national cultural heritage. By dialling up his/her professional pride and having a shared sense of profession and a strong positive organisational culture, you can subdue the national cultural elements which may be undesirable for the professional outcome.

But even within a company with a homogenous workforce, there might be differences in acceptance of the new style. The study done by NLR showed a difference of appreciation, or perception of improvement, between the pilots and the instructor group. The instructor group being more positive than the pilot group. More study is needed to understand, but the instructor group is probably more aware of the why’s and how's.

Another point that is worth looking at is the difference in acceptance by age group. One airline reported that the younger generation adopted and embraced the EBT/CBTA concept quickly, whilst the older generation needed more time adapting. Change is always challenging, and if you have been training in a specific manner and system for 30-plus years, it may be normal to be a bit hesitant.

EASA EBT regulations state some policies on how to handle low grades during training. They mainly ask for individualised training well ahead of a fail-state. To increase pilot trust in EBT, a straightforward (non-punitive) airline policy of handling weaknesses observed during training must be in place. Even voluntary training requests could be an option.

In any case, changes like this take time to be accepted, and this is again something airlines need to consider. The better the communication, the easier and quicker the new rules will be received.

One group where this acceptance is, of course, of prime importance is the instructor group.

Image credit: EBT Foundation.

Instructor Issues

During the entire workshop, one element popped up repeatedly: the instructor!

The whole concept of EBT/CBTA is dependent on the quality and standardisation of the instructors.

The first question could be: can all present instructors be (re-)trained to the new standards? Some argued that older instructors would have more difficulty compared to younger ones. But the consensus was that age is not a factor.

One participant stated: "A good instructor under present rules will be a good instructor under the new system." But unfortunately, a few instructors will need some extra time and effort to be ready, and some will not make it at all.

The main blocking point for the instructors is not the instructional part as such. But the assessment part is more complicated. There are many new terminologies, new concepts, the Observable Behaviours, the need to justify your grades and look for the root causes.

All that is quite complex and will probably be easier for somebody starting the career as an instructor but not impossible for the experienced instructor. There is a need for adequate and individualised training.

It all starts with convincing the instructors. They must believe in the value of EBT and see why this is worth the investment.

One airline explained that they give their instructors a session by the Head of Human Factors, who gives a thorough briefing based on the scientific data of why this new system is better.

They may not all remember the science behind it but are convinced of the value and ready for the change. Then, and only then, effective training for the instructors is possible.

Pilots have strengths and weaknesses, and ideally, the organisation should tailor instruction to cater for these individual differences. Instructors also have their strengths and weaknesses in their competency set. Surely we should individualise the instructor training also to the unique needs of the instructor. For the moment, the instructor training is a one-size-fits-all, but as we move forward, we should start the development of tailored instructor training as we are developing the tailored pilot training.

Competency-based training is all about more training and less checking. But then we start talking about assessment and grading by the instructor, which the pilot group may easily interpret as even more checking. It is, therefore, of utmost importance that the instructors do not fall into the trap of behaving as examiners.

The assessments and grading are done only with the idea of establishing the strengths and weaknesses of the student, and by looking for the root causes of the weaker points, help the student develop himself to get to the required competency.

And finally, there is the question of outsourced EBT training. Can an instructor of an ATO give recurrent training under an EBT programme of an airline? Theoretically, yes. An airline can outsource or subcontract the actions of training, but not the responsibility!

If an airline chooses to subcontract the recurrent training to an ATO, the airline should invest in the training and standardisation of the instructors. And not only as a start but on a continued basis. There must be a mature relationship between the airline and the ATO and well-established arrangements that allow the programme to evolve. The instructor standardisation programme shall incorporate airline policies.

It is not impossible but adds a layer of complexity.

Implementation Process

Before deciding to go ahead with the EBT, an airline must know why it is doing it and take all advice it can!

Once an airline has decided to go down the route of using EBT, there are things to watch out for, traps to avoid and strategies to ensure a smooth implementation.

The first issue is appreciating the magnitude of the task and who the stakeholders are.

The whole process is for the pilot's benefit, making sense to get the buy-in from the pilot workforce. If you are a unionised airline, the union and international organisations such as IFALPA and ECA can be of great help. Both IATA and ICAO are making this clear, and it is recommended not to take this task lightly. Pilots may see this as a potential threat since it is a change! Especially if your company has a reputation as a checking airline, this will be viewed as a different form of checking, and it will take some convincing to the contrary!

The second group is the regulators. In some regions, this is already done on a higher level, as in Europe with EASA, but in other parts of the world, the regulators are still strongly linked to their country and sometimes hardly speak with the neighbours. EBT is quite a giant leap for these regulators as it is the first step toward outcome-based regulation, and they transfer a lot of responsibility to the airlines.

The airline needs to explain why and how it wants to do this and take the regulator with them on the journey. The regulator needs to build up trust in the system, and that takes time and effort.

Where the process went wrong, it was mainly linked to the instructors: inadequate training, rushed implementation and poor understanding of the underlying fundamentals.

Whilst age is not a determining element for the successful cross-training of an instructor, a certain cynicism that comes with age and experience is not helpful. With careful preparation and spending the necessary time and effort, it is within reach for most.

But maybe not for all. And that is also to be considered: how do we deal with that?

Communication is key. Consider that the workforce (pilots and instructors) could be hostile or at least doubtful. There is no use in starting the implementation if you do not have the buy-in from your workforce. You need to explain why you want to do it, and the benefits for the pilot. What does it mean for the instructors? What is the gain for the company?

The pilot representatives can be a great help or a severe blocker if they have not been informed thoroughly.

Implementing EBT is a serious project that requires serious change management! But this should not scare you away from it! While there are complexities, especially in instructor standardisation, the baseline programme in itself should be reasonably easy to implement. It is almost plug-and-play. Making variations within the baseline programme based on the company's SMS data has its complexities. But for some operators, just running the baseline programme with a grading system as set out in the EASA Guidance Manual is a good stepping-stone while leaving the variation element until more maturity has been garnered.

Before starting, get all the info, prepare the communication and start informal talks with stakeholders. Talk to others who did it or get professional help from outside.

Managing Data

EBT is all about training more and checking less, and yet we are collecting enormous amounts of data. This data collection is undoubtedly an area that needs careful consideration.

First, there is the protection of data. We know that collecting data is a sensitive matter. And there is no denying that the data can be misused. Some companies have demonstrated that they misuse the FOQA data in a punitive way. One can hardly blame the pilots if they are worried about data from their training to be collected and stored!

Logically, this element is a mandatory item of agreement with the pilot representation. EASA has even included that in their recommendations.

Secondly, there is the use of the data. It looks like many airlines do not invest enough effort in the analysis of the data and may also lack the required skills to do an in-depth discussion or in-depth analysis, especially in looking at the external data. Another area that the airline and the ATO can cooperate is the outsourcing of data analysis. Most airlines have a lot of data but do not have the bandwidth, economy of scale or expertise to do this analysis in-house.

For example, if you look at an incident or accident that happened on the other side of the world, in a different type of aeroplane, it still may be valid evidence to be considered if the root cause was Human Factors.

According to consultants with good insight, some companies pull lots of data, do flawed analysis, and come up with weird conclusions.

Data analysis is undoubtedly an area where more guidance would be helpful. For example, showing how the analysis should look would be beneficial, as it is not easy if you have never seen an in-depth analysis of an event.

We recommended starting with the baseline programme. Refine your inner loop, train the instructors well, get good inter-rater reliability and get the pilots to understand that we are talking a few years to get that sorted out. Once you are confident with that and the programme is up and running, gradually transition your Flight Ops data from the outer loop into the programme.

Additionally, it is essential to remember that measurement of the effectiveness of training programmes must consider measures of efficiency and safety derived from flight operations data sources such as FDM and LOSA. However, results from LOSA or incident investigation are not directly usable for EBT. There is a "translation" necessary in OBs and competencies. So do not be in a hurry but move slowly and gradually.

As a final word on data, the ab initio world is worth looking at. Flight academies have learned, since the introduction of MPL, to work with competency-based training. The good schools, independent of MPL or ATP integrated, are now aligned with the whole CBT philosophy and have a massive amount of data collected on their students. It is frustrating for them to see that most airlines are not interested in taking these data into account to continue building the training for the new hire pilots. It looks like a waste to ignore all the excellent work done in the academy, and they want the airlines to be more engaged in the training process that happened before the hiring process.

Enabling Technology

Implementing EBT requires the effective use of software. There are companies that can provide software to manage all the data we need. Some claim to be "EBT compatible" but are not really. But there are enough good ones around to give you a choice. As long as the system gives you good insight on what is happening with the assessments, how the instructors compare to each other and an independent source, such as simulator telemetry and so on, it is okay. Again, before deciding, it may be good to talk to some colleagues of more experienced airlines to understand what you need.

There is also new technology coming available every day. Does it matter from an EBT standpoint? Not really. If the technology reduces the instructor's workload and gives more time to observe, technology can be helpful.

Technology cannot replace an instructor but can help to allow them more time to observe and determine the root cause.

Conclusions and Recommendations

Resources

https://www.nlr.org/evidence-based-training/ebt-global-review-2020/

https://ebt.solutions/guidance-material

https://www.easa.europa.eu/oversight-guidance-transition-ebt-mixed-checklist

https://www.iata.org/contentassets/c0f61fc821dc4f62bb6441d7abedb076/ebt-implementation-guide.pdf

https://www.skybrary.aero/index.php/Evidence_based_training_(EBT)

https://www.eurocockpit.be/positions-publications/implementation-evidence-based-training-ebt

https://store.icao.int/en/manual-of-evidence-based-training-doc-9995

A Market-Drive Informational Webinar Series

The CAT Leader Forum emerged from an annual Heads of Training (HoT) meeting which Halldale typically conducts with EASA at the European Aviation Training Symposium – EATS. (This year’s EATS will be 2 -3 November in Berlin)

As there was no live EATS in 2020 because of pandemic restrictions, EASA and Halldale agreed to conduct a virtual HoT meeting.

From that meeting emerged five “HoT” topics facing the airline training community in Europe which warranted deeper discussion.

Led by Capt. Jacques Drappier, JDR Consult, former Vice President of Flight Operations and Training Services for Airbus, and Chair of EATS, Halldale is conducting intensive workshops with about a dozen leading subject experts on each of the five topics. Assisting Capt. Drappier in the series are Allyson Kukel and Rick Adams.

From the workshop discussions, a subset of subject experts are then presenting webinars summarizing the group’s findings, as well as a detailed text report and executive summary.