Performance Based Oversight

The regulatory oversight of aeronautical service providers has traditionally been prescriptive. Mario Pierobon investigates the performance-based training and regulatory oversight initiative, particularly the work underway by European regulatory authorities.

Approved training organisations (ATO) and organisations managing flight simulator training devices (FSTD) have traditionally been overseen by their competent authorities in relation to a given set of universally applicable regulatory requirements. Non-compliances have traditionally been identified in relation to this given set of requirements.

Out of the recognition that the modern aviation industry is complex and that the ‘one size fits all approach’ can prevent the comprehensiveness of safety assessments, the European Union Aviation Safety Agency (EASA) has been promoting performance based oversight in order for authorities to exercise their oversight functions not just against a fixed set of requirements but also by considering the inherent risks of the organisations under oversight, including organisations operating in the domain of crew training.

“Performance based oversight is a methodology to conduct the oversight functions of an aviation authority based on the definition and assessment of the safety performance of its undertakings. This allows to better target the areas of greater concern or needs,” said EASA. “By taking into consideration safety and performance data, performance-based oversight encourages each organisation to improve their overall performance. This is fostering a better organisation efficiency and a more calibrated oversight. Those beneficial effects are very much appreciated, especially by the aviation industry.”

Beyond Compliance

Very much like safety management systems (SMS) have not replaced the role of compliance management within aviation organisations, performance-based oversight is also not meant to substitute the established compliance evaluation activities of the aviation authorities.

“Performance based oversight does not as such replace the traditional compliance assessment system but represents the development of an integrated tool, adding versatility to the latter. While compliance remains as a foundation to ensure that regulatory requirements are attended to, performance-based oversight implementation supports the authorities in defining their oversight plan to tailor the individual level of attention with a view to optimise its efficiency. In the aircrew domain performance-based oversight is applied for defining and performing the oversight of ATOs and FSTD operators,” said EASA.

One way to conduct performance-based oversight is by looking at the training organisation as a whole, at what the organisation defines as the goals that it sets for itself, and then look at how these goals are achieved. Then it is possible to do the measuring directly by talking to the people within the structure to see how they play their role within the structure.

Regulatory Environment

Current EASA regulations have a mixture of prescriptive and performance-based elements. “The current regulatory framework foresees that the determination of oversight planning cycles depends on the nature of organisations, the complexity of activities, the results of past certification and/or oversight activities and shall be based on the assessment and management of associated risks,” said EASA. “On the organisations’ side the management system must be able to define and monitor the safety performance in order to provide the necessary information to the national aviation authorities (NAA). The analysis of these elements, including the assessment of specific safety performance data, will then determine the frequency and scope of the oversight plan for each organisation, avoiding applying a ‘one size fits all’ solution. As a result, the resources for oversight will be deployed on a better need basis, achieving more effectiveness but also an increased efficiency.”

A novelty is represented by Part-DTO (declared training organisations) which has been recently added to the European regulatory requirements. “This type of training organisation can deliver training for general aviation licenses and associated ratings, such as the light aircraft pilot license (LAPL) and private pilot licenses (PPL). To start operations, the organisations simply declare their existence, place and scope of activities to the NAA. No upfront certification occurs, however DTOs are required to present annual reports on activity and safety performance. This data integrated with other information, such as occurrence reporting, will support the NAA in determining a performance-based oversight planning to allow focusing on the most critical cases,” commented EASA.

Expertise Development

A limitation exists with regard to performance-based oversight reaching its full potential in that it requires a level of expertise, while instead in many parts of Europe the approach tends to be eminently bureaucratic because of the limited resources available. When operators, ATOs and other aviation organisations are required to do something, they will develop the capability and demonstrate to the authority they are performing in accordance with a given requirement, but conversely the authorities may have no capability to understand or even to interrogate those claims because they simply do not have the expertise.

There are multiple elements that we should assess in a performance based way and surely training programmes and assessments given by authorities or the efficacy of training programmes are easier to evaluate because it is possible to set up a series of learning objectives within the ATO for whichever course you are delivering and it is easier to find expertise to measure those efforts by looking at student performance and data collected based on student performance. Indeed, there are some parts of Europe where there is a significant lack of expertise, whereas there are some major authorities which have expertise as it is much easier for them develop.

One way to develop expertise is buy it in, i.e. sourcing people with experience in the given roles and then develop that experience into expertise for performance-based oversight. If the expertise is not bought in it is quite difficult to implement performance-based oversight especially when there is a need for qualitative judgements as to how effective certain measures are. It may be possible to conduct the oversight activities at a bureaucratic level to a degree, but when it comes down to the real delivery of performance-based oversight - which should include a programme of audits and inspections – this by definition must be done by people with expertise. The problem is that the certain authorities, which are government institutions that are publicly funded, may find it difficult to sustain the financial commitment of sourcing the needed expertise. In general terms, most of the authorities, apart from the major ones, are purely prescriptive, they are audit based and this is essentially driven by financial necessities.

According to EASA every NAA is in the position to apply performance-based oversight, however the level of implementation may vary from a basic performance-based oversight to advanced systems. “Mainly the differences are depending on key factors such as availability of data, IT tools, qualification of inspectors and general level of maturity of the organisations. There is an on-going standardisation effort to share and align practices in Europe in order to achieve a more harmonised level of implementation,” said EASA.

Standardisation

Indeed, EASA is supporting standardisation across the national aviation authorities and this in the long run will be beneficial to a more harmonised implementation of performance-based oversight across Europe. EASA knows that where expertise lies is within the national authorities. So, if the European model had to be taken to its fullest extent then there should be a team of experts working directly or indirectly for EASA made available to state authorities, as the US Federal Aviation Administration (FAA) does that in certain specific areas. This would be something really positive because if a particular state had a particular need with regard to performance-based oversight it could seek help from such a team of experts under the leadership of EASA. Of course, the funding of this mechanism would need to be determined.

Within an organisation’s management system there should be hazard identification and risk assessment. The way the authority provides oversight of this activity is important and cannot be done bureaucratically. While performance-based oversight has considerable potential, much of its development depends on the people that are recruited into the authorities to accomplish the performance-based tasks. It is a specialised role, so the ideal is to have a pool of experts able to make assessments and judgements based on experience on what organisations have set out to do and really measure that effectively.

Published in CAT issue 6/2018