What is being evaluated
The assessment analyzes the consequences of introducing an AI system into a work activity. This could be a tool that advises the professional, organizes shifts, assigns tasks, interprets images, or acts on equipment. The starting point is to describe what changes for people and what decisions depend on the system’s outcome.
A distinction must be made between AI used to assess risks and the risks generated by its use. While both may overlap, they are not equivalent. A preventive tool can save time and, simultaneously, introduce oversights or overconfidence. The assessment must consider both expected benefits and potential adverse effects under real-world conditions.
Define the system and its users
It is important to identify the purpose, the affected population, input data, outputs, and connections to other processes. It is also necessary to describe who uses the system, what training they receive, and what leeway they have to deviate from a recommendation. The same model can pose different risks depending on whether it only produces a draft or if its output triggers action on equipment or people.
Change management allows for the study of these aspects before implementation. Prevention personnel, process owners, technical specialists, and relevant employees should all be involved. Information from the vendor is necessary, but it must be supplemented with knowledge of the job and the specific conditions under which the tool will be used.
Identify damage and failure situations
Situations to be analyzed include incorrect results, incomplete information, data changes, unavailability, and use outside the intended scope. It is necessary to assess what happens if an alert is not issued, if too many alarms are generated, or if an action incompatible with the actual procedure is proposed. The consequence depends on how the result is used.
The analysis also includes interactions between people and the system. A confusing interface, an overly confident response, or difficulty in accessing the source can hinder error detection. It’s advisable to study both normal operation and foreseeable exceptions, shift changes, and times when less technical support is available.
Organization and psychosocial risks
Algorithmic management can modify pace, autonomy, supervision, and the relationship with management. It is necessary to verify whether the objectives or evaluations generate pressure, whether people understand the rules, and whether they have channels to question results. Process observation and consultation complement the technical metrics.
Psychosocial analysis should examine the organization as a whole, not attribute all problems to individual adaptation to technology. A tool that requires reviewing numerous recommendations in a short period can increase mental workload even if it automates part of the task. Measures should adjust the scope, functions, support, and decision-making capacity to the actual demands.
Data, privacy and applicable requirements
Data quality influences the reliability of the results. It must be verified whether the records accurately represent the centers, tasks, and circumstances being studied. Gaps, duplicates, or changes in criteria can create an appearance of accuracy that does not correspond to the available information.
The preventive assessment must be coordinated with data protection requirements and applicable AI regulations. The European AI Regulation establishes obligations based on the type of system and roles, with specific rules and an implementation timeline. This legal classification does not replace the occupational risk assessment. A use that is not considered high-risk under this regulation may require preventive measures in the workplace.
Select and verify preventive measures
Measures may include redesigning the process, limiting functions, improving data, preventing automated actions with critical consequences, and establishing effective human oversight. Training should explain limitations and potential errors, as well as teach how to use the interface. A generic warning does not compensate for a configuration that makes intervention difficult.
Before deployment, representative tests should be performed and acceptance criteria defined. A safe way to continue work if the system fails or needs to be taken offline is also necessary. Tests should focus on safety and health consequences, in addition to computing performance, and should document any problems encountered and the corrections made.
Practical example
In a hypothetical scenario, a company tests a system that prioritizes inspections based on reported incidents. Facilities that report more incidents receive more inspections, while others with few reports appear safe. The evaluation identifies that the system may mistake a good communication culture for increased risk.
The variables, data coverage, and priority criteria are reviewed. Independent checks are incorporated, and the result is not used as the sole basis for planning. The case demonstrates that a correct calculation can lead to an inappropriate preventive decision when the information used does not represent the exposure in a comparable way.
Follow-up after implementation
The assessment must be kept up-to-date throughout use. Changes in the model, rules, data, roles, or connections can alter the risks. There must be designated individuals responsible for reviewing incidents, complaints, deviations, and unforeseen effects, as well as for modifying or discontinuing the tool when the measures are no longer sufficient.
Effectiveness is verified by observing the work and its results, not just by counting users or responses generated. It’s important to assess the quality of decisions, review workload, access to support, and the functioning of controls. Implementation is more robust when the organization can explain what risks it has identified and what evidence shows that they are under control.
