safety by design: Product development processes

Safety By Design - By eSafety Comission

Factors to consider

Embedding safety considerations in product design and development helps to prevent and reduce online harms.

Consider integrating formal safety reviews – including consultation and testing – into the design process from the beginning, and through the lifecycle of the product, platform or feature.

These reviews should include employees from teams across the organisation who are responsible for online safety, including the executive team.

Initial design

Some factors to consider when developing a safety review process:

• scenario testing for known types of risk and harm and known techniques used for harm and abuse

• new forms or techniques of abuse

• assessment of false positives or negatives (in moderation processes or reported abuse)

• internal safety vulnerability scans and penetration testing

• external safety vulnerability scans and penetration testing

• user behaviours, needs and impacts for at-risk groups

• health and wellbeing impacts on employees, workers, community moderators, and users.

Formal safety reviews

Formal safety reviews should be conducted throughout the lifecycle of online platforms and services. The factors to consider in safety reviews often change, so these processes should be frequently refined, particularly as new updates are released.

The types of online harms and the techniques and tactics that abusers use will play an important role in scenario testing and are covered extensively in the Online Harms module.

Good practice for safety reviews

Timing

Each stage of product development presents an opportunity to conduct a thorough safety review.

Suggested timing for safety reviews include:

• pre-development

• during development

• pre-launch

• post-launch review of all features and functions

• post-launch review of new features and functions

• post-development reassessments

• during platform updates or refreshes.

Scenario testing

Safety reviews should include scenario testing:

• of specific edge cases, such as unusual user behaviour or incidents that require special handling

• across all channels, features and tools

• within different regions and jurisdictions.

Analysis

Safety reviews should include analysis of:

• patterns of behaviour and network effects, focusing on abusive actors

• online signals such as metadata and traffic signals

• behavioural signals including patterns of interaction – this includes search activity, group membership and activity, violation indicators (such as reports and connection activity or friend requests), content creation and sharing, profiles and accounts

• behavioural and online signals for at-risk groups.

Environmental scanning

The context your platform or service operates in constantly changes, so safety review processes require:

• rapid assessment of new forms or techniques of abuse occurring on the platform

• external research and analysis of new forms or techniques of abuse

• cross-industry practices and information sharing

• understanding the needs of victims/survivors and at-risk and marginalised groups.

Assessment

Safety review processes should assess the effectiveness, accuracy and impact of:

• automated and human moderation systems

• user safety controls and tools

• prevention messaging

• prevention interventions

• reporting systems and processes

• disruption techniques

• detection tools

• automated responses

• feedback systems and processes

• reduction of harms or risks, with a focus on at-risk or marginalised groups

• user confusion or misunderstanding relating to how a product or feature functions.

Testing

As part of the review process, safety vulnerability scans and/or penetration testing should be introduced, both internally and externally.

External expertise

Safety review processes should be assessed by external experts or independent auditors where possible, along with other safety policies and procedures. Seek out innovations and research that will improve safety review processes.

Standards and frameworks

Your safety review should be informed by national and international regulatory frameworks, standards and industry standards.