The limits of general-purpose survey software in regulated consultations

There are hundreds of options for general-purpose survey software for those wanting to conduct a survey. However, regulated consultations require something far more sophisticated to ensure legal compliance. This article looks at the risks associated with using general-purpose software for consultations.

Regulated consultations place demands that generic tools can’t meet

There are a lot of reasons that ordinary businesses, community groups or individuals might conduct a survey. Maybe a business wants to know how their regulars feel about a new menu, or how well received their customer service is. Perhaps an individual would like to get a feel for how their neighbours feel about hosting a street fair, or a community group would like to see what different sections of their community get out of their activities. 

When a survey is conducted, people often turn to generic survey tools to do so. These are general-purpose tools that allow people to enter data through text, checkboxes and radio buttons, and send responses to an administrator every time the survey is completed. Some generic software will even have some basic analytics, letting the person running it see how many people have said ‘Yes’ to question A.

These generic options can be an excellent and cheap (even free for the most basic tools) choice for the purposes listed above. However, they are a less viable choice for a regulated consultation.

Regulated consultations - such as statutory consultations conducted as part of a regulatory process - set a very high bar for:

• Evidential integrity.
• Traceability of inputs.
• Durability of data.

Generic software, which hasn’t been designed for the level of complexity and external scrutiny that a consultation often requires, are rarely able to meet these expectations. GovTech should never cut corners or make assumptions in the way a simple survey tool can. That’s why consultations require purpose-built software, which is designed to meet the standards expected of a crucial part of the regulatory process.

Survey tools treat responses as data points, not formal records

One of the key problems that occur when generic survey tools are used for processes they weren’t designed for is a lack of context and clarity. Where survey tools may be used for data collection, consultation software must treat each input as a formal piece of evidence that must be presented as part of the decision-making process. 

When responses are treated as simple data points, common issues include:

A loss of structure

Standard survey responses are often stored as flat datasets, with inputs separated from the questions that prompted them or from the broader consultation framework such as supporting documents. This separation creates a risk that responses are interpreted outside their intended context, particularly when consultation outputs are later summarised or referenced in reports.

When decision-makers later review these inputs, it can become difficult to trace how individual responses related to specific proposals or questions. This weakens the evidence and may even render evidence unusable, which is likely to render the consultation invalid.

Concerns around metadata and provenance

Some of the most widely used generic survey tools capture only basic submission data. They rarely preserve richer contextual information about how questions were presented, how consultation materials evolved over time, or how responses were categorised and analysed. 

Without this metadata, organisations may struggle to defend the consultation process if questions about validity were brought into legal question.

How inputs should be treated in a formal consultation

Data from a formal consultation must be collected in formats that maintain clear links between questions, supporting materials and inputs. Metadata - including respondent types and engagement channels - must be preserved in a way that ensures that they can be viewed as easily three years after a consultation as they could on day one. 

Purpose-built platforms like Citizen Space can be used to ensure that responses are embedded within a wider consultation framework, which gives analysts the ability to view data both as simple data points and within context.

Auditability and traceability are difficult to retrofit

A large part of why ensuring the above is so important is that a regulated consultation process must be auditable and traceable. That means showing where data came from in a way that shows inputs were:

• Collected in a defined consultation process.
• Submitted in response to specific questions and that the questions or context have not been altered since they were submitted. 
• Attributable to identifiable groups or individuals. This is particularly important in consultations that are vulnerable to cyber attacks.
• Traceable from submission to analysis to reporting.

The final point is particularly important to stress. It is not enough for a regulated consultation to deliver a report that says ‘some residents said they had a problem with A’ and expect auditors or legal representatives to take their word for it. Instead, they must be able to present the evidence of which residents, how many there were, and why their submission was interpreted to be in agreement with a statement. Traceability is key to the defensibility of a decision made as the result of a consultation process.

This can be tough to do with a survey tool that simply exports a spreadsheet or delivers a simple numerical answer to ‘how many people said yes to question 1’. 

Instead, a platform must have built in audit trails to ensure that decision-makers are ready and able to demonstrate the strength of their evidence.

Accessibility and inclusion requirements expose tool limitations

For a regulated consultation process to be legal, it must meet the required minimum standards of accessibility. In many cases, facilitators will also be required to show that the consultation was appropriately inclusive. That means demonstrating that all affected stakeholders had an equitable opportunity to take part in the process and make their voice heard, regardless of demographic. 

While some generic survey software will be equipped to meet some online accessibility standards - such as being WCAG 2.2 compliant - not all will. Even where those basic standards are met, such as enabling screen readers, that is far from the be all and end all of accessibility requirements in a regulated consultation. 

Requirements differ depending on sector and context, but examples include:

• Language requirements, such as having translated versions of supporting documents (and not simply relying on automatic translations which may be unevenly successful between languages).
• Requirements around the simplicity of language, e.g. using plain english. Even though you can generally control how questions are phrased, things like the privacy policy of the site may not meet the standards required.
• Being available in multiple formats, such as mobile or paper versions.

Ensuring inclusivity is sometimes even more complex than accessibility. If a consultation is required to show that a representative number of people from all relevant demographics have taken part, then it is essential that there is an easy way to monitor this during the consultation process. It is rare for generic survey tools to have anything like this level of information, particularly for it to be available during the consultation as opposed to realising afterwards that the process has not met requirements.

Long-term retention and reuse reveal structural weaknesses

The final and perhaps most important reason that generic survey software is not suitable for a regulated consultation are their failures to retain data over long periods of time. 

Regulated consultations are important legal documents, which demonstrate that public bodies and organisations have met their obligations to involve relevant stakeholders in the decisions that affect them. These decisions are not always universally popular, and therefore often come up against legal challenges. Solicitors may pick apart a planning decision made on the basis of a consultation by attacking the legitimacy of the consultation itself, and this may happen years later.

For example, if a consultation is conducted on a Local Plan and a certain area is designated for an industrial site, it may be five years later than anything is actually built. Nearby residents that are unhappy with the project may seek to challenge the development, and their solicitors may argue that these residents were not adequately consulted. 

If the consultation was conducted properly, it should be easy to evidence the concerns that were raised and how these concerns were addressed and mitigated. However, if a generic survey tool was used and all that remains are a few disparate spreadsheets, it will be far harder to prove that the consultation met legal requirements. 

Mistakes like this can cost large amounts of time and money, and could be easily avoided by choosing software that is fit for complex use.

Why organisations move to specialist engagement platforms

Given the potential risks of using general-purpose software for regulatory processes, it is clear why organisations have moved en masse to using specialist engagement platforms. A small saving made by using a basic tool can end up costing thousands of times that saving if a consultation is found to be non-compliant due to an issue with the data.

Platforms like Citizen Space can:

• Ensure that all responses are contextualised and have the appropriate metadata to demonstrate that context.
• Give clear audit trails that can be easily followed by even a lay-audience, removing the need for technical expertise to see how decisions were made and why.
• Implement accessibility and inclusivity standards as a default into all consultation templates
• Ensure data remains clear and traceable even years down the line.

Of course, the points made in this article largely show how the platform can remove the legal risks associated with generic software. It is also important to note that purpose-built software such as Citizen Space often has more sophisticated technical abilities too. For example, using Citizen Space Geospatial, supporting evidence can be provided and inputs enabled using GIS data, which enables place-based engagement.

‍

Citizen Space is the go-to govtech platform for engaging with citizens, managing large scale government consultations and simplifying statutory processes. If you’d like to learn more about how our software can be used to run regulatory consultations, book a free demo today. 

Sign up for the Delib newsletter here to get relevant updates posted to your email inbox.

‍