Asking the right questions

Robust question banks are foundational to high integrity assessments, and are core to Aspeq’s business.

Aspeq maintains question banks for over 25 international clients. Aspeq offers its clients flexible options when it comes to developing, managing and maintaining question banks. Its services range from sourcing subject matter experts to write questions from scratch, through to maintaining and delivering high-stakes assessments through a secured portal.

Sourcing global expertise

Aspeq has a global network of over 80 technical and non-technical subject matter experts (SMEs) write, review and test questions.

These experts are highly qualified and often leaders in their fields, with the necessary certifications and experience to provide their industries’ knowledge and regulatory requirements. Often, SMEs are required to hold a practising license, and Aspeq also completes a recruitment process with careful security and conflict of interest checks to ensure potential SMEs are trustworthy and of high standing.

Moreover, new SMEs are paired with experienced SMEs, who have a trusted relationship with Aspeq, throughout the question development process.

Aspeq’s network of experts are based throughout the world, which gives Aspeq the advantage of tapping into local talent as well as accessing global expertise.

As Aspeq’s Assessment Coordinator, Rebecca Premraj, explains “we have SMEs all over the world, and each country has its own regulatory requirements, which is important to understand when preparing an assessment. Likewise, we’re not restrained by geography and are able to hire the best person for the job, regardless of where they are based.”

The art of question writing

Whatever its clients' requirements, Aspeq has a robust process for writing, editing, reviewing and testing questions. 

“A new question goes through at least six pairs of hands, so four technical SMEs and two non-technical SMEs review each question,” says Rebecca.

Question bank development begins with a syllabus, which outlines the topics candidates should be tested on. The exam structure and requirements are captured in an exam blueprint detailing how questions are allocated across the syllabus.

After a question is written, a non-technical editor with expertise in copywriting, editing and style guides will work with the technical writer to review the question for clarity and consistency. If an

The question continues through several rounds of iteration before going to an independent moderator who is a technical expert, followed by a proofreader.

The proofreader who ensures each new question is clear, grammatically correct, and aligns with the industry’s style guide.

“This is looking for things like, how are units of measure represented in this industry, and ensuring that the questions reflect that to prevent any possible confusion.” 

Finally, before being incorporated into an assessment, testing is done by a technical expert who was not involved in the question development process.

“In this step, it’s really important that the SME is presented the questions in the same environment as candidates will experience them, so they actually sit down and review the question in an exam environment as if they were a candidate,” Rebecca says.

This process catches any possible ambiguities or potential technical issues with the question.

Stress testing

Once new questions are incorporated into an assessment, they are closely analysed by Aspeq’s in-house statisticians.

“Wexpected. This highlights any questions that are performing worse or better than expected, which indicates they may need to be reviewed.”

Likewise, candidates are also able to comment on questions and these comments help inform Aspeq’s analysis of how the questions are performing.

“All of this information is put into a comprehensive report that we share with our clients, and we repeat this process on a regular basis.”

“This keeps us very busy, but it’s important as it prevents the questions from getting outdated or over used, and minimises the risk of questions being passed on from candidates throughout the years. Most importantly it helps identify problematic questions which need to be updated to ensure candidates are not being disadvantaged and clients are getting a robust assessment.” 

Meet the team

Aspeq’s question banks are maintained by the Assessment Services Team, consisting of ten specialists working out of the organisation’s Lower Hutt office.

The team is led by Manager Duncan Macgregor. Alongside the assessment coordinators, the team also has a technical specialist, an in-house statistician and a support specialist.

While specialists can work across different client portfolios, they tend to split the work by client category, such as aviation or health.

“This means that we can be there for our clients as much as possible. We build a relationship with our clients who get to know us and we get to know their subject matter and requirements intimately. However, we are a close knit team and often jump in to support each other,” explains Rebecca, who is a member of the team.

“I’m proud to be part of this team. We deliver exams to the highest integrity while keeping our clients and candidates interests at the heart of everything we do.”