Questionnaire Pilot Test To Evaluate The Responses, Behaviour And Reaction Of Audience

Question

Task: Reflection on survey administration (Cover the questions below)

• Description of the steps/procedures in conducting the questionnaire pilot test
• Justification of the sample size of 10 used for the pilot test
• Your reasons for selecting the particular method of survey administration used (e.g. telephone, online or face-to-face)
• What were the advantages and disadvantages of using this method? Check Week 6 Lecture slides on VU Collaborate for ideas
• Was the administration procedure a success i.e., did respondents understand the questions (look confused or ask questions)? was there missing data (i.e., skipped questions)? on average, how long did it take for respondents to complete the survey? any other challenges or difficulties?
• What suggestions would you make for improving the administration procedure used? Or would you recommend choosing a different administration technique in the future and why?

Reflection on the questionnaire (Cover the questions below)

• What, if any changes, would you make to the questionnaire before its final distribution to a large group of respondents? Provide justification for these changes.

{Hint: The changes may concern the layout/format, the text of the questions, the order of the questions, the format of the questions (e.g. open/closed), the scales, the length of the questionnaire, typos/errors, etc. For ideas, you can use the Questionnaire Checklist (available on VU Collaborate > Learning Space > Assessment information > Assessment 2B).

• If you don’t recommend any changes, you need to justify why the layout/format, the text of the questions, the order of the questions, the format of the questions (e.g. open/closed), and the scales are designed appropriately.

Overall reflection on lessons learned

• On reflection, what were the key lessons you learnt from the survey administration process?

Answer

Introduction
As instructed for preparing current Questionnaire pilot test, after being divided into respective groups, we students staying within our respective groups brainstormed and prepared a questionnaire to be asked to a group of people who have recently dined in a boutique hotel in the span of last six months. Before actually conducting the survey and spreading the questionnaire among the real audience, we conducted a pilot study that involved circulating the formulated questionnaire among a sample audience of 10 to understand their views and get reviews on the questions framed. The purpose of the pilot study was to judge if the questionnaire prepared by us was easily understandable by the audience, covered only the necessary questions and asked for unbiased views. The pilot test would give us as a team and me as an individual a chance to gauge the responses, behaviour and reaction of the audience and take corrective measure to improve the questionnaire for the final survey. Through the course of the report, the process of preparing the questionnaire and conducting the pilot study would be analysed so that the pros and cons of the methods used can be weighed. Moreover, the choice of the survey administration method used would be critiqued and some recommendations would be provided for undertaking necessary changes in the questionnaire pilot test.

Reflection on survey administration
The first thing that we as a team and I as an individual team member did for conducting the questionnaire pilot test is to check whether the instructions we have given to the sample audience and the questionnaire formulated by us is comprehensible or not, as it would determine the authenticity of their responses. Post it, we ensured that the sample audience chosen were relevant and aware of the concept of a boutique hotel and have even dined there in a recent six-month period. The questionnaire was circulated and the responses and actions of the sample population were noted down as it would provide as a base for understanding the loopholes in the questions framed. The responses were gathered and analysed to understand the degree of biasness that the questions probed.

As I mentioned above, Questionnaire pilot test is a means of small-scale pre-testing to understand the relevance of the questions and the level of comprehensibility of questions asked. Therefore, we as a group opined that we should select only ten people from the relevant audience group for the test. The choice of five audiences was also an option; however, we thought that it would be too small a group to gather results from. Therefore, as per our opinion, group of them was fine as we could include people from different age groups above 18, different economic background, gender and frequency of dining at a boutique hotel. This group of ten people would help us in understanding if the questions were easy to comprehend by all types of sample audience or not.

After the choice of a number of people in the audience group, we decided to choose the online method as the survey administration tool as it would ensure a wider reach among the population. The face-to-face or telephonic interview would be a more cumbersome process for us involving higher time and cost due to which it was eliminated and the online method was chosen. As per our group opinion, the online method would be most suitable for questionnaire pilot test as it would assure higher response rate, higher flexibility, convenience, reduced cost and real-time access to the responses generated which makes the process of analysis effortless with the use of online tools. However, we are aware of some the certain shortcomings that we might have to face using this method. Some disadvantages are – delayed responses, cooperation issues, lower validity, lack of interest in answering open-ended questions and disinterest of the audience.

After weighing the pros and cons of the survey administration method, we decided to go ahead with the online tool and circulated the questionnaire pilot test using Google Forms among the ten people. The reasons for choosing the online survey were increased flexibility and reduce time, and cost considerations. Moreover, the choice of face-to-face or telephonic interview would make the process tedious and difficulties would arise in the analysis process as the data would be in text and writing. This also arose the possibility of biasness or human mistakes of the interviewer. However, on the other hand, the data of the responses generated would be stored online eliminating human errors and even making analysis easier with the help of online tools and surveys.

For the majority part, if considered, I thought that the test was a success as seven people responded quickly and did not face issues in understanding the questions. However, the rest three got confused in the questions that required them to rate and lost interest mid-way as they thought that the questionnaire pilot test was too long and time-consuming. On average, respondents took time to answer the questions as there were too many questions and they skipped several rating questions. One issue faced during Questionnaire pilot test, was that some respondents were clueless as they could not recall their dining experience because either it was too long or they were disinterested.

As per my views, we should go ahead with the online tool however, the number of questions needs to be reduced and demographic questions need to be placed in the starting. I feel that we have added certain repetitive questions, which can be omitted to make the questionnaire more concise. This would ensure that the sample population does not lose heart in the middle of answering the questions.

Reflection
From the pilot testing phase, I would like to make certain changes in the questionnaire as I opine that it would ensure higher comprehensibility on the part of the respondents and even enhance the process of an online survey. The first thing that I would like to change in the questionnaire is to reduce the number of questions. On a bird’s, eye view, the number of questions seems to be only 17, however, the questions between 6 to 10 are long as the respondents have to rate the dining experience on several parameters, making the process tedious and lengthy. This suggestion from my end came after realizing that people were becoming disinterested in answering so many questions leading to incorrect or delayed responses.

Secondly, I would like to change the layout of the questions so that the questionnaire pilot test seems to be more appealing to the respondents. For example, I would want to shift the demographic questions above so that the relevance of the respondents can be judged in the beginning. Moreover, I would like to re-frame the third question, as the above question seems to be overlapping. The third question would be changed to the frequency of the respondents dining in the boutique hotel. Thirdly, the scales of the questions provided between questions 6 to 10 would also be altered. Respondents seem to be confused with seven-rating scale, as they do not understand the meaning of all. Therefore, the scale would be reduced to 5 ratings where – one would signify strongly agree, two would agree, three would be neither, fourth would disagree and lastly fifth would strongly disagree. I think that change in the scale and making it more descriptive would enhance the responses with people increased understanding of the people. With changes in the scale, the number of options in the questions between six to ten would be reduced as they seem to be repetitive and cumbersome. For example, two options in questions number eight – “I felt thrilled about the new dining experience” and “The dining experience was really enjoyable” are repetitive with respondents not wanting to answer it.

What are the lessons you have learned from the study of Questionnaire pilot test? Write a reflection.
The survey administration process for Questionnaire pilot test was very enriching and learning experience for me as an individual as it taught me several important aspects. I gained knowledge about the various survey methods that could be deployed for obtaining responses from the sample population. I also learnt the skill of judging, which survey method would be effective and helpful in which circumstances making my sense of judgement strong. Most importantly, I learnt how to prepare a questionnaire that would be easy, comprehensible and concise making the survey process effective. I understood that the questionnaire pilot test should not be elongated but concise so that the respondents can answer them quickly. Another factor that I learnt is that the questions should not probe biasness and should be neutral to gauge the correct responses from the sample audience.

Another thing I learnt during the survey administration process for this Questionnaire pilot test was teamwork. We were divided into groups to make the questionnaire and hence, it involved team efforts and group brainstorming sessions. Hence, I think that at the end of the process, I have developed more patience and understanding power to act on people’s suggestions and ideas. In future, as I aspire to work in a multinational organization, teamwork would be an important skill for me as they would expect me to be a team player. Moreover, an understanding of the survey administration process would also foster growth for me, as it is required in organizations as well. Hence, the entire process was enriching and elevating for me.