Saturday, October 17, 2015

Assessment of Collaborative Problem Solving Skills in Market Research Industry

The collaborative problem solving teaching and assessment activity described here is in a business environment - specifically for the services industry of Market Research. The activity is designed for someone who is responsible for management or implementation of a typical quantitative market research project where data is collected from large samples using survey forms. For the purposes of this assignment, let us focus on three key roles that exist in a market research organization:

  •       Client servicing: Responsible for securing new business and managing the client deliverables (Submitting proposals, designing survey forms, data interpretation, report and overall project management)
  •       Operations: Responsible for data collection (Recruit survey participants directly or through vendors who collect data on their behalf, script questionnaire, provide incentives, check data quality)
  •       Data Processing & Analytics: Collate all the data collected, clean up data if required, generate tables, statistical models used for interpretation and provide it to the client servicing team

Each of these roles is unique, requires specialized skills and successful implementation requires full participation and extensive collaboration by all team members. Though the process is same for most projects, the research objectives vary in terms or scope and complexity and the outcome is uncertain.

As we are dealing with graduates or post graduate students (though in the early stages of their careers), the participants are assumed to have the requisite ICT skills in order to be able to use the online collaboration platform.

The objective of this activity is to train the employees on market research project management and assess them on the quality of the research report delivered and their collaborative problem solving skills. As all members working on a project bring completely different skills and have distinct roles to play, collaboration is fraught with risks of inadequate understanding of responsibilities of all players and communication gaps. Nevertheless, the extent of collaboration does impact the quality of the final research report.

The teaching and assessment is designed as a role play activity within an office environment, and is carried out over a period of three to four weeks using an online collaboration platform that also allows for project management.
The training is designed for employees in their first year of employment. This is designed as a group project with each group comprising three members, representing the three roles described earlier. The training can be conducted by a senior staff member who can double up as a “Client”. A trainer should be able to work with 4-6 groups in a training session ~ 12-18 employees in total.
The rationale for the extended period of training is to simulate real work scenarios: giving time for client interaction, source market information, design a questionnaire, data processing, report writing and use the online collaboration platform to implement the project.
An example of a platform that can be customized for such activity would be
The customized online collaboration platform records all the activities of trainees as they are tasked with execution of a market research project. All activities of the market research project will need to be carried out barring actual data collection by the operations team. The need for data collection can be eliminated by selecting for training the specifications of an old project for which data has already been collected. Here, the trainer will provide the raw data collected to the analytics team to generate tables and statistical models. The trainer can adjust the level of complexity of the project to a level that is anticipated for employees in junior to middle management. Teams can be assigned mentors in case they need guidance on the technical aspects of the project.
The trainer must provide challenges that necessitate interaction between the three roles in each group and are representative of those encountered by the employees on an ongoing basis as a part of their work. Some examples of such challenges could be:
·         - Summary analysis half way through the project
·         - Additional analysis at end of the project
·        -  Reduction in the cost of data collection
·         - Modification in sample design
·         - Poor response to survey
·         - Errors in data collected

A CPS assessment rubric has been designed for this activity that takes into account both – Social and Cognitive dimensions and relies on peer and self-assessment. Some components of the rubric are specific to certain roles in the organization (and hence on the assignment) and this has been indicated.

The online collaboration platform allows trainees to rate the contributions made by themselves and their colleagues in terms of usefulness by clicking on vote buttons. The trainer (representing a client) also rates all the trainees. The voting scale proposed is as follows:

This contribution is
+2: …vital to the successful implementation of the project
+1: … valuable and improves the quality of delivery of the project
  0: … somewhat useful
-1: … not useful
-2: …misleading or of no relevance to the project

The self-assessed score for each contribution can be multiplied by 3 and correlated with the total peer assessed score (as there are 3 assessors for each trainee, including the trainer). The total peer assessed score and self-assessed score and the correlation for each trainee is made visible real time, incorporating an element of gamification but detailed votes for each contribution are made visible only at the end of the assignment. A high positive score and high correlation between self and peer assessment indicates that the performance of the trainee and expectations of other team members is aligned. The rationale for not sharing detailed votes is to avoid conflict/ finger pointing in the midst of training while the overall score will give encouragement and an opportunity to reflect in an event the scores are not meeting expectations.
The trainer (or 2-3 mentors) will review the chat history and assess the performance of the trainees on a rubric designed for measuring social and cognitive dimensions.
Thus, the assessment is quick, uses multiple sources (self, peer and expert view) without requiring too much time and can be benchmarked for the entire batch of trainees and compared with historical data. The assessment is anticipated to promote learning by making transparent the thought and action of the trainees via the online platform. Areas of strength as well as those for improvement can be identified for each employee at the end of the training.

As each group submits the final research report to the “client”, the report is evaluated on the following parameters by the trainer (and 2-3 mentors):
·         Meeting research objectives by answering client’s business issues
·         Timeliness of delivery
·         Quality of presentation
·         Quality of report submitted

In providing feedback, the trainer should juxtapose the quality of the report with the CPS assessments for the individual members of the group to pinpoint presence/ absence of collaborative actions that contributed to the success (or failure) in delivering a project report that meets client expectations.

Given the workplace setting, we are not looking for a one-off performance but a habit forming change in the workplace. The assessment brings to fore aspects of performance that will impact delivery of real business projects. Providing feedback for improvement of specific collaboration skills makes the assessment authentic. Once the organization has identified the level of learner on CPS skills, the subsequent learning needs can be identified. The training can be repeated annually at entry and mid-level executives.

Griffin, P. (2012). Assessment and Teaching of 21st Century Skills. Springer Science + Business Media.
 Brief intro to 21st century skills

No comments: