Developing Efficient Employee Surveys

 A Tested Method for Revising Culture Surveys

John E. Jones, PhD, and William L. Bearley, EdD
Organizational Universe Systems

We have a strong bias in favor of limiting employee surveys to 25-35 items (Jones & Bearley, 1996), but our clients often want to measure the details of their "culture." Our concern is with the feedback process. It is easy to crunch survey numbers these days, but it is never easy to facilitate feeding back large numbers of statistics to work groups. People either become lost in the mass of results, or the feedback is presented in terms of broad, general indexes. This brief paper spells out a method we used to work with a client organization to shorten their culture-survey instrument while improving its interpretability and, therefore, its usefulness.

The argument for a shorter version of the instrument was presented to the organization development staff. A shorter version would be easier to administer, simpler to analyze, and quicker to respond to; this should yield higher rates of participation and make the feedback process operate more effectively. The goal was to construct an instrument that would be directly comparable from the previous year to the present and beyond. The criteria were to retain the organization's Vision Values indexes as the primary feedback-report focus, to make the statistics on items in each index available to work groups, to have about 35 items, plus demographics, and to keep the essential content of the indexes intact.

Methodology

Data from 500 respondents to the previous year's Culture Survey (97 items) were subjected to a principal-components analysis, with factors that had eigenvalues >1 retained. The results showed a degree of almost uninterpretable complexity in the underlying structure of the instrument. Twenty-two dimensions emerged in the analysis. The factors bore little or no resemblance to the five Vision Values indexes on which the feedback was delivered from the previous year's survey.

Second-level analyses of only those items that constituted each of the Vision Values indexes were conducted. These results indicated which items "loaded" highest on the dimension under review. The items on each Vision Values index were rank-ordered according to their loading in the principal-component analysis. This information would be the primary basis for retaining items for the revised instrument.

Alpha reliability analyses were carried out on each of the previous year's Vision Values indexes. The results showed that the long questionnaire had highly stable dimensions. Each item was correlated with its "scale," or index. In addition, the analysis determined the resulting reliability if each item were omitted from the revised instrument. These two sets of results would be the secondary considerations in selecting items to keep in the revised questionnaire.

Finally, the content of the previous year's Vision Values indexes was analyzed for conceptual "purity," that is, how the items appeared to "hang together" in terms of their actual content. This would constitute the tertiary consideration in retaining items for the revision.
 

Results

Based on the combination of considerations-factor loadings, item-index correlation, reliability loss/gain by deleting the item, and content validity-37 items were retained verbatim for the revised Culture Survey questionnaire. The three demographic items were also retained.

The previous year's data were re-analyzed according to the revised questionnaire. In other words, what would the indexes have looked like had the organization used only the 37 items in the previous year? Frequency distributions and graphics were developed to compare the previous year's and the revised instruments as scored on the Vision Values indexes. The results were almost identical. The shapes of the distributions were the same. The analyses led to the conclusion that the new, shorter questionnaire could be used for meaningful, useful comparisons with the previous year's baseline assessment of the organization's culture. The revision was then set up to generate year-to-year comparisons on identical sets of items (Vision Values indexes), by the overall organization and by department. Feedback reports from the revised form include "then and now" graphs.

Alpha reliability estimates were computed for the retained items and new indexes, based on the revised questionnaire, with previous year's results. The table below shows a comparison of the indexes in the two forms of the survey questionnaire.
 

These results indicate that the revised questionnaire is actually as reliable as the previous year's version, with only 38% of its items. Since reliability is directly related to the number of items on the indexes, these estimates strongly indicate that the process that was used to develop the revised questionnaire successfully retained the "essence" of the Vision Values indexes. In fact, the new instrument is conceptually "purer" than the earlier version. The items that were deleted were less uniform in their measurement of the underlying perceptions that constitute the Vision Values dimensions.
 

Reference
Jones, J.E., & Bearley, W.L. (1996). Surveying Employees: A Practical Guidebook. Amherst, MA: HRD Press.