AJC Survey of American Jewish Opinion Methodology
This survey was administered on behalf of the American Jewish Committee (AJC) by Knowledge Networks, and data collection utilized the proprietary KnowledgePanel®, a web-based respondent panel designed to be representative of the United States.
Questionnaires were completed online by panelists who were invited to participate due to meeting either qualification criteria (in a previously administered profile interview): they identified their religion as “Jewish” (75 percent of respondents) or they said they considered themselves Jewish for any other reason (25 percent of respondents).
All panelists who met the qualifications (1,356 in total) were invited to participate in this survey, and 1,074 (79 percent) completed the questionnaire during the period of March 14 – 27, 2012.
Knowledge Networks calculates the margin of sampling error as plus or minus 4.8 percent at the 95 percent confidence level.
Knowledge Networks Methodology
The following description of the panel recruitment and survey administration was provided by Knowledge Networks.
Knowledge Networks (KN) has recruited the first online research panel that is representative of the entire U.S. population. Panel members are randomly recruited through probability-based sampling, and households are provided with access to the Internet and hardware if needed.
Knowledge Networks selects households by using address-based sampling methods; formerly, KN relied on random-digit dialing (RDD). Once households are recruited for the panel, they are contacted by e-mail for survey taking or panelists visit their online member page for survey taking (instead of being contacted by phone or postal mail). This allows surveys to be fielded very quickly and economically. In addition, this approach reduces the burden placed on respondents, since e-mail notification is less intrusive than telephone calls, and most respondents find answering Web questionnaires more interesting and engaging than being questioned by a telephone interviewer. Furthermore, respondents have the freedom to choose what time of day to participate in research.
Documentation regarding KnowledgePanel sampling, data collection procedures, weighting, and IRB-bearing issues are available at the below online resources.
When Knowledge Networks began recruiting in 1999, the company established the first online research panel (now called KnowledgePanel®) based on probability sampling covering both the online and offline populations in the U.S. Panel members are recruited through national random samples, originally by telephone and now almost entirely by postal mail. Households are provided with access to the Internet and hardware if needed. Unlike Internet convenience panels, also known as opt-in panels, that includes only individuals with Internet access who volunteer themselves for research, KnowledgePanel recruitment uses dual sampling frames that includes both listed and unlisted telephone numbers, telephone and non-telephone households, and cell-phone-only households, as well as households with and without Internet access. Only persons sampled through these probability-based techniques are eligible to participate on KnowledgePanel. Unless invited to do so as part of these national samples, no one on their own can volunteer to be on the panel.
KN employed list-assisted RDD sampling techniques based on a sample frame of the U.S. residential landline telephone universe. For purposes of efficiency, KN excludes only those banks of telephone numbers (a bank consists of 100 numbers) that had fewer than two directory listings. Additionally, an oversampling was conducted within a stratum of telephone exchanges that had high concentrations of African American and Hispanic households based on Census data. Note that recruitment sampling is done without replacement, thus numbers already fielded do not get fielded again.
A telephone number for which a valid postal address can be matched occurred in about 67-70% of each sample. These address-matched cases were all mailed an advance letter informing them that they had been selected to participate in KnowledgePanel. For purposes of efficiency, the unmatched numbers were most recently under-sampled at a rate of 0.75 relative to the matched numbers. Both the minority oversampling mentioned above and this under-sampling of non-address households are adjusted appropriately in the panel’s weighting procedures.
Following the mailings, telephone recruitment by trained interviewers/recruiters begins for all sampled telephone numbers. Telephone numbers for cases sent to recruiters were dialed for up to 90 days, with at least 14 dial attempts for cases in which no one answers the phone, and for numbers known to be associated with households. Extensive refusal conversion was also performed. The recruitment interview, about 10 minutes in length, begins with informing the household member that the household had been selected to join KnowledgePanel. If the household does not have a computer and access to the Internet, the household member is told that in return for completing a short survey weekly, the household will be provided with free monthly Internet access and a laptop computer (in the past, the household was provided with a WebTV device). All members of the household are enumerated, and some initial demographic and background information on prior computer and Internet use was collected.
Households that informed recruiters that they had a home computer and Internet access were asked to take KN surveys using their own equipment and Internet connection. Incentive points per survey, redeemable for cash, are given to these PC (personal computer) respondents for completing their surveys. Panel members provided with a laptop computer and free Internet access do not participate in this per-survey points-incentive program. However, all panel members do receive special incentive points for select surveys to improve response rates and/or for all longer surveys as a modest compensation for the extra burden of their time and participation.
All new panel members receive an initial survey for the dual purpose of welcoming them as new panel members and introducing them to how online survey questionnaires work. New panel members also complete a separate profile survey that collects essential demographic information such as gender, age, race, income, and education to create a personal member profile. This information can be used to determine eligibility for specific studies and is factored in for weighting purposes. Operationally, once the profile information is stored, it does not need to be re-collected as a part of each and every survey. This information is also updated annually for all panel members. Once new members have completed their profile surveys, they are designated as “active,” and considered ready to be sampled for client studies. [Note: Parental or legal guardian consent is also collected for the purpose of conducting surveys with teenage panel members, aged 13 to17.]
Once a household is recruited and each household member’s e-mail address is either obtained or provided, panel members are sent survey invitations linked through a personalized e-mail message (instead of by phone or postal mail). This contact method permits surveys to be fielded quickly and economically, and also facilitates longitudinal research. In addition, this approach reduces the burden placed on respondents, since e-mail notification is less intrusive than telephone calls and allows research subjects to participate in research when it is convenient for them.
After conducting an extensive pilot project in 2008, KN made the decision to move toward address-based sample (ABS) frame in response to the growing number of cell-phone- only households that are outside the RDD frame. Before conducting the ABS pilot, we also experimented with supplementing its RDD samples with cell-phone samples. However, this approach would was not cost effective—and raised a number of other operational, data quality, and liability issues (for example, calling cell phones while respondents were driving).
The key advantage of the ABS sample frame is that it allows sampling of almost all U.S. households. An estimated 97% of households is “covered” in sampling nomenclature. Regardless of household telephone status, those households can be reached and contacted through postal mail. ABS involves probability-based sampling of addresses from the U.S. Postal Service’s Delivery Sequence File. Randomly sampled addresses are invited to join KnowledgePanel through a series of mailings and, in some cases, telephone follow-up calls to non-responders when a telephone number can be matched to the sampled address.
After initially accepting the invitation to join the panel, respondents are then “profiled” online by answering key demographic questions about themselves. This profile is maintained through the same procedures that were previously established for RDD-recruited panel members. Respondents not having an Internet connection are provided a laptop computer and free Internet service. Respondents sampled from the ABS frame, like those sampled from the RDD frame, are offered the same privacy terms and confidentiality protections that we have developed over the years and that have been reviewed by dozens of Institutional Review Boards.
Large-scale ABS sampling for KnowledgePanel recruitment began in April 2009. As a result, sample coverage on KnowledgePanel of CPOHHs, young adults, and non-whites has been increasing steadily since that time.
Because KnowledgePanel members have been recruited from two different sample frames, RDD and ABS, KN implemented several technical processes to merge samples sourced from these frames. KN’s approach preserves the representative structure of the overall panel for the selection of individual client study samples. An advantage of mixing ABS frame panel members in any KnowledgePanel sample is a reduction in the variance of the weights. ABS-sourced samples tend to align more closely to the overall demographic distributions in the population, and thus the associated adjustment weights are somewhat more uniform and less varied. This variance reduction efficaciously attenuates the sample’s design effect and confirms a real advantage for study samples drawn from KnowledgePanel with its dual frame construction.
For client surveys, samples are drawn at random from among active panel members. Depending on the study, eligibility criteria will be applied or in-field screening of the sample will be carried out. Sample sizes can range widely depending on the objectives and design of the study.
Once assigned to a survey, members receive a notification e-mail letting them know there is a new survey available for them to take. This email notification contains a link that sends them to the survey questionnaire. No login name or password is required. The field period depends on the client’s needs and can range anywhere from a few hours to several weeks.
After three days, automatic email reminders are sent to all non-responding panel members in the sample. If email reminders do not generate a sufficient response, an automated telephone reminder call can be initiated. The usual protocol is to wait at least three to four days after the e-mail reminder before calling. To assist panel members with their survey taking, each individual has a personalized “home page” that lists all the surveys that were assigned to that member and have yet to be completed.
Knowledge Networks also operates an ongoing modest incentive program to encourage participation and create member loyalty. Members can enter special raffles or can be entered into special sweepstakes with both cash rewards and other prizes to be won.
The typical survey commitment for panel members is one survey per week or four per month with duration of 10 to 15 minutes per survey. Some client surveys exceed this time, and in the case of longer surveys, an additional incentive can be provided.
Survey Sampling from KnowledgePanel
Once Panel Members are recruited and profiled, they become eligible for selection for specific client surveys. In most cases, the specific survey sample represents a simple random sample from the panel, for example, a general population survey. Customized stratified random sampling based on profile data can also be conducted as required by the study design.
The general sampling rule is to assign no more than one survey per week to members. Allowing for rare exceptions during some weeks, this limits a member’s total assignments per month to four or six surveys. In certain cases, a survey sample calls for pre-screening, that is, members are drawn from a subsample of the panel (such as females, Republicans, grocery shoppers, etc.). In such cases, care is taken to ensure that all subsequent survey samples drawn that week are selected in such a way as to result in a sample that remains representative of the panel distributions.
For this survey, a sample of U.S. adults (18 and older) who self-identify as Jewish by religion or background (meaning no religion but considers self Jewish) was selected.
The design for KnowledgePanel® recruitment begins as an equal probability sample with several enhancements incorporated to improve efficiency. Since any alteration in the selection process is a deviation from a pure equal probability sample design, statistical weighting adjustments are made to the data to offset known selection deviations. These adjustments are incorporated in the sample’s base weight.
There are also several sources of survey error that are an inherent part of any survey process, such as non-coverage and non-response due to panel recruitment methods and to inevitable panel attrition. We address these sources of sampling and non-sampling error by using a panel demographic post-stratification weight as an additional adjustment.
All the above weighting is done before the study sample is drawn. Once a study sample is finalized (all data collected and a final data set made), a set of study-specific post-stratification weights are constructed so that the study data can be adjusted for the study’s sample design and for survey non-response.
A description of these types of weights follows.
The Base Weight
In a KnowledgePanel sample there are seven known sources of deviation from an equal probability of selection design. These are corrected in the Base Weight.
The Panel Demographic Post-stratification Weight
To reduce the effects of any non-response and non-coverage bias in the overall panel membership (before the study sample is drawn), a post-stratification adjustment is applied based on demographic distributions from the most recent data from the Current Population Survey (CPS). The benchmark distributions for Internet access among the U.S. population of adults are obtained from the most recent special CPS supplemental survey measuring Internet access (October 2009).
The overall panel post-stratification variables include:
Study-Specific Post-Stratification Weights
Once the sample has been selected and fielded, and all the study data are collected and made final, a post-stratification process is used to adjust for any survey non-response as well as any non-coverage or under- and over-sampling resulting from the study-specific sample design. For this study, demographic and geographic distributions for the non-institutionalized, civilian population that self-identify as Jewish by religion or background from KnowledgePanel are used as benchmarks in this adjustment.
Comparable distributions are calculated by using all completed cases from the field data (n = 1074). Since study sample sizes are typically too small to accommodate a complete cross-tabulation of all the survey variables with the benchmark variables, a raking procedure is used for the post-stratification weighting adjustment. Using the base weight as the starting weight, this procedure adjusts the sample data back to the selected benchmark proportions. Through an iterative convergence process, the weighted sample data are optimally fitted to the marginal distributions.
After this final post-stratification adjustment, the distribution of the calculated weights are examined to identify and, if necessary, trim outliers at the extreme upper and lower tails of the weight distribution. The post-stratified and trimmed weights are then scaled to the sum of the total sample size of all eligible respondents.
Following KN weighting, adjustments were made by the client that incorporated the weight and corrected for denomination. The benchmark used by the client to develop this weight was the 2008 Pew Religious Landscape Survey. KN then trimmed and scaled this new weight to the sum of the total sample size.