# Comparing survey results with different sample sizes

Percents are understood by nearly everyone, and therefore, they are the most popular statistics cited in research. Researchers are often interested in comparing two percentages to determine whether there is a significant difference between them.

There are two kinds of t-tests between percents. Which test you use depends upon whether you're comparing percentages from one or two samples. Every percentage can be expressed as a fraction. By looking at the denominator of the fraction we can determine whether to use a one-sample or two-sample t-test between percents.

If the denominators used to calculate the two percentages represent the same people, we use a one-sample t-test between percents to compare the two percents.

If the denominators represent different people, we use the two-sample t-test between percents.

Of the people, 80 said yes, said no, and 20 didn't know. You could summarize the responses as:. Obviously, there is a difference; but how sure are we that the difference didn't just happen by chance?

In other words, how reliable is the difference? Notice that the denominator used to calculate the percent of yes responses represents the same people as the denominator used to calculate the percent of no responses Therefore, we use a one-sample t-test between proportions.

The key is that the denominators represent the same people not that they are the same number. After you completed your survey, another group of researchers tried to replicate your study. They also used a sample size ofand asked the identical question. Of the people in their survey, 60 said yes, said no, and 40 didn't know. They summarized their results as:. To compare the yes responses between the two surveys, we would use a two-sample t-test between percents.

Even though both denominators werethey do not represent the same people. When there are more than two choices, you can do the t-test between any two of them. Thus, you could actually perform three separate t-tests If this was your analysis plan, you would also use Bonferroni's theorem to adjust the critical alpha level because the plan involved multiple tests of the same type and family.

Are the beliefs of your sample different than those of the previous study? Is there a significant difference between men and women? Is there a significant difference in product awareness between the Eastern and Western regions?

This test can be performed to determine whether respondents are more likely to prefer one alternative or another. The research question is: Is there a significant difference between the percent of people who say they would vote for candidate A and the percent of people who say they will vote for candidate B?

The null hypothesis is: There is no significant difference between the percent of people who say they will vote for candidate A or candidate B.Often, one of the ways you decide how to view and act on the results of one survey is by comparing past and present survey results.

This gives you a way to evaluate how customer responses have changed over time, which may give clues as to market changes or how customers have responded to different steps taken by your business.

There are several things you should consider when comparing past and present survey results. If your survey questions or survey methods have changed over time, it can cloud your results, making certain responses seem more or less significant than they might actually be. Statistical significance refers to the probability that the results are not due to chance, but rather some relevant variable. Every survey will sample a different set of people, who may differ in their responses for a wide variety of reasons.

Your software should provide a way to determine whether or not results are statistically significant. Before you compare results, make a list of any relevant events that happened near either set of results, such as:. What to look for when comparing past and present survey results.

Consistency in Survey Methods If your survey questions or survey methods have changed over time, it can cloud your results, making certain responses seem more or less significant than they might actually be.

Statistical Significance Statistical significance refers to the probability that the results are not due to chance, but rather some relevant variable. Isolated Incidents vs. Subscribe to mTab Insights. We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are okay with it.Studies involving fMRIs, which cost a lot to operate, have limited sample sizes as well [pdf] as do studies using laboratory animals.

To put it another way, statistical analysis with small samples is like making astronomical observations with binoculars. You are limited to seeing big things: planets, stars, moons and the occasional comet. Again, the key limitation is that you are limited to detecting large differences between designs or measures.

Fortunately, in user-experience research we are often most concerned about these big differencesâ€”differences users are likely to notice, such as changes in the navigation structure or the improvement of a search results page. If you need to compare completion rates, task times, and rating scale data for two independent groups, there are two procedures you can use for small and large sample sizes. The right one depends on the type of data you have: continuous or discrete-binary.

Comparing Means : If your data is generally continuous not binarysuch as task time or rating scales, use the two sample t-test. This is a variation on the better known Chi-Square test it is algebraically equivalent to the N-1 Chi-Square test.

When expected cell counts fall below one, the Fisher Exact Test tends to perform better. The online calculator handles this for you and we discuss the procedure in Chapter 5 of Quantifying the User Experience.

While the confidence interval width will be rather wide usually 20 to 30 percentage pointsthe upper or lower boundary of the intervals can be very helpful in establishing how often something will occur in the total user population. There are three approaches to computing confidence intervals based on whether your data is binary, task-time or continuous. Confidence interval around a mean : If your data is generally continuous not binary such as rating scales, order amounts in dollars, or the number of page views, the confidence interval is based on the t-distribution which takes into account sample size.

There is a lower boundary of 0 seconds. The online calculator handles all this. For the best overall average for small sample sizes, we have two recommendations for task-time and completion rates, and a more general recommendation for all sample sizes for rating scales.

Completion Rate : For small-sample completion rates, there are only a few possible values for each task.

## What to look for when comparing past and present survey results

It sounds too good to be true. We experimented [pdf] with several estimators with small sample sizes and found the LaPlace estimator and the simple proportion referred to as the Maximum Likelihood Estimator generally work well for the usability test data we examined.

When you want the best estimate, the calculator will generate it based on our findings. Rating Scales : Rating scales are a funny type of metric, in that most of them are bounded on both ends e. There are in fact many ways to report the scores from rating scales, including top-two boxes. Average Time : One long task time can skew the arithmetic mean and make it a poor measure of the middle.

Unfortunately, the median tends to be less accurate and more biased than the mean when sample sizes are less than about In these circumstances, the geometric mean average of the log values transformed back tends to be a better measure of the middle.

Sample Survey

When sample sizes get above 25, the median works fine. There are appropriate statistical methods to deal with small sample sizes. Comparing If you need to compare completion rates, task times, and rating scale data for two independent groups, there are two procedures you can use for small and large sample sizes.

Sign-up to receive weekly updates. Fall Delivered Online.Jill Boylston Herndon, Ph. A significant component of our work at the Institute for Child Health Policy is evaluating state Medicaid and CHIP programs, which frequently involves conducting surveys and analyzing survey data.

The focus of my talk today will be on some key considerations when working with multiple survey data sources. There are three main areas that we will address. The first involves considerations when comparing data on a similar domain from different surveys.

Finally, we will discuss some of the opportunities for linking state survey data to other data sources. The motivation for this webinar comes from the various ways that people use survey data for conducting research and policy analysis. So, a lot of times we may be interested in comparing data on a given health domain from different surveys. For example, if you're conducting a state or local survey, you may want to compare your results with those from national surveys, or you may be interested in using data from different surveys in order to provide contextual information.

There also is often interest in using national survey data to conduct state or local analyses or to inform state and local policymaking. You also may want to link survey data to other types of data, such as administrative data, in order to create richer analytic data sets. However, these various data sources may not be directly comparable or easily connected, which has important implications for both the ability to conduct the desired analyses and the interpretation of results.

So, the purposes of this webinar are to provide an overview of the key considerations in comparing and linking survey data and to offer strategies and resources for working with different data sources. For a variety of reasons there is often interest in using or comparing data from different surveys and there are certain health domains that are commonly included in national and state surveys.

For example, health insurance coverage is measured by several national surveys as well as many state surveys. There also are multiple national surveys that allow one to estimate the percentage of people who have received dental care. However, the estimates derived from the different surveys on each of these domains are different and sometimes the magnitude of the differences can be substantial.

The first consideration is the primary purpose of the survey. This may seem pretty fundamental but it's easy to get focused on the particular domains and data elements that you are interested in and lose sight of the larger context in which the data were collected. That larger context has significant implications for a number of factors that can influence how the domains of interest are measured.

These factors include the target population for the survey. For example, is it working aged adults or does it also include children and individuals 65 years and older? Attention may be more or less focused on the topics you're investigating. Moreover, it affects how in depth the domains of interest are covered. In addition, the primary purpose of the survey affects the context in which questions are asked and their placement in the survey.

Are the questions of interest placed early on, at the end of the survey, or somewhere in between?By using our site, you acknowledge that you have read and understand our Cookie PolicyPrivacy Policyand our Terms of Service. Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up. I have several populations of people, actually which vary in size from 5 to I would like to visualize the ratio of women vs.

I will get, for instance. Both percentages in the first cases are the same but a change of one person in each of the populations obviously changes percentages in a vastly different proportion. Should I take that into account when presenting the data?

I am working on a whole population, not samples, so I would tend to say no. I also have a gut feeling that the differences in the population size should still be accounted in some way. What I am trying to achieve at the end is the ability to state "all cases are similar" or "case 15 is significantly different" - again with the constraint of wildly varying population sizes.

You could present the actual population size using an axis label on any simple display e. A quite different plot would just be women versus men; the sex ratios would then be different slopes. Provided all values are positive, logarithmic scale might help. An audience naive or nervous about logarithmic scale might be encouraged by seeing raw and log scale side by side. The problem that you have presented is very valid and is similar to the difference between probabilities and odds ratio in a manner of speaking.

The percentage that you have calculated is similar to calculating probabilities in the sense that it is scale dependent. I would suggest that you calculate the Female to Male ratio the odds ratio which is scale independent and will give you an overall picture across varying populations.

Sign up to join this community. The best answers are voted up and rise to the top.After a very long period of pre-production, James Cameron and Robert Rodriguez are finally bringing the cult classic Battle Angel Alita to live-action. Watch the trailer now.

The Flash sports a new anime-inspired costume and design that captures the essence of the character while giving him a unique style. See photos of this new Justice League Variant Play Arts Kai Action Figure now.

Yami Yugi Figma Acts As Card Master Dec 08, 2017 Anniversaries Dominate Heritage Animation Dec 07, 2017 ComicLink Seeks More Consignments Dec 07, 2017 googletag. Share your thoughts here.

Jamie JacksonLive Sky Sports Premier LeagueThis season G13, Y49, R2, 3. Jamie Jackson Kick-off Sunday 4. It will be very helpful if you could share your feedback via this short survey that should take less than 2 minutes to fill out. Read the feature highlight summary below, and check out the Visual Studio 2017 version 15. The CPU Usage tool (available during F5 Debugging in the Diagnostic Tools window and in the ALT-F2 Performance Profiler) now displays source line highlighting based on the CPU consumption of specific lines of code.

This feature requires that source information be included in the generated PDB which is controlled by the project settings. Projects for which PDBs do not have source information will be unable to display either the line attribution or the source file. In addition to create, you can now delete tags, push tags, and create a new branch from tags.

Visual Studio Team Services users can now check out pull request branches, which makes it easier to review pull requests, test changes, and build your code. Last month we introduced the App Authentication Extension which makes it easy to configure your machine to use these protected settings so that you can develop and debug apps locally using your Visual Studio credentials.

With Visual Studio Version 15. Learn more about managing secrets in the cloud. With Visual Studio 2017 version 15. In addition, the ImageWatch extension has been updated to work with Visual Studio 2017IntelliSense for Python code now no longer requires a completion database. Instead of waiting up to four hours after installing a new package, you can start using it immediately. We have also added experimental support for managing Anaconda packages, new code snippets, and more customizable syntax highlighting.

Read our blog post for full details on these improvements and how to enable our experimental features. Real time test discovery is a new Visual Studio feature for managed projects that uses the Roslyn compiler to discover tests and populate the Test Explorer in real-time without requiring you to build your project. This feature was introduced behind a feature flag in version 15. This feature not only makes test discovery significantly faster, but it also keeps the Test Explorer in sync with code changes such as adding or removing tests.

To learn more, check out the Real Time Test Discovery blog post and Channel9 video. With this Preview, Visual Studio now supports configuring continuous delivery to Azure for Team Foundation Version Control (TFVC), Git SSH remotes, and Web Apps for containers. Read more about these features on this post about Continuous Delivery Tools for Visual Studio. The WCF Web Service Reference connected service provider now supports updating an existing service reference.

This simplifies the process for regenerating the WCF client proxy code for an updated web service.He believed that the world was growing nearer and nearer to the Apocalypse due to what he viewed as the rampant immorality of the times in Europe. After the prophecy failed, he changed the date three more times. The fallout of the group after the prediction failed was the basis for the 1956 book When Prophecy Fails.

The failure of the prophecy led to the split of the sect into several subsects, the most prominent led by Benjamin and Lois Roden.

Dixon predicted a planetary alignment on this day was to bring destruction to the world. Mass prayer meetings were held in India. The Brahma Kumaris founder, Lekhraj Kirpalani, has made a number of predictions of a global Armageddon which the religion believes it will inspire, internally calling it "Destruction". During Destruction, Brahma Kumari leaders teach the world will be purified, all of the rest of humanity killed by nuclear or civil wars and natural disasters which will include the sinking of all other continents except India.

Smith identified that he "could be wrong" but continued to say in the same sentence that his prediction was "a deep conviction in my heart, and all my plans are predicated upon that belief. After his September predictions failed to come true, Whisenant revised his prediction date to October 3.

Later, after Prophet's prediction did not come to pass, she was diagnosed with epilepsy and Alzheimer's disease. Berg predicted the tribulation would start in 1989 and that the Second Coming would take place in 1993. When it failed to occur he revised the date to September 29 and then to October 2. Applewhite, leader of the Heaven's Gate cult, claimed that a spacecraft was trailing the Comet Hale-Bopp and argued that suicide was "the only way to evacuate this Earth" so that the cult members' souls could board the supposed craft and be taken to another "level of existence above human".

Applewhite and 38 of his followers committed mass suicide. The 1st-century bishop of Edessa predicted this date to be the birth date of the Antichrist and the end of the universe. Moreover, God would have the same physical appearance as Chen himself.

Chen chose to base his cult in Garland, Texas, because he thought it sounded like "God's Land. He did not predict how it would occur, stating that it might involve nuclear devastation, asteroid impact, pole shift or other Earth changes. JenkinsThese Christian authors stated that the Y2K bug would trigger global economic chaos, which the Antichrist would use to rise to power. As the date approached, however, they changed their minds. The leader of the True and Living Church of Jesus Christ of Saints of the Last Days predicted the Second Coming of Christ would occur on this day.

According to her website, aliens in the Zeta Reticuli star system told her through messages via a brain implant of a planet which would enter our solar system and cause a pole shift on Earth that would destroy most of humanity.

This Japanese cult predicted the world would be destroyed by a nuclear war between October 30 and November 29, 2003. In his 1990 book The New Millennium, Robertson suggests this date as the day of Earth's destruction. He prophesied nuclear explosions in U. After his prophecy failed to come true he changed the date for the return of Jesus Christ to May 27, 2012. When his original prediction failed to come about, Camping revised his prediction and said that on May 21, a "Spiritual Judgment" took place, and that both the physical Rapture and the end of the world would occur on October 21, 2011.