tokenpocket官网下载app苹果手机|collecting data

作者： tokenpocket官网下载app苹果手机

2024-03-08 21:36:37

解决PyCharm调试查看变量时一直显示collecting data并报错Timeout waiting for response且看不到任何内容_pycharm collecting data-CSDN博客

解决PyCharm调试查看变量时一直显示collecting data并报错Timeout waiting for response且看不到任何内容

最新推荐文章于 2023-12-08 23:04:45 发布

越野者

最新推荐文章于 2023-12-08 23:04:45 发布

阅读量2.6w

点赞数

149

分类专栏：

Python

Debug

PyCharm

文章标签：

Debug

Python

PyCharm

本文链接：https://blog.csdn.net/discoverer100/article/details/99501027

版权

Python

同时被 3 个专栏收录

47 篇文章

1 订阅

订阅专栏

Debug

17 篇文章

0 订阅

订阅专栏

PyCharm

1 篇文章

0 订阅

订阅专栏

1. 问题描述

如题，在用PyCharm进行Python代码调试查看具体变量时，会随机遇到一直显示collecting data，到最后报错Timeout waiting for response，在界面中看不到变量内部的内容，如下图所示：

2. 解决办法

在PyCharm，打开Setting界面，在如下设置项中勾选“Gevent compatible”即可，如下图所示：至此，问题得到解决。

参考资料：

https://stackoverflow.com/questions/39371676/debugger-times-out-at-collecting-data

优惠劵

越野者

关注

149

踩

觉得还不错?

一键收藏

知道了

解决PyCharm调试查看变量时一直显示collecting data并报错Timeout waiting for response且看不到任何内容

1. 问题描述如题，在用PyCharm进行Python代码调试查看具体变量时，会随机遇到一直显示collecting data，到最后报错Timeout waiting for response，在界面中看不到变量内部的内容，如下图所示：2. 解决办法在PyCharm，打开Setting界面，在如下设置项中勾选“Gevent compatible”即可，如下图所示：至此，问题得到解决。...

复制链接

扫一扫

专栏目录

解决Pycharm运行时找不到文件的问题

09-19

今天小编就为大家分享一篇解决Pycharm运行时找不到文件的问题，具有很好的参考价值，希望对大家有所帮助。一起跟随小编过来看看吧

pycharm运行和调试不显示结果的解决方法

09-19

今天小编就为大家分享一篇pycharm运行和调试不显示结果的解决方法，具有很好的参考价值，希望对大家有所帮助。一起跟随小编过来看看吧

45 条评论

您还未登录，请先

后发表或查看评论

pycharm debug的时候变量显示不出来，一直Collecting data...问题解决

Data Collection | Definition, Methods & Examples

FAQ

About us

Our editors

Apply as editor

Team

Jobs

Contact

My account

Orders

Upload

Account details

Logout

My account

Overview

Availability

Information package

Account details

Logout

Admin

Proofreading & Editing

Thesis

Paper

AI Proofreader

Essay Checker

PhD dissertation

APA editing

Academic editing

College admissions essay

Personal statement

English proofreading

Spanish, French, or German

About our services

Proofreading services

Proofreading & editing example

Essay coaching example

Happiness guarantee

Plagiarism Checker

Citation Tools

Citation Generator

Check your Citations

Cite with Chrome

AI Writing

AI Proofreader

Paraphrasing Tool

Grammar Checker

Summarizer

AI Detector

Knowledge Base

Proofreading & Editing

Plagiarism Checker

Citation Tools

AI Writing

Knowledge Base

FAQ

About us

My account

Admin

Nederlands

English

Deutsch

Français

Italiano

Español

Svenska

Dansk

Suomi

Norwegian Bokmål

Back

Thesis

Paper

AI Proofreader

Essay Checker

PhD dissertation

APA editing

Academic editing

College admissions essay

Personal statement

English proofreading

Spanish, French, or German

About our services

Proofreading services

Proofreading & editing example

Essay coaching example

Happiness guarantee

Back

Citation Generator

Check your Citations

Cite with Chrome

Back

AI Proofreader

Paraphrasing Tool

Grammar Checker

Summarizer

AI Detector

Back

Our editors

Apply as editor

Team

Jobs

Contact

Back

Orders

Upload

Account details

Logout

Back

Overview

Availability

Information package

Account details

Logout

Have a language expert improve your writing

Proofreading Services

Run a free plagiarism check in 10 minutes

Plagiarism Checker

Generate accurate citations for free

Citation Generator

Home

Knowledge Base

Methodology

Data Collection | Definition, Methods & Examples

StatisticsStatistical analysis step by stepData collectionData collection guideExperimental designPopulations and samplesPopulations and samplesSampling methodsTypes of variablesTypes of variablesLevels of measurementNominal dataOrdinal dataInterval dataRatio dataData cleansingData cleansing guideMissing dataOutliersDescriptive statisticsOverview descriptive statisticsMeasures of central tendencyOverviewModeMedianMeanGeometric meanMeasures of variabilityOverviewRangeInterquartile rangeStandard deviationVarianceFrequency distributionFrequency distributionQuartiles & quantilesProbability (distributions)Probability distributionsOverviewNormal distributionStandard normal distributionPoisson distributionChi-square distributionChi-square tablet distributiont tableInferential statisticsOverview inferential statisticsDegrees of freedomCentral limit theoremParameters & test statisticsParameters vs. test statisticsTest statisticsEstimationStandard errorConfidence intervalsHypothesis testingHypothesis testing guideNull vs. alternative hypothesesStatistical significancep valueType I & Type II errorsStatistical powerStatistical testsChoosing the right testAssumptions for hypothesis testingSkewnessKurtosisCorrelationCorrelation vs. causationCorrelation coefficientPearson correlationRegression analysisSimple linear regressionMultiple linear regressionLinear regression in Rt testsANOVAsOne-way ANOVATwo-way ANOVAANOVA in RChi-squareOverview chi-square testsChi-square goodness of fit testChi-square test of independenceEffect sizeOverview of effect sizesCoefficient of determinationModel selectionAkaike information criterionReporting statistics in APA

Interesting topics

AMA style

Working with sources

IEEE

Commonly confused words

Commas

Definitions

UK vs. US English

Research bias

Nouns and pronouns

College essay

Parts of speech

Sentence structure

Verbs

Common mistakes

Effective communication

Using AI tools

Fallacies

Rhetoric

APA Style 6th edition

Applying to graduate school

Statistics

Chicago Style

Language rules

Methodology

MLA Style

Research paper

Academic writing

Starting the research process

Dissertation

Essay

Tips

APA Style 7th edition

APA citation examples

Citing sources

Plagiarism

Try our other services

Proofreading & Editing

Have a human editor polish your writing to ensure your arguments are judged on merit, not grammar errors.

Get expert writing help

AI Proofreader

Get unlimited proofreading for 30 days

Try for free

Plagiarism Checker

Compare your paper to billions of pages and articles with Scribbr’s Turnitin-powered plagiarism checker.

Run a free check

Citation Generator

Generate accurate APA, MLA, and Chicago citations for free with Scribbr's Citation Generator.

Start citing

Paraphraser

Rewrite and paraphrase texts instantly with our AI-powered paraphrasing tool.

Try for free

Grammar Checker

Eliminate grammar errors and improve your writing with our free AI-powered grammar checker.

Try for free

Data Collection | Definition, Methods & Examples

Published on

June 5, 2020

Pritha Bhandari.

Revised on

June 21, 2023.

Data collection is a systematic process of gathering observations or measurements. Whether you are performing research for business, governmental or academic purposes, data collection allows you to gain first-hand knowledge and original insights into your research problem.

While methods and aims may differ between fields, the overall process of data collection remains largely the same. Before you begin collecting data, you need to consider:

The aim of the research

The type of data that you will collect

The methods and procedures you will use to collect, store, and process the data

To collect high-quality data that is relevant to your purposes, follow these four steps.

Table of contentsStep 1: Define the aim of your researchStep 2: Choose your data collection methodStep 3: Plan your data collection proceduresStep 4: Collect the dataOther interesting articlesFrequently asked questions about data collection

Step 1: Define the aim of your research

Before you start the process of data collection, you need to identify exactly what you want to achieve. You can start by writing a problem statement: what is the practical or scientific issue that you want to address and why does it matter?

Next, formulate one or more research questions that precisely define what you want to find out. Depending on your research questions, you might need to collect quantitative or qualitative data:

Quantitative data is expressed in numbers and graphs and is analyzed through statistical methods.

Qualitative data is expressed in words and analyzed through interpretations and categorizations.

If your aim is to test a hypothesis, measure something precisely, or gain large-scale statistical insights, collect quantitative data. If your aim is to explore ideas, understand experiences, or gain detailed insights into a specific context, collect qualitative data. If you have several aims, you can use a mixed methods approach that collects both types of data.

Examples of quantitative and qualitative research aimsYou are researching employee perceptions of their direct managers in a large organization.

Your first aim is to assess whether there are significant differences in perceptions of managers across different departments and office locations.

Your second aim is to gather meaningful feedback from employees to explore new ideas for how managers can improve.

You decide to use a mixed-methods approach to collect both quantitative and qualitative data.

Here's why students love Scribbr's proofreading services

Trustpilot

Discover proofreading & editing

Step 2: Choose your data collection method

Based on the data you want to collect, decide which method is best suited for your research.

Experimental research is primarily a quantitative method.

Interviews, focus groups, and ethnographies are qualitative methods.

Surveys, observations, archival research and secondary data collection can be quantitative or qualitative methods.

Carefully consider what method you will use to gather data that helps you directly answer your research questions.

Data collection methods

Method

When to use

How to collect data

Experiment

To test a causal relationship.

Manipulate variables and measure their effects on others.

Survey

To understand the general characteristics or opinions of a group of people.

Distribute a list of questions to a sample online, in person or over-the-phone.

Interview/focus group

To gain an in-depth understanding of perceptions or opinions on a topic.

Verbally ask participants open-ended questions in individual interviews or focus group discussions.

Observation

To understand something in its natural setting.

Measure or survey a sample without trying to affect them.

Ethnography

To study the culture of a community or organization first-hand.

Join and participate in a community and record your observations and reflections.

Archival research

To understand current or historical events, conditions or practices.

Access manuscripts, documents or records from libraries, depositories or the internet.

Secondary data collection

To analyze data from populations that you can’t access first-hand.

Find existing datasets that have already been collected, from sources such as government agencies or research organizations.

Step 3: Plan your data collection procedures

When you know which method(s) you are using, you need to plan exactly how you will implement them. What procedures will you follow to make accurate observations or measurements of the variables you are interested in?

For instance, if you’re conducting surveys or interviews, decide what form the questions will take; if you’re conducting an experiment, make decisions about your experimental design (e.g., determine inclusion and exclusion criteria).

Operationalization

Sometimes your variables can be measured directly: for example, you can collect data on the average age of employees simply by asking for dates of birth. However, often you’ll be interested in collecting data on more abstract concepts or variables that can’t be directly observed.

Operationalization means turning abstract conceptual ideas into measurable observations. When planning how you will collect data, you need to translate the conceptual definition of what you want to study into the operational definition of what you will actually measure.

Example of operationalizationYou have decided to use surveys to collect quantitative data. The concept you want to measure is the leadership of managers. You operationalize this concept in two ways:

You ask managers to rate their own leadership skills on 5-point scales assessing the ability to delegate, decisiveness and dependability.

You ask their direct employees to provide anonymous feedback on the managers regarding the same topics.

Using multiple ratings of a single concept can help you cross-check your data and assess the test validity of your measures.

Sampling

You may need to develop a sampling plan to obtain data systematically. This involves defining a population, the group you want to draw conclusions about, and a sample, the group you will actually collect data from.

Your sampling method will determine how you recruit participants or obtain measurements for your study. To decide on a sampling method you will need to consider factors like the required sample size, accessibility of the sample, and timeframe of the data collection.

Standardizing procedures

If multiple researchers are involved, write a detailed manual to standardize data collection procedures in your study.

This means laying out specific step-by-step instructions so that everyone in your research team collects data in a consistent way – for example, by conducting experiments under the same conditions and using objective criteria to record and categorize observations. This helps you avoid common research biases like omitted variable bias or information bias.

This helps ensure the reliability of your data, and you can also use it to replicate the study in the future.

Creating a data management plan

Before beginning data collection, you should also decide how you will organize and store your data.

If you are collecting data from people, you will likely need to anonymize and safeguard the data to prevent leaks of sensitive information (e.g. names or identity numbers).

If you are collecting data via interviews or pencil-and-paper formats, you will need to perform transcriptions or data entry in systematic ways to minimize distortion.

You can prevent loss of data by having an organization system that is routinely backed up.

Step 4: Collect the data

Finally, you can implement your chosen methods to measure or observe the variables you are interested in.

Examples of collecting qualitative and quantitative dataTo collect data about perceptions of managers, you administer a survey with closed- and open-ended questions to a sample of 300 company employees across different departments and locations.

The closed-ended questions ask participants to rate their manager’s leadership skills on scales from 1–5. The data produced is numerical and can be statistically analyzed for averages and patterns.

The open-ended questions ask participants for examples of what the manager is doing well now and what they can do better in the future. The data produced is qualitative and can be categorized through content analysis for further insights.

To ensure that high quality data is recorded in a systematic way, here are some best practices:

Record all relevant information as and when you obtain data. For example, note down whether or how lab equipment is recalibrated during an experimental study.

Double-check manual data entry for errors.

If you collect quantitative data, you can assess the reliability and validity to get an indication of your data quality.

Here's why students love Scribbr's proofreading services

Trustpilot

Discover proofreading & editing

7 Data Collection Methods in Business Analytics

CoursesOpen Courses Mega Menu

Business Essentials

Credential of Readiness (CORe)

Business Analytics

Economics for Managers

Financial Accounting

Leadership & Management

Leadership Principles

Management Essentials

Negotiation Mastery

Organizational Leadership

Strategy Execution

Power and Influence for Positive Impact

Leadership, Ethics, and Corporate Accountability

Credential of Leadership, Impact, and Management in Business (CLIMB)

Entrepreneurship & Innovation

Entrepreneurship Essentials

Disruptive Strategy

Negotiation Mastery

Design Thinking and Innovation

Launching Tech Ventures

Winning with Digital Platforms

Strategy

Strategy Execution

Business Strategy

Economics for Managers

Disruptive Strategy

Global Business

Sustainable Business Strategy

Marketing

Digital Marketing Strategy

*New* Digital Transformation

Winning with Digital Platforms

Finance & Accounting

Financial Accounting

Leading with Finance

Alternative Investments

Sustainable Investing

Business in Society

Sustainable Business Strategy

Global Business

Sustainable Investing

Power and Influence for Positive Impact

Leadership, Ethics, and Corporate Accountability

Business and Climate Change

All Courses

For OrganizationsOpen For Organizations Mega Menu

Corporate LearningHelp your employees master essential business concepts, improve effectiveness, and

expand leadership capabilities.

Academic SolutionsIntegrate HBS Online courses into your curriculum to support programs and create unique

educational opportunities.

Need Help?

Frequently Asked Questions

Pathways to Business

Stories designed to inspire future business leaders.

InsightsOpen Insights Mega Menu

Business Insights Blog

Career Development

Communication

Decision-Making

Earning Your MBA

Entrepreneurship & Innovation

Finance

Leadership

Management

Negotiation

Strategy

All Topics

Sample Business Lessons and E-Books

Gain new insights and knowledge from leading faculty and industry experts.

Podcast

The Parlor Room: Where business concepts come to life. Listen now on your favorite podcast platform.

More InfoOpen More Info Mega Menu

Learning ExperienceMaster real-world business skills with our immersive platform and engaged community.

Certificates, Credentials, & CreditsLearn how completing courses can boost your resume and move your career forward.

Learning TracksTake your career to the next level with this specialization.

Financing & Policies

Employer Reimbursement

Payment & Financial Aid

Policies

Connect

Student Stories

Community

Need Help?

Frequently Asked Questions

Request Information

Support Portal

Apply Now

My CoursesAccess your courses and engage with your peers

My AccountManage your account, applications, and payments.

HBS Home

About HBS

Academic Programs

Alumni

Faculty & Research

Baker Library

Giving

Harvard Business Review

Initiatives

News

Recruit

Map / Directions

HBS Online

Courses

Business Essentials

Leadership & Management

Credential of Leadership, Impact, and Management in Business (CLIMB)

Entrepreneurship & Innovation

Strategy

Marketing

*New* Digital Transformation

Finance & Accounting

Business in Society

For Organizations

Insights

More Info

About

Support Portal

Media Coverage

Founding Donors

Leadership Team

Careers

My Courses

My Account

Apply Now

…→

Harvard Business School→

HBS Online→

Business Insights→

Business Insights

Harvard Business School Online's Business Insights Blog provides the career insights you need to achieve your goals and gain confidence in your business skills.

Filter Results

Arrow Down

Arrow Up

Topics

Accounting

Analytics

Business Essentials

Business in Society

Career Development

Communication

Community

ConneXt

Decision-Making

Earning Your MBA

Entrepreneurship & Innovation

Finance

Leadership

Management

Marketing

Negotiation

News & Events

Productivity

Staff Spotlight

Strategy

Student Profiles

Technology

Work-Life Balance

Courses

Alternative Investments

Business Analytics

Business Strategy

Business and Climate Change

CLIMB

CORe

Design Thinking and Innovation

Digital Marketing Strategy

Disruptive Strategy

Economics for Managers

Entrepreneurship Essentials

Financial Accounting

Global Business

Launching Tech Ventures

Leadership Principles

Leadership, Ethics, and Corporate Accountability

Leading with Finance

Management Essentials

Negotiation Mastery

Organizational Leadership

Power and Influence for Positive Impact

Strategy Execution

Sustainable Business Strategy

Sustainable Investing

Winning with Digital Platforms

Subscribe to the Blog

Email*

Please complete this required field.

Email must be formatted correctly.

Please complete all required fields.

RSS feed

Filters

Topics

Accounting

Analytics

Business Essentials

Business in Society

Career Development

Communication

Community

ConneXt

Decision-Making

Earning Your MBA

Entrepreneurship & Innovation

Finance

Leadership

Management

Marketing

Negotiation

News & Events

Productivity

Staff Spotlight

Strategy

Student Profiles

Technology

Work-Life Balance

Courses

Alternative Investments

Business Analytics

Business Strategy

Business and Climate Change

CLIMB

CORe

Design Thinking and Innovation

Digital Marketing Strategy

Disruptive Strategy

Economics for Managers

Entrepreneurship Essentials

Financial Accounting

Global Business

Launching Tech Ventures

Leadership Principles

Leadership, Ethics, and Corporate Accountability

Leading with Finance

Management Essentials

Negotiation Mastery

Organizational Leadership

Power and Influence for Positive Impact

Strategy Execution

Sustainable Business Strategy

Sustainable Investing

Winning with Digital Platforms

Subscribe to the Blog

Email*

Please complete this required field.

Email must be formatted correctly.

Please complete all required fields.

RSS feed

7 Data Collection Methods in Business Analytics

02 Dec 2021

Catherine Cote

Author

Staff

tag

Analytics

Business Analytics

Business Essentials

CORe

Data is being generated at an ever-increasing pace. According to Statista, the total volume of data was 64.2 zettabytes in 2020; it’s predicted to reach 181 zettabytes by 2025. This abundance of data can be overwhelming if you aren’t sure where to start.

So, how do you ensure the data you use is relevant and important to the business problems you aim to solve? After all, a data-driven decision is only as strong as the data it’s based on. One way is to collect data yourself.

Here’s a breakdown of data types, why data collection is important, what to know before you begin collecting, and seven data collection methods to leverage.

Free E-Book: A Beginner's Guide to Data & Analytics

Access your free e-book today.

DOWNLOAD NOW

What Is Data Collection?

Data collection is the methodological process of gathering information about a specific subject. It’s crucial to ensure your data is complete during the collection phase and that it’s collected legally and ethically. If not, your analysis won’t be accurate and could have far-reaching consequences.

In general, there are three types of consumer data:

First-party data, which is collected directly from users by your organization

Second-party data, which is data shared by another organization about its customers (or its first-party data)

Third-party data, which is data that’s been aggregated and rented or sold by organizations that don’t have a connection to your company or users

Although there are use cases for second- and third-party data, first-party data (data you’ve collected yourself) is more valuable because you receive information about how your audience behaves, thinks, and feels—all from a trusted source.

Data can be qualitative (meaning contextual in nature) or quantitative (meaning numeric in nature). Many data collection methods apply to either type, but some are better suited to one over the other.

In the data life cycle, data collection is the second step. After data is generated, it must be collected to be of use to your team. After that, it can be processed, stored, managed, analyzed, and visualized to aid in your organization’s decision-making.

Before collecting data, there are several factors you need to define:

The question you aim to answer

The data subject(s) you need to collect data from

The collection timeframe

The data collection method(s) best suited to your needs

The data collection method you select should be based on the question you want to answer, the type of data you need, your timeframe, and your company’s budget.

The Importance of Data Collection

Collecting data is an integral part of a business’s success; it can enable you to ensure the data’s accuracy, completeness, and relevance to your organization and the issue at hand. The information gathered allows organizations to analyze past strategies and stay informed on what needs to change.

The insights gleaned from data can make you hyperaware of your organization’s efforts and give you actionable steps to improve various strategies—from altering marketing strategies to assessing customer complaints.

Basing decisions on inaccurate data can have far-reaching negative consequences, so it’s important to be able to trust your own data collection procedures and abilities. By ensuring accurate data collection, business professionals can feel secure in their business decisions.

Explore the options in the next section to see which data collection method is the best fit for your company.

7 Data Collection Methods Used in Business Analytics

1. Surveys

Surveys are physical or digital questionnaires that gather both qualitative and quantitative data from subjects. One situation in which you might conduct a survey is gathering attendee feedback after an event. This can provide a sense of what attendees enjoyed, what they wish was different, and areas in which you can improve or save money during your next event for a similar audience.

While physical copies of surveys can be sent out to participants, online surveys present the opportunity for distribution at scale. They can also be inexpensive; running a survey can cost nothing if you use a free tool. If you wish to target a specific group of people, partnering with a market research firm to get the survey in front of that demographic may be worth the money.

Something to watch out for when crafting and running surveys is the effect of bias, including:

Collection bias: It can be easy to accidentally write survey questions with a biased lean. Watch out for this when creating questions to ensure your subjects answer honestly and aren’t swayed by your wording.

Subject bias: Because your subjects know their responses will be read by you, their answers may be biased toward what seems socially acceptable. For this reason, consider pairing survey data with behavioral data from other collection methods to get the full picture.

Related: 3 Examples of Bad Survey Questions & How to Fix Them

2. Transactional Tracking

Each time your customers make a purchase, tracking that data can allow you to make decisions about targeted marketing efforts and understand your customer base better.

Often, e-commerce and point-of-sale platforms allow you to store data as soon as it’s generated, making this a seamless data collection method that can pay off in the form of customer insights.

3. Interviews and Focus Groups

Interviews and focus groups consist of talking to subjects face-to-face about a specific topic or issue. Interviews tend to be one-on-one, and focus groups are typically made up of several people. You can use both to gather qualitative and quantitative data.

Through interviews and focus groups, you can gather feedback from people in your target audience about new product features. Seeing them interact with your product in real-time and recording their reactions and responses to questions can provide valuable data about which product features to pursue.

As is the case with surveys, these collection methods allow you to ask subjects anything you want about their opinions, motivations, and feelings regarding your product or brand. It also introduces the potential for bias. Aim to craft questions that don’t lead them in one particular direction.

One downside of interviewing and conducting focus groups is they can be time-consuming and expensive. If you plan to conduct them yourself, it can be a lengthy process. To avoid this, you can hire a market research facilitator to organize and conduct interviews on your behalf.

4. Observation

Observing people interacting with your website or product can be useful for data collection because of the candor it offers. If your user experience is confusing or difficult, you can witness it in real-time.

Yet, setting up observation sessions can be difficult. You can use a third-party tool to record users’ journeys through your site or observe a user’s interaction with a beta version of your site or product.

While less accessible than other data collection methods, observations enable you to see firsthand how users interact with your product or site. You can leverage the qualitative and quantitative data gleaned from this to make improvements and double down on points of success.

5. Online Tracking

To gather behavioral data, you can implement pixels and cookies. These are both tools that track users’ online behavior across websites and provide insight into what content they’re interested in and typically engage with.

You can also track users’ behavior on your company’s website, including which parts are of the highest interest, whether users are confused when using it, and how long they spend on product pages. This can enable you to improve the website’s design and help users navigate to their destination.

Inserting a pixel is often free and relatively easy to set up. Implementing cookies may come with a fee but could be worth it for the quality of data you’ll receive. Once pixels and cookies are set, they gather data on their own and don’t need much maintenance, if any.

It’s important to note: Tracking online behavior can have legal and ethical privacy implications. Before tracking users’ online behavior, ensure you’re in compliance with local and industry data privacy standards.

6. Forms

Online forms are beneficial for gathering qualitative data about users, specifically demographic data or contact information. They’re relatively inexpensive and simple to set up, and you can use them to gate content or registrations, such as webinars and email newsletters.

You can then use this data to contact people who may be interested in your product, build out demographic profiles of existing customers, and in remarketing efforts, such as email workflows and content recommendations.

Related: What Is Marketing Analytics?

7. Social Media Monitoring

Monitoring your company’s social media channels for follower engagement is an accessible way to track data about your audience’s interests and motivations. Many social media platforms have analytics built in, but there are also third-party social platforms that give more detailed, organized insights pulled from multiple channels.

You can use data collected from social media to determine which issues are most important to your followers. For instance, you may notice that the number of engagements dramatically increases when your company posts about its sustainability efforts.

Building Your Data Capabilities

Understanding the variety of data collection methods available can help you decide which is best for your timeline, budget, and the question you’re aiming to answer. When stored together and combined, multiple data types collected through different methods can give an informed picture of your subjects and help you make better business decisions.

Do you want to become a data-driven professional? Explore our eight-week Business Analytics course and our three-course Credential of Readiness (CORe) program to deepen your analytical skills and apply them to real-world business problems. Not sure which course is right for you? Download our free flowchart.

This post was updated on October 17, 2022. It was originally published on December 2, 2021.

About the AuthorCatherine Cote is a marketing coordinator at Harvard Business School Online. Prior to joining HBS Online, she worked at an early-stage SaaS startup where she found her passion for writing content, and at a digital consulting agency, where she specialized in SEO. Catherine holds a B.A. from Holy Cross, where she studied psychology, education, and Mandarin Chinese. When not at work, you can find her hiking, performing or watching theatre, or hunting for the best burger in Boston.

All FAQsTop FAQs

How are HBS Online courses delivered?

+–

We offer self-paced programs (with weekly deadlines) on the HBS Online course platform.

Our platform features short, highly produced videos of HBS faculty and guest business experts, interactive graphs and exercises, cold calls to keep you engaged, and opportunities to contribute to a vibrant online community.

Are HBS Online programs available in languages other than English?

+–

We expect to offer our courses in additional languages in the future but, at this time, HBS Online can only be provided in English.

All course content is delivered in written English. Closed captioning in English is available for all videos. There are no live interactions during the course that requires the learner to speak English. Coursework must be completed in English.

Do I need to come to campus to participate in HBS Online programs?

+–

No, all of our programs are 100 percent online, and available to participants regardless of their location.

How do I enroll in a course?

+–

All programs require the completion of a brief application. The applications vary slightly from program to program, but all ask for some personal background information. You can apply for and enroll in programs here. If you are new to HBS Online, you will be required to set up an account before starting an application for the program of your choice.

Our easy online application is free, and no special documentation is required. All applicants must be at least 18 years of age, proficient in English, and committed to learning and engaging with fellow participants throughout the program.

Updates to your application and enrollment status will be shown on your Dashboard. We confirm enrollment eligibility within one week of your application. HBS Online does not use race, gender, ethnicity, or any protected class as criterion for admissions for any HBS Online program.

Does Harvard Business School Online offer an online MBA?

+–

No, Harvard Business School Online offers business certificate programs.

What are my payment options?

+–

We accept payments via credit card, wire transfer, Western Union, and (when available) bank loan. Some candidates may qualify for scholarships or financial aid, which will be credited against the Program Fee once eligibility is determined. Please refer to the Payment & Financial Aid page for further information.

We also allow you to split your payment across 2 separate credit card transactions or send a payment link email to another person on your behalf. If splitting your payment into 2 transactions, a minimum payment of $350 is required for the first transaction.

In all cases, net Program Fees must be paid in full (in US Dollars) to complete registration.

What are the policies for refunds and deferrals?

+–

After enrolling in a program, you may request a withdrawal with refund (minus a $100 nonrefundable enrollment fee) up until 24 hours after the start of your program. Please review the Program Policies page for more details on refunds and deferrals. If your employer has contracted with HBS Online for participation in a program, or if you elect to enroll in the undergraduate credit option of the Credential of Readiness (CORe) program, note that policies for these options may differ.

Email*

• Please complete this required field.

• Email must be formatted correctly.

• Please complete all required fields.

Subject Areas

Business Essentials

Leadership & Management

Entrepreneurship & Innovation

Strategy

Marketing

Finance & Accounting

Business & Society

Quick Links

FAQs

Request Info

Apply Now

Support Portal

About

About Us

Media Coverage

Founding Donors

Leadership Team

Careers @ HBS Online

Legal

Policies

Site Map

Trademark Notice

Digital Accessibility

Chapter 5: Collecting data | Cochrane Training

Jump to navigation

Top menuContact

Cochrane.org

Cochrane Community

Cochrane

Training

Trusted evidence.

Informed decisions.

Better health.

You are hereHome › Cochrane Handbook for Systematic Reviews of Interventions › Version 6.4 › Part 2: Core methods › Chapter 5: Collecting data

Chapter 5: Collecting data

Tianjing Li, Julian PT Higgins, Jonathan J Deeks

Key Points:

Systematic reviews have studies, rather than reports, as the unit of interest, and so multiple reports of the same study need to be identified and linked together before or after data extraction.

Because of the increasing availability of data sources (e.g. trials registers, regulatory documents, clinical study reports), review authors should decide on which sources may contain the most useful information for the review, and have a plan to resolve discrepancies if information is inconsistent across sources.

Review authors are encouraged to develop outlines of tables and figures that will appear in the review to facilitate the design of data collection forms. The key to successful data collection is to construct easy-to-use forms and collect sufficient and unambiguous data that faithfully represent the source in a structured and organized manner.

Effort should be made to identify data needed for meta-analyses, which often need to be calculated or converted from data reported in diverse formats.

Data should be collected and archived in a form that allows future access and data sharing.

Cite this chapter as: Li T, Higgins JPT, Deeks JJ (editors). Chapter 5: Collecting data. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023. Available from www.training.cochrane.org/handbook.

5.1 Introduction

Systematic reviews aim to identify all studies that are relevant to their research questions and to synthesize data about the design, risk of bias, and results of those studies. Consequently, the findings of a systematic review depend critically on decisions relating to which data from these studies are presented and analysed. Data collected for systematic reviews should be accurate, complete, and accessible for future updates of the review and for data sharing. Methods used for these decisions must be transparent; they should be chosen to minimize biases and human error. Here we describe approaches that should be used in systematic reviews for collecting data, including extraction of data directly from journal articles and other reports of studies.

5.2 Sources of data

Studies are reported in a range of sources which are detailed later. As discussed in Section 5.2.1, it is important to link together multiple reports of the same study. The relative strengths and weaknesses of each type of source are discussed in Section 5.2.2. For guidance on searching for and selecting reports of studies, refer to Chapter 4.

Journal articles are the source of the majority of data included in systematic reviews. Note that a study can be reported in multiple journal articles, each focusing on some aspect of the study (e.g. design, main results, and other results).

Conference abstracts are commonly available. However, the information presented in conference abstracts is highly variable in reliability, accuracy, and level of detail (Li et al 2017).

Errata and letters can be important sources of information about studies, including critical weaknesses and retractions, and review authors should examine these if they are identified (see MECIR Box 5.2.a).

Trials registers (e.g. ClinicalTrials.gov) catalogue trials that have been planned or started, and have become an important data source for identifying trials, for comparing published outcomes and results with those planned, and for obtaining efficacy and safety data that are not available elsewhere (Ross et al 2009, Jones et al 2015, Baudard et al 2017).

Clinical study reports (CSRs) contain unabridged and comprehensive descriptions of the clinical problem, design, conduct and results of clinical trials, following a structure and content guidance prescribed by the International Conference on Harmonisation (ICH 1995). To obtain marketing approval of drugs and biologics for a specific indication, pharmaceutical companies submit CSRs and other required materials to regulatory authorities. Because CSRs also incorporate tables and figures, with appendices containing the protocol, statistical analysis plan, sample case report forms, and patient data listings (including narratives of all serious adverse events), they can be thousands of pages in length. CSRs often contain more data about trial methods and results than any other single data source (Mayo-Wilson et al 2018). CSRs are often difficult to access, and are usually not publicly available. Review authors could request CSRs from the European Medicines Agency (Davis and Miller 2017). The US Food and Drug and Administration had historically avoided releasing CSRs but launched a pilot programme in 2018 whereby selected portions of CSRs for new drug applications were posted on the agency’s website. Many CSRs are obtained through unsealed litigation documents, repositories (e.g. clinicalstudydatarequest.com), and other open data and data-sharing channels (e.g. The Yale University Open Data Access Project) (Doshi et al 2013, Wieland et al 2014, Mayo-Wilson et al 2018)).

Regulatory reviews such as those available from the US Food and Drug Administration or European Medicines Agency provide useful information about trials of drugs, biologics, and medical devices submitted by manufacturers for marketing approval (Turner 2013). These documents are summaries of CSRs and related documents, prepared by agency staff as part of the process of approving the products for marketing, after reanalysing the original trial data. Regulatory reviews often are available only for the first approved use of an intervention and not for later applications (although review authors may request those documents, which are usually brief). Using regulatory reviews from the US Food and Drug Administration as an example, drug approval packages are available on the agency’s website for drugs approved since 1997 (Turner 2013); for drugs approved before 1997, information must be requested through a freedom of information request. The drug approval packages contain various documents: approval letter(s), medical review(s), chemistry review(s), clinical pharmacology review(s), and statistical reviews(s).

Individual participant data (IPD) are usually sought directly from the researchers responsible for the study, or may be identified from open data repositories (e.g. www.clinicalstudydatarequest.com). These data typically include variables that represent the characteristics of each participant, intervention (or exposure) group, prognostic factors, and measurements of outcomes (Stewart et al 2015). Access to IPD has the advantage of allowing review authors to reanalyse the data flexibly, in accordance with the preferred analysis methods outlined in the protocol, and can reduce the variation in analysis methods across studies included in the review. IPD reviews are addressed in detail in Chapter 26.

MECIR Box 5.2.a Relevant expectations for conduct of intervention reviews

C48: Examining errata (Mandatory)

Examine any relevant retraction statements and errata for information.

Some studies may have been found to be fraudulent or may for other reasons have been retracted since publication. Errata can reveal important limitations, or even fatal flaws, in included studies. All of these may potentially lead to the exclusion of a study from a review or meta-analysis. Care should be taken to ensure that this information is retrieved in all database searches by downloading the appropriate fields together with the citation data.

5.2.1 Studies (not reports) as the unit of interest

In a systematic review, studies rather than reports of studies are the principal unit of interest. Since a study may have been reported in several sources, a comprehensive search for studies for the review may identify many reports from a potentially relevant study (Mayo-Wilson et al 2017a, Mayo-Wilson et al 2018). Conversely, a report may describe more than one study.

Multiple reports of the same study should be linked together (see MECIR Box 5.2.b). Some authors prefer to link reports before they collect data, and collect data from across the reports onto a single form. Other authors prefer to collect data from each report and then link together the collected data across reports. Either strategy may be appropriate, depending on the nature of the reports at hand. It may not be clear that two reports relate to the same study until data collection has commenced. Although sometimes there is a single report for each study, it should never be assumed that this is the case.

MECIR Box 5.2.b Relevant expectations for conduct of intervention reviews

C42: Collating multiple reports (Mandatory)

Collate multiple reports of the same study, so that each study rather than each report is the unit of interest in the review.

It is wrong to consider multiple reports of the same study as if they are multiple studies. Secondary reports of a study should not be discarded, however, since they may contain valuable information about the design and conduct. Review authors must choose and justify which report to use as a source for study results.

It can be difficult to link multiple reports from the same study, and review authors may need to do some ‘detective work’. Multiple sources about the same trial may not reference each other, do not share common authors (Gøtzsche 1989, Tramèr et al 1997), or report discrepant information about the study design, characteristics, outcomes, and results (von Elm et al 2004, Mayo-Wilson et al 2017a).

Some of the most useful criteria for linking reports are:

trial registration numbers;

authors’ names;

sponsor for the study and sponsor identifiers (e.g. grant or contract numbers);

location and setting (particularly if institutions, such as hospitals, are named);

specific details of the interventions (e.g. dose, frequency);

numbers of participants and baseline data; and

date and duration of the study (which also can clarify whether different sample sizes are due to different periods of recruitment), length of follow-up, or subgroups selected to address secondary goals.

Review authors should use as many trial characteristics as possible to link multiple reports. When uncertainties remain after considering these and other factors, it may be necessary to correspond with the study authors or sponsors for confirmation.

5.2.2 Determining which sources might be most useful

A comprehensive search to identify all eligible studies from all possible sources is resource-intensive but necessary for a high-quality systematic review (see Chapter 4). Because some data sources are more useful than others (Mayo-Wilson et al 2018), review authors should consider which data sources may be available and which may contain the most useful information for the review. These considerations should be described in the protocol. Table 5.2.a summarizes the strengths and limitations of different data sources (Mayo-Wilson et al 2018). Gaining access to CSRs and IPD often takes a long time. Review authors should begin searching repositories and contact trial investigators and sponsors as early as possible to negotiate data usage agreements (Mayo-Wilson et al 2015, Mayo-Wilson et al 2018).

Table 5.2.a Strengths and limitations of different data sources for systematic reviews

Source

Strengths

Limitations

Public sources

Journal articles

Found easily

Data extracted quickly

Include useful information about methods and results

Available for some, but not all studies (with a risk of reporting biases: see Chapter 7 and Chapter 13)

Contain limited study characteristics and methods

Can omit outcomes, especially harms

Conference abstracts

Identify unpublished studies

Include little information about study design

Include limited and unclear information for meta-analysis

May result in double-counting studies in meta-analysis if not correctly linked to other reports of the same study

Trial registrations

Identify otherwise unpublished trials

May contain information about design, risk of bias, and results not included in other public sources

Link multiple sources about the same trial using unique registration number

Limited to more recent studies that comply with registration requirements

Often contain limited information about trial design and quantitative results

May report only harms (adverse events) occurring above a threshold (e.g. 5%)

May be inaccurate or incomplete for trials whose methods have changed during the conduct of the study, or results not kept up to date

Regulatory information

Identify studies not reported in other public sources

Describe details of methods and results not found in other sources

Available only for studies submitted to regulators

Available for approved indications, but not ‘off-label’ uses

Not always in a standard format

Not often available for old products

Non-public sources

Clinical study reports (CSRs)

Contain detailed information about study characteristics, methods, and results

Can be particularly useful for identifying detailed information about harms

Describe aggregate results, which are easy to analyse and sufficient for most reviews

Do not exist or difficult to obtain for most studies

Require more time to obtain and analyse than public sources

Individual participant data

Allow review authors to use contemporary statistical methods and to standardize analyses across studies

Permit additional analyses that the review authors desire (e.g. subgroup analyses)

Require considerable expertise and time to obtain and analyse

May lead to the same results that can be found in aggregate report

May not be necessary if one has a CSR

5.2.3 Correspondence with investigators

Review authors often find that they are unable to obtain all the information they seek from available reports about the details of the study design, the full range of outcomes measured and the numerical results. In such circumstances, authors are strongly encouraged to contact the original investigators (see MECIR Box 5.2.c). Contact details of study authors, when not available from the study reports, often can be obtained from more recent publications, from university or institutional staff listings, from membership directories of professional societies, or by a general search of the web. If the contact author named in the study report cannot be contacted or does not respond, it is worthwhile attempting to contact other authors.

Review authors should consider the nature of the information they require and make their request accordingly. For descriptive information about the conduct of the trial, it may be most appropriate to ask open-ended questions (e.g. how was the allocation process conducted, or how were missing data handled?). If specific numerical data are required, it may be more helpful to request them specifically, possibly providing a short data collection form (either uncompleted or partially completed). If IPD are required, they should be specifically requested (see also Chapter 26). In some cases, study investigators may find it more convenient to provide IPD rather than conduct additional analyses to obtain the specific statistics requested.

MECIR Box 5.2.c Relevant expectations for conduct of intervention reviews

C49: Obtaining unpublished data (Highly desirable)

Seek key unpublished information that is missing from reports of included studies.

Contacting study authors to obtain or confirm data makes the review more complete, potentially enhances precision and reduces the impact of reporting biases. Missing information includes details to inform risk of bias assessments, details of interventions and outcomes, and study results (including breakdowns of results by important subgroups).

5.3 What data to collect

5.3.1 What are data?

For the purposes of this chapter, we define ‘data’ to be any information about (or derived from) a study, including details of methods, participants, setting, context, interventions, outcomes, results, publications, and investigators. Review authors should plan in advance what data will be required for their systematic review, and develop a strategy for obtaining them (see MECIR Box 5.3.a). The involvement of consumers and other stakeholders can be helpful in ensuring that the categories of data collected are sufficiently aligned with the needs of review users (Chapter 1, Section 1.3). The data to be sought should be described in the protocol, with consideration wherever possible of the issues raised in the rest of this chapter.

The data collected for a review should adequately describe the included studies, support the construction of tables and figures, facilitate the risk of bias assessment, and enable syntheses and meta-analyses. Review authors should familiarize themselves with reporting guidelines for systematic reviews (see online Chapter III and the PRISMA statement; (Liberati et al 2009) to ensure that relevant elements and sections are incorporated. The following sections review the types of information that should be sought, and these are summarized in Table 5.3.a (Li et al 2015).

MECIR Box 5.3.a Relevant expectations for conduct of intervention reviews

C44: Describing studies (Mandatory)

Collect characteristics of the included studies in sufficient detail to populate a table of ‘Characteristics of included studies’.

Basic characteristics of each study will need to be presented as part of the review, including details of participants, interventions and comparators, outcomes and study design.

Table 5.3.a Checklist of items to consider in data collection

Information about data extraction from reports

Name of data extractors, date of data extraction, and identification features of each report from which data are being extracted

Eligibility criteria

Confirm eligibility of the study for the review

Reason for exclusion

Study methods

Study design:

Parallel, factorial, crossover, cluster aspects of design for randomized trials, and/or study design features for non-randomized studies

Single or multicentre study; if multicentre, number of recruiting centres

Recruitment and sampling procedures used (including at the level of individual participants and clusters/sites if relevant)

Enrolment start and end dates; length of participant follow-up

Details of random sequence generation, allocation sequence concealment, and masking for randomized trials, and methods used to prevent and control for confounding, selection biases, and information biases for non-randomized studies*

Methods used to prevent and address missing data*

Statistical analysis:

Unit of analysis (e.g. individual participant, clinic, village, body part)

Statistical methods used if computed effect estimates are extracted from reports, including any covariates included in the statistical model

Likelihood of reporting and other biases*

Source(s) of funding or other material support for the study

Authors’ financial relationship and other potential conflicts of interest

Participants

Setting

Region(s) and country/countries from which study participants were recruited

Study eligibility criteria, including diagnostic criteria

Characteristics of participants at the beginning (or baseline) of the study (e.g. age, sex, comorbidity, socio-economic status)

Intervention

Description of the intervention(s) and comparison intervention(s), ideally with sufficient detail for replication:

Components, routes of delivery, doses, timing, frequency, intervention protocols, length of intervention

Factors relevant to implementation (e.g. staff qualifications, equipment requirements)

Integrity of interventions (i.e. the degree to which specified procedures or components of the intervention were implemented as planned)

Description of co-interventions

Definition of ‘control’ groups (e.g. no intervention, placebo, minimally active comparator, or components of usual care)

Components, dose, timing, frequency

For observational studies: description of how intervention status was assessed; length of exposure, cumulative exposure

Outcomes

For each pre-specified outcome domain (e.g. anxiety) in the systematic review:

Whether there is evidence that the outcome domain was assessed (especially important if the outcome was assessed but the results not presented; see Chapter 13)

Measurement tool or instrument (including definition of clinical outcomes or endpoints); for a scale, name of the scale (e.g. the Hamilton Anxiety Rating Scale), upper and lower limits, and whether a high or low score is favourable, definitions of any thresholds if appropriate

Specific metric (e.g. post-intervention anxiety, or change in anxiety from baseline to a post-intervention time point, or post-intervention presence of anxiety (yes/no))

Method of aggregation (e.g. mean and standard deviation of anxiety scores in each group, or proportion of people with anxiety)

Timing of outcome measurements (e.g. assessments at end of eight-week intervention period, events occurring during the eight-week intervention period)

Adverse outcomes need special attention depending on whether they are collected systematically or non-systematically (e.g. by voluntary report)

Results

For each group, and for each outcome at each time point: number of participants randomly assigned and included in the analysis; and number of participants who withdrew, were lost to follow-up or were excluded (with reasons for each)

Summary data for each group (e.g. 2×2 table for dichotomous data; means and standard deviations for continuous data)

Between-group estimates that quantify the effect of the intervention on the outcome, and their precision (e.g. risk ratio, odds ratio, mean difference)

If subgroup analysis is planned, the same information would need to be extracted for each participant subgroup

Miscellaneous

Key conclusions of the study authors

Reference to other relevant studies

Correspondence required

Miscellaneous comments from the study authors or by the review authors

*Full description required for assessments of risk of bias (see Chapter 8, Chapter 23 and Chapter 25).

5.3.2 Study methods and potential sources of bias

Different research methods can influence study outcomes by introducing different biases into results. Important study design characteristics should be collected to allow the selection of appropriate methods for assessment and analysis, and to enable description of the design of each included study in a table of ‘Characteristics of included studies’, including whether the study is randomized, whether the study has a cluster or crossover design, and the duration of the study. If the review includes non-randomized studies, appropriate features of the studies should be described (see Chapter 24).

Detailed information should be collected to facilitate assessment of the risk of bias in each included study. Risk-of-bias assessment should be conducted using the tool most appropriate for the design of each study, and the information required to complete the assessment will depend on the tool. Randomized studies should be assessed using the tool described in Chapter 8. The tool covers bias arising from the randomization process, due to deviations from intended interventions, due to missing outcome data, in measurement of the outcome, and in selection of the reported result. For each item in the tool, a description of what happened in the study is required, which may include verbatim quotes from study reports. Information for assessment of bias due to missing outcome data and selection of the reported result may be most conveniently collected alongside information on outcomes and results. Chapter 7 (Section 7.3.1) discusses some issues in the collection of information for assessments of risk of bias. For non-randomized studies, the most appropriate tool is described in Chapter 25. A separate tool also covers bias due to missing results in meta-analysis (see Chapter 13).

A particularly important piece of information is the funding source of the study and potential conflicts of interest of the study authors.

Some review authors will wish to collect additional information on study characteristics that bear on the quality of the study’s conduct but that may not lead directly to risk of bias, such as whether ethical approval was obtained and whether a sample size calculation was performed a priori.

5.3.3 Participants and setting

Details of participants are collected to enable an understanding of the comparability of, and differences between, the participants within and between included studies, and to allow assessment of how directly or completely the participants in the included studies reflect the original review question.

Typically, aspects that should be collected are those that could (or are believed to) affect presence or magnitude of an intervention effect and those that could help review users assess applicability to populations beyond the review. For example, if the review authors suspect important differences in intervention effect between different socio-economic groups, this information should be collected. If intervention effects are thought constant over such groups, and if such information would not be useful to help apply results, it should not be collected. Participant characteristics that are often useful for assessing applicability include age and sex. Summary information about these should always be collected unless they are not obvious from the context. These characteristics are likely to be presented in different formats (e.g. ages as means or medians, with standard deviations or ranges; sex as percentages or counts for the whole study or for each intervention group separately). Review authors should seek consistent quantities where possible, and decide whether it is more relevant to summarize characteristics for the study as a whole or by intervention group. It may not be possible to select the most consistent statistics until data collection is complete across all or most included studies. Other characteristics that are sometimes important include ethnicity, socio-demographic details (e.g. education level) and the presence of comorbid conditions. Clinical characteristics relevant to the review question (e.g. glucose level for reviews on diabetes) also are important for understanding the severity or stage of the disease.

Diagnostic criteria that were used to define the condition of interest can be a particularly important source of diversity across studies and should be collected. For example, in a review of drug therapy for congestive heart failure, it is important to know how the definition and severity of heart failure was determined in each study (e.g. systolic or diastolic dysfunction, severe systolic dysfunction with ejection fractions below 20%). Similarly, in a review of antihypertensive therapy, it is important to describe baseline levels of blood pressure of participants.

If the settings of studies may influence intervention effects or applicability, then information on these should be collected. Typical settings of healthcare intervention studies include acute care hospitals, emergency facilities, general practice, and extended care facilities such as nursing homes, offices, schools, and communities. Sometimes studies are conducted in different geographical regions with important differences that could affect delivery of an intervention and its outcomes, such as cultural characteristics, economic context, or rural versus city settings. Timing of the study may be associated with important technology differences or trends over time. If such information is important for the interpretation of the review, it should be collected.

Important characteristics of the participants in each included study should be summarized for the reader in the table of ‘Characteristics of included studies’.

5.3.4 Interventions

Details of all experimental and comparator interventions of relevance to the review should be collected. Again, details are required for aspects that could affect the presence or magnitude of an effect or that could help review users assess applicability to their own circumstances. Where feasible, information should be sought (and presented in the review) that is sufficient for replication of the interventions under study. This includes any co-interventions administered as part of the study, and applies similarly to comparators such as ‘usual care’. Review authors may need to request missing information from study authors.

The Template for Intervention Description and Replication (TIDieR) provides a comprehensive framework for full description of interventions and has been proposed for use in systematic reviews as well as reports of primary studies (Hoffmann et al 2014). The checklist includes descriptions of:

the rationale for the intervention and how it is expected to work;

any documentation that instructs the recipient on the intervention;

what the providers do to deliver the intervention (procedures and processes);

who provides the intervention (including their skill level), how (e.g. face to face, web-based) and in what setting (e.g. home, school, or hospital);

the timing and intensity;

whether any variation is permitted or expected, and whether modifications were actually made; and

any strategies used to ensure or assess fidelity or adherence to the intervention, and the extent to which the intervention was delivered as planned.

For clinical trials of pharmacological interventions, key information to collect will often include routes of delivery (e.g. oral or intravenous delivery), doses (e.g. amount or intensity of each treatment, frequency of delivery), timing (e.g. within 24 hours of diagnosis), and length of treatment. For other interventions, such as those that evaluate psychotherapy, behavioural and educational approaches, or healthcare delivery strategies, the amount of information required to characterize the intervention will typically be greater, including information about multiple elements of the intervention, who delivered it, and the format and timing of delivery. Chapter 17 provides further information on how to manage intervention complexity, and how the intervention Complexity Assessment Tool (iCAT) can facilitate data collection (Lewin et al 2017).

Important characteristics of the interventions in each included study should be summarized for the reader in the table of ‘Characteristics of included studies’. Additional tables or diagrams such as logic models (Chapter 2, Section 2.5.1) can assist descriptions of multi-component interventions so that review users can better assess review applicability to their context.

5.3.4.1 Integrity of interventions

The degree to which specified procedures or components of the intervention are implemented as planned can have important consequences for the findings from a study. We describe this as intervention integrity; related terms include adherence, compliance and fidelity (Carroll et al 2007). The verification of intervention integrity may be particularly important in reviews of non-pharmacological trials such as behavioural interventions and complex interventions, which are often implemented in conditions that present numerous obstacles to idealized delivery.

It is generally expected that reports of randomized trials provide detailed accounts of intervention implementation (Zwarenstein et al 2008, Moher et al 2010). In assessing whether interventions were implemented as planned, review authors should bear in mind that some interventions are standardized (with no deviations permitted in the intervention protocol), whereas others explicitly allow a degree of tailoring (Zwarenstein et al 2008). In addition, the growing field of implementation science has led to an increased awareness of the impact of setting and context on delivery of interventions (Damschroder et al 2009). (See Chapter 17, Section 17.1.2.1 for further information and discussion about how an intervention may be tailored to local conditions in order to preserve its integrity.)

Information about integrity can help determine whether unpromising results are due to a poorly conceptualized intervention or to an incomplete delivery of the prescribed components. It can also reveal important information about the feasibility of implementing a given intervention in real life settings. If it is difficult to achieve full implementation in practice, the intervention will have low feasibility (Dusenbury et al 2003).

Whether a lack of intervention integrity leads to a risk of bias in the estimate of its effect depends on whether review authors and users are interested in the effect of assignment to intervention or the effect of adhering to intervention, as discussed in more detail in Chapter 8, Section 8.2.2. Assessment of deviations from intended interventions is important for assessing risk of bias in the latter, but not the former (see Chapter 8, Section 8.4), but both may be of interest to decision makers in different ways.

An example of a Cochrane Review evaluating intervention integrity is provided by a review of smoking cessation in pregnancy (Chamberlain et al 2017). The authors found that process evaluation of the intervention occurred in only some trials and that the implementation was less than ideal in others, including some of the largest trials. The review highlighted how the transfer of an intervention from one setting to another may reduce its effectiveness when elements are changed, or aspects of the materials are culturally inappropriate.

5.3.4.2 Process evaluations

Process evaluations seek to evaluate the process (and mechanisms) between the intervention’s intended implementation and the actual effect on the outcome (Moore et al 2015). Process evaluation studies are characterized by a flexible approach to data collection and the use of numerous methods to generate a range of different types of data, encompassing both quantitative and qualitative methods. Guidance for including process evaluations in systematic reviews is provided in Chapter 21. When it is considered important, review authors should aim to collect information on whether the trial accounted for, or measured, key process factors and whether the trials that thoroughly addressed integrity showed a greater impact. Process evaluations can be a useful source of factors that potentially influence the effectiveness of an intervention.

5.3.5 Outcomes

An outcome is an event or a measurement value observed or recorded for a particular person or intervention unit in a study during or following an intervention, and that is used to assess the efficacy and safety of the studied intervention (Meinert 2012). Review authors should indicate in advance whether they plan to collect information about all outcomes measured in a study or only those outcomes of (pre-specified) interest in the review. Research has shown that trials addressing the same condition and intervention seldom agree on which outcomes are the most important, and consequently report on numerous different outcomes (Dwan et al 2014, Ismail et al 2014, Denniston et al 2015, Saldanha et al 2017a). The selection of outcomes across systematic reviews of the same condition is also inconsistent (Page et al 2014, Saldanha et al 2014, Saldanha et al 2016, Liu et al 2017). Outcomes used in trials and in systematic reviews of the same condition have limited overlap (Saldanha et al 2017a, Saldanha et al 2017b).

We recommend that only the outcomes defined in the protocol be described in detail. However, a complete list of the names of all outcomes measured may allow a more detailed assessment of the risk of bias due to missing outcome data (see Chapter 13).

Review authors should collect all five elements of an outcome (Zarin et al 2011, Saldanha et al 2014):

1. outcome domain or title (e.g. anxiety);

2. measurement tool or instrument (including definition of clinical outcomes or endpoints); for a scale, name of the scale (e.g. the Hamilton Anxiety Rating Scale), upper and lower limits, and whether a high or low score is favourable, definitions of any thresholds if appropriate;

3. specific metric used to characterize each participant’s results (e.g. post-intervention anxiety, or change in anxiety from baseline to a post-intervention time point, or post-intervention presence of anxiety (yes/no));

4. method of aggregation (e.g. mean and standard deviation of anxiety scores in each group, or proportion of people with anxiety);

5. timing of outcome measurements (e.g. assessments at end of eight-week intervention period, events occurring during eight-week intervention period).

Further considerations for economics outcomes are discussed in Chapter 20, and for patient-reported outcomes in Chapter 18.

5.3.5.1 Adverse effects

Collection of information about the harmful effects of an intervention can pose particular difficulties, discussed in detail in Chapter 19. These outcomes may be described using multiple terms, including ‘adverse event’, ‘adverse effect’, ‘adverse drug reaction’, ‘side effect’ and ‘complication’. Many of these terminologies are used interchangeably in the literature, although some are technically different. Harms might additionally be interpreted to include undesirable changes in other outcomes measured during a study, such as a decrease in quality of life where an improvement may have been anticipated.

In clinical trials, adverse events can be collected either systematically or non-systematically. Systematic collection refers to collecting adverse events in the same manner for each participant using defined methods such as a questionnaire or a laboratory test. For systematically collected outcomes representing harm, data can be collected by review authors in the same way as efficacy outcomes (see Section 5.3.5).

Non-systematic collection refers to collection of information on adverse events using methods such as open-ended questions (e.g. ‘Have you noticed any symptoms since your last visit?’), or reported by participants spontaneously. In either case, adverse events may be selectively reported based on their severity, and whether the participant suspected that the effect may have been caused by the intervention, which could lead to bias in the available data. Unfortunately, most adverse events are collected non-systematically rather than systematically, creating a challenge for review authors. The following pieces of information are useful and worth collecting (Nicole Fusco, personal communication):

any coding system or standard medical terminology used (e.g. COSTART, MedDRA), including version number;

name of the adverse events (e.g. dizziness);

reported intensity of the adverse event (e.g. mild, moderate, severe);

whether the trial investigators categorized the adverse event as ‘serious’;

whether the trial investigators identified the adverse event as being related to the intervention;

time point (most commonly measured as a count over the duration of the study);

any reported methods for how adverse events were selected for inclusion in the publication (e.g. ‘We reported all adverse events that occurred in at least 5% of participants’); and

associated results.

Different collection methods lead to very different accounting of adverse events (Safer 2002, Bent et al 2006, Ioannidis et al 2006, Carvajal et al 2011, Allen et al 2013). Non-systematic collection methods tend to underestimate how frequently an adverse event occurs. It is particularly problematic when the adverse event of interest to the review is collected systematically in some studies but non-systematically in other studies. Different collection methods introduce an important source of heterogeneity. In addition, when non-systematic adverse events are reported based on quantitative selection criteria (e.g. only adverse events that occurred in at least 5% of participants were included in the publication), use of reported data alone may bias the results of meta-analyses. Review authors should be cautious of (or refrain from) synthesizing adverse events that are collected differently.

Regardless of the collection methods, precise definitions of adverse effect outcomes and their intensity should be recorded, since they may vary between studies. For example, in a review of aspirin and gastrointestinal haemorrhage, some trials simply reported gastrointestinal bleeds, while others reported specific categories of bleeding, such as haematemesis, melaena, and proctorrhagia (Derry and Loke 2000). The definition and reporting of severity of the haemorrhages (e.g. major, severe, requiring hospital admission) also varied considerably among the trials (Zanchetti and Hansson 1999). Moreover, a particular adverse effect may be described or measured in different ways among the studies. For example, the terms ‘tiredness’, ‘fatigue’ or ‘lethargy’ may all be used in reporting of adverse effects. Study authors also may use different thresholds for ‘abnormal’ results (e.g. hypokalaemia diagnosed at a serum potassium concentration of 3.0 mmol/L or 3.5 mmol/L).

No mention of adverse events in trial reports does not necessarily mean that no adverse events occurred. It is usually safest to assume that they were not reported. Quality of life measures are sometimes used as a measure of the participants’ experience during the study, but these are usually general measures that do not look specifically at particular adverse effects of the intervention. While quality of life measures are important and can be used to gauge overall participant well-being, they should not be regarded as substitutes for a detailed evaluation of safety and tolerability.

5.3.6 Results

Results data arise from the measurement or ascertainment of outcomes for individual participants in an intervention study. Results data may be available for each individual in a study (i.e. individual participant data; see Chapter 26), or summarized at arm level, or summarized at study level into an intervention effect by comparing two intervention arms. Results data should be collected only for the intervention groups and outcomes specified to be of interest in the protocol (see MECIR Box 5.3.b). Results for other outcomes should not be collected unless the protocol is modified to add them. Any modification should be reported in the review. However, review authors should be alert to the possibility of important, unexpected findings, particularly serious adverse effects.

MECIR Box 5.3.b Relevant expectations for conduct of intervention reviews

C50: Choosing intervention groups in multi-arm studies (Mandatory)

If a study is included with more than two intervention arms, include in the review only interventions that meet the eligibility criteria.

There is no point including irrelevant interventions in the review. Authors should, however, make it clear in the table of ‘Characteristics of included studies’ that these interventions were present in the study.

Reports of studies often include several results for the same outcome. For example, different measurement scales might be used, results may be presented separately for different subgroups, and outcomes may have been measured at different follow-up time points. Variation in the results can be very large, depending on which data are selected (Gøtzsche et al 2007, Mayo-Wilson et al 2017a). Review protocols should be as specific as possible about which outcome domains, measurement tools, time points, and summary statistics (e.g. final values versus change from baseline) are to be collected (Mayo-Wilson et al 2017b). A framework should be pre-specified in the protocol to facilitate making choices between multiple eligible measures or results. For example, a hierarchy of preferred measures might be created, or plans articulated to select the result with the median effect size, or to average across all eligible results for a particular outcome domain (see also Chapter 9, Section 9.3.3). Any additional decisions or changes to this framework made once the data are collected should be reported in the review as changes to the protocol.

Section 5.6 describes the numbers that will be required to perform meta-analysis, if appropriate. The unit of analysis (e.g. participant, cluster, body part, treatment period) should be recorded for each result when it is not obvious (see Chapter 6, Section 6.2). The type of outcome data determines the nature of the numbers that will be sought for each outcome. For example, for a dichotomous (‘yes’ or ‘no’) outcome, the number of participants and the number who experienced the outcome will be sought for each group. It is important to collect the sample size relevant to each result, although this is not always obvious. A flow diagram as recommended in the CONSORT Statement (Moher et al 2001) can help to determine the flow of participants through a study. If one is not available in a published report, review authors can consider drawing one (available from www.consort-statement.org).

The numbers required for meta-analysis are not always available. Often, other statistics can be collected and converted into the required format. For example, for a continuous outcome, it is usually most convenient to seek the number of participants, the mean and the standard deviation for each intervention group. These are often not available directly, especially the standard deviation. Alternative statistics enable calculation or estimation of the missing standard deviation (such as a standard error, a confidence interval, a test statistic (e.g. from a t-test or F-test) or a P value). These should be extracted if they provide potentially useful information (see MECIR Box 5.3.c). Details of recalculation are provided in Section 5.6. Further considerations for dealing with missing data are discussed in Chapter 10, Section 10.12.

MECIR Box 5.3.c Relevant expectations for conduct of intervention reviews

C47: Making maximal use of data (Mandatory)

Collect and utilize the most detailed numerical data that might facilitate similar analyses of included studies. Where 2×2 tables or means and standard deviations are not available, this might include effect estimates (e.g. odds ratios, regression coefficients), confidence intervals, test statistics (e.g. t, F, Z, Chi2) or P values, or even data for individual participants

Data entry into RevMan is easiest when 2×2 tables are reported for dichotomous outcomes, and when means and standard deviations are presented for continuous outcomes. Sometimes these statistics are not reported but some manipulations of the reported data can be performed to obtain them. For instance, 2×2 tables can often be derived from sample sizes and percentages, while standard deviations can often be computed using confidence intervals or P values. Furthermore, the inverse-variance data entry format can be used even if the detailed data required for dichotomous or continuous data are not available, for instance if only odds ratios and their confidence intervals are presented. The RevMan calculator facilitates many of these manipulations.

C51: Checking accuracy of numeric data in the review (Mandatory)

Compare magnitude and direction of effects reported by studies with how they are presented in the review, taking account of legitimate differences.

This is a reasonably straightforward way for authors to check a number of potential problems, including typographical errors in studies’ reports, accuracy of data collection and manipulation, and data entry into RevMan. For example, the direction of a standardized mean difference may accidentally be wrong in the review. A basic check is to ensure the same qualitative findings (e.g. direction of effect and statistical significance) between the data as presented in the review and the data as available from the original study. Results in forest plots should agree with data in the original report (point estimate and confidence interval) if the same effect measure and statistical model is used.

5.3.7 Other information to collect

We recommend that review authors collect the key conclusions of the included study as reported by its authors. It is not necessary to report these conclusions in the review, but they should be used to verify the results of analyses undertaken by the review authors, particularly in relation to the direction of effect. Further comments by the study authors, for example any explanations they provide for unexpected findings, may be noted. References to other studies that are cited in the study report may be useful, although review authors should be aware of the possibility of citation bias (see Chapter 7, Section 7.2.3.2). Documentation of any correspondence with the study authors is important for review transparency.

5.4 Data collection tools

5.4.1 Rationale for data collection forms

Data collection for systematic reviews should be performed using structured data collection forms (see MECIR Box 5.4.a). These can be paper forms, electronic forms (e.g. Google Form), or commercially or custom-built data systems (e.g. Covidence, EPPI-Reviewer, Systematic Review Data Repository (SRDR)) that allow online form building, data entry by several users, data sharing, and efficient data management (Li et al 2015). All different means of data collection require data collection forms.

MECIR Box 5.4.a Relevant expectations for conduct of intervention reviews

C43: Using data collection forms (Mandatory)

Use a data collection form, which has been piloted.

Review authors often have different backgrounds and level of systematic review experience. Using a data collection form ensures some consistency in the process of data extraction, and is necessary for comparing data extracted in duplicate. The completed data collection forms should be available to the CRG on request. Piloting the form within the review team is highly desirable. At minimum, the data collection form (or a very close variant of it) must have been assessed for usability.

The data collection form is a bridge between what is reported by the original investigators (e.g. in journal articles, abstracts, personal correspondence) and what is ultimately reported by the review authors. The data collection form serves several important functions (Meade and Richardson 1997). First, the form is linked directly to the review question and criteria for assessing eligibility of studies, and provides a clear summary of these that can be used to identify and structure the data to be extracted from study reports. Second, the data collection form is the historical record of the provenance of the data used in the review, as well as the multitude of decisions (and changes to decisions) that occur throughout the review process. Third, the form is the source of data for inclusion in an analysis.

Given the important functions of data collection forms, ample time and thought should be invested in their design. Because each review is different, data collection forms will vary across reviews. However, there are many similarities in the types of information that are important. Thus, forms can be adapted from one review to the next. Although we use the term ‘data collection form’ in the singular, in practice it may be a series of forms used for different purposes: for example, a separate form could be used to assess the eligibility of studies for inclusion in the review to assist in the quick identification of studies to be excluded from or included in the review.

5.4.2 Considerations in selecting data collection tools

The choice of data collection tool is largely dependent on review authors’ preferences, the size of the review, and resources available to the author team. Potential advantages and considerations of selecting one data collection tool over another are outlined in Table 5.4.a (Li et al 2015). A significant advantage that data systems have is in data management (Chapter 1, Section 1.6) and re-use. They make review updates more efficient, and also facilitate methodological research across reviews. Numerous ‘meta-epidemiological’ studies have been carried out using Cochrane Review data, resulting in methodological advances which would not have been possible if thousands of studies had not all been described using the same data structures in the same system.

Some data collection tools facilitate automatic imports of extracted data into RevMan (Cochrane’s authoring tool), such as CSV (Excel) and Covidence. Details available here https://documentation.cochrane.org/revman-kb/populate-study-data-260702462.html

Table 5.4.a Considerations in selecting data collection tools

Paper forms

Electronic forms

Data systems

Examples

Forms developed using word processing software

Microsoft Access

Google Forms

Covidence

EPPI-Reviewer

Systematic Review Data Repository (SRDR)

DistillerSR (Evidence Partners)

Doctor Evidence

Suitable review type and team sizes

Small-scale reviews (<10 included studies)

Small team with 2 to 3 data extractors in the same physical location

Small- to medium-scale reviews (10 to 20 studies)

Small to moderate-sized team with 4 to 6 data extractors

For small-, medium-, and especially large-scale reviews (>20 studies), as well as reviews that need constant updating

All team sizes, especially large teams (i.e. >6 data extractors)

Resource needs

Low

Low to medium

Low (open-access tools such as Covidence or SRDR, or tools for which authors have institutional licences)

High (commercial data systems with no access via an institutional licence)

Advantages

Do not rely on access to computer and network or internet connectivity

Can record notes and explanations easily

Require minimal software skills

Allow extracted data to be processed electronically for editing and analysis

Allow electronic data storage, sharing and collation

Easy to expand or edit forms as required

Can automate data comparison with additional programming

Can copy data to analysis software without manual re-entry, reducing errors

Specifically designed for data collection for systematic reviews

Allow online data storage, linking, and sharing

Easy to expand or edit forms as required

Can be integrated with title/abstract, full-text screening and other functions

Can link data items to locations in the report to facilitate checking

Can readily automate data comparison between independent data collection for the same study

Allow easy monitoring of progress and performance of the author team

Facilitate coordination among data collectors such as allocation of studies for collection and monitoring team progress

Allow simultaneous data entry by multiple authors

Can export data directly to analysis software

In some cases, improve public accessibility through open data sharing

Disadvantages

Inefficient and potentially unreliable because data must be entered into software for analysis and reporting

Susceptible to errors

Data collected by multiple authors must be manually collated

Difficult to amend as the review progresses

If the papers are lost, all data will need to be re-created

Require familiarity with software packages to design and use forms

Susceptible to changes in software versions

Upfront investment of resources to set up the form and train data extractors

Structured templates may not be as flexible as electronic forms

Cost of commercial data systems

Require familiarity with data systems

Susceptible to changes in software versions

5.4.3 Design of a data collection form

Regardless of whether data are collected using a paper or electronic form, or a data system, the key to successful data collection is to construct easy-to-use forms and collect sufficient and unambiguous data that faithfully represent the source in a structured and organized manner (Li et al 2015). In most cases, a document format should be developed for the form before building an electronic form or a data system. This can be distributed to others, including programmers and data analysts, and as a guide for creating an electronic form and any guidance or codebook to be used by data extractors. Review authors also should consider compatibility of any electronic form or data system with analytical software, as well as mechanisms for recording, assessing and correcting data entry errors.

Data described in multiple reports (or even within a single report) of a study may not be consistent. Review authors will need to describe how they work with multiple reports in the protocol, for example, by pre-specifying which report will be used when sources contain conflicting data that cannot be resolved by contacting the investigators. Likewise, when there is only one report identified for a study, review authors should specify the section within the report (e.g. abstract, methods, results, tables, and figures) for use in case of inconsistent information.

If review authors wish to automatically import their extracted data into RevMan, it is advised that their data collection forms match the data extraction templates available via the RevMan Knowledge Base. Details available here https://documentation.cochrane.org/revman-kb/data-extraction-templates-260702375.html.

A good data collection form should minimize the need to go back to the source documents. When designing a data collection form, review authors should involve all members of the team, that is, content area experts, authors with experience in systematic review methods and data collection form design, statisticians, and persons who will perform data extraction. Here are suggested steps and some tips for designing a data collection form, based on the informal collation of experiences from numerous review authors (Li et al 2015).

Step 1. Develop outlines of tables and figures expected to appear in the systematic review, considering the comparisons to be made between different interventions within the review, and the various outcomes to be measured. This step will help review authors decide the right amount of data to collect (not too much or too little). Collecting too much information can lead to forms that are longer than original study reports, and can be very wasteful of time. Collection of too little information, or omission of key data, can lead to the need to return to study reports later in the review process.

Step 2. Assemble and group data elements to facilitate form development. Review authors should consult Table 5.3.a, in which the data elements are grouped to facilitate form development and data collection. Note that it may be more efficient to group data elements in the order in which they are usually found in study reports (e.g. starting with reference information, followed by eligibility criteria, intervention description, statistical methods, baseline characteristics and results).

Step 3. Identify the optimal way of framing the data items. Much has been written about how to frame data items for developing robust data collection forms in primary research studies. We summarize a few key points and highlight issues that are pertinent to systematic reviews.

Ask closed-ended questions (i.e. questions that define a list of permissible responses) as much as possible. Closed-ended questions do not require post hoc coding and provide better control over data quality than open-ended questions. When setting up a closed-ended question, one must anticipate and structure possible responses and include an ‘other, specify’ category because the anticipated list may not be exhaustive. Avoid asking data extractors to summarize data into uncoded text, no matter how short it is.

Avoid asking a question in a way that the response may be left blank. Include ‘not applicable’, ‘not reported’ and ‘cannot tell’ options as needed. The ‘cannot tell’ option tags uncertain items that may promote review authors to contact study authors for clarification, especially on data items critical to reach conclusions.

Remember that the form will focus on what is reported in the article rather what has been done in the study. The study report may not fully reflect how the study was actually conducted. For example, a question ‘Did the article report that the participants were masked to the intervention?’ is more appropriate than ‘Were participants masked to the intervention?’

Where a judgement is required, record the raw data (i.e. quote directly from the source document) used to make the judgement. It is also important to record the source of information collected, including where it was found in a report or whether information was obtained from unpublished sources or personal communications. As much as possible, questions should be asked in a way that minimizes subjective interpretation and judgement to facilitate data comparison and adjudication.

Incorporate flexibility to allow for variation in how data are reported. It is strongly recommended that outcome data be collected in the format in which they were reported and transformed in a subsequent step if required. Review authors also should consider the software they will use for analysis and for publishing the review (e.g. RevMan).

Step 4. Develop and pilot-test data collection forms, ensuring that they provide data in the right format and structure for subsequent analysis. In addition to data items described in Step 2, data collection forms should record the title of the review as well as the person who is completing the form and the date of completion. Forms occasionally need revision; forms should therefore include the version number and version date to reduce the chances of using an outdated form by mistake. Because a study may be associated with multiple reports, it is important to record the study ID as well as the report ID. Definitions and instructions helpful for answering a question should appear next to the question to improve quality and consistency across data extractors (Stock 1994). Provide space for notes, regardless of whether paper or electronic forms are used.

All data collection forms and data systems should be thoroughly pilot-tested before launch (see MECIR Box 5.4.a). Testing should involve several people extracting data from at least a few articles. The initial testing focuses on the clarity and completeness of questions. Users of the form may provide feedback that certain coding instructions are confusing or incomplete (e.g. a list of options may not cover all situations). The testing may identify data that are missing from the form, or likely to be superfluous. After initial testing, accuracy of the extracted data should be checked against the source document or verified data to identify problematic areas. It is wise to draft entries for the table of ‘Characteristics of included studies’ and complete a risk of bias assessment (Chapter 8) using these pilot reports to ensure all necessary information is collected. A consensus between review authors may be required before the form is modified to avoid any misunderstandings or later disagreements. It may be necessary to repeat the pilot testing on a new set of reports if major changes are needed after the first pilot test.

Problems with the data collection form may surface after pilot testing has been completed, and the form may need to be revised after data extraction has started. When changes are made to the form or coding instructions, it may be necessary to return to reports that have already undergone data extraction. In some situations, it may be necessary to clarify only coding instructions without modifying the actual data collection form.

5.5 Extracting data from reports

5.5.1 Introduction

In most systematic reviews, the primary source of information about each study is published reports of studies, usually in the form of journal articles. Despite recent developments in machine learning models to automate data extraction in systematic reviews (see Section 5.5.9), data extraction is still largely a manual process. Electronic searches for text can provide a useful aid to locating information within a report. Examples include using search facilities in PDF viewers, internet browsers and word processing software. However, text searching should not be considered a replacement for reading the report, since information may be presented using variable terminology and presented in multiple formats.

5.5.2 Who should extract data?

Data extractors should have at least a basic understanding of the topic, and have knowledge of study design, data analysis and statistics. They should pay attention to detail while following instructions on the forms. Because errors that occur at the data extraction stage are rarely detected by peer reviewers, editors, or users of systematic reviews, it is recommended that more than one person extract data from every report to minimize errors and reduce introduction of potential biases by review authors (see MECIR Box 5.5.a). As a minimum, information that involves subjective interpretation and information that is critical to the interpretation of results (e.g. outcome data) should be extracted independently by at least two people (see MECIR Box 5.5.a). In common with implementation of the selection process (Chapter 4, Section 4.6), it is preferable that data extractors are from complementary disciplines, for example a methodologist and a topic area specialist. It is important that everyone involved in data extraction has practice using the form and, if the form was designed by someone else, receives appropriate training.

Evidence in support of duplicate data extraction comes from several indirect sources. One study observed that independent data extraction by two authors resulted in fewer errors than data extraction by a single author followed by verification by a second (Buscemi et al 2006). A high prevalence of data extraction errors (errors in 20 out of 34 reviews) has been observed (Jones et al 2005). A further study of data extraction to compute standardized mean differences found that a minimum of seven out of 27 reviews had substantial errors (Gøtzsche et al 2007).

MECIR Box 5.5.a Relevant expectations for conduct of intervention reviews

C45: Extracting study characteristics in duplicate (Highly desirable)

Use (at least) two people working independently to extract study characteristics from reports of each study, and define in advance the process for resolving disagreements.

Duplicating the data extraction process reduces both the risk of making mistakes and the possibility that data selection is influenced by a single person’s biases. Dual data extraction may be less important for study characteristics than it is for outcome data, so it is not a mandatory standard for the former.

C46: Extracting outcome data in duplicate (Mandatory)

Use (at least) two people working independently to extract outcome data from reports of each study, and define in advance the process for resolving disagreements.

Duplicating the data extraction process reduces both the risk of making mistakes and the possibility that data selection is influenced by a single person’s biases. Dual data extraction is particularly important for outcome data, which feed directly into syntheses of the evidence and hence to conclusions of the review.

5.5.3 Training data extractors

Training of data extractors is intended to familiarize them with the review topic and methods, the data collection form or data system, and issues that may arise during data extraction. Results of the pilot testing of the form should prompt discussion among review authors and extractors of ambiguous questions or responses to establish consistency. Training should take place at the onset of the data extraction process and periodically over the course of the project (Li et al 2015). For example, when data related to a single item on the form are present in multiple locations within a report (e.g. abstract, main body of text, tables, and figures) or in several sources (e.g. publications, ClinicalTrials.gov, or CSRs), the development and documentation of instructions to follow an agreed algorithm are critical and should be reinforced during the training sessions.

Some have proposed that some information in a report, such as its authors, be blinded to the review author prior to data extraction and assessment of risk of bias (Jadad et al 1996). However, blinding of review authors to aspects of study reports generally is not recommended for Cochrane Reviews as there is little evidence that it alters the decisions made (Berlin 1997).

5.5.4 Extracting data from multiple reports of the same study

Studies frequently are reported in more than one publication or in more than one source (Tramèr et al 1997, von Elm et al 2004). A single source rarely provides complete information about a study; on the other hand, multiple sources may contain conflicting information about the same study (Mayo-Wilson et al 2017a, Mayo-Wilson et al 2017b, Mayo-Wilson et al 2018). Because the unit of interest in a systematic review is the study and not the report, information from multiple reports often needs to be collated and reconciled. It is not appropriate to discard any report of an included study without careful examination, since it may contain valuable information not included in the primary report. Review authors will need to decide between two strategies:

Extract data from each report separately, then combine information across multiple data collection forms.

Extract data from all reports directly into a single data collection form.

The choice of which strategy to use will depend on the nature of the reports and may vary across studies and across reports. For example, when a full journal article and multiple conference abstracts are available, it is likely that the majority of information will be obtained from the journal article; completing a new data collection form for each conference abstract may be a waste of time. Conversely, when there are two or more detailed journal articles, perhaps relating to different periods of follow-up, then it is likely to be easier to perform data extraction separately for these articles and collate information from the data collection forms afterwards. When data from all reports are extracted into a single data collection form, review authors should identify the ‘main’ data source for each study when sources include conflicting data and these differences cannot be resolved by contacting authors (Mayo-Wilson et al 2018). Flow diagrams such as those modified from the PRISMA statement can be particularly helpful when collating and documenting information from multiple reports (Mayo-Wilson et al 2018).

5.5.5 Reliability and reaching consensus

When more than one author extracts data from the same reports, there is potential for disagreement. After data have been extracted independently by two or more extractors, responses must be compared to assure agreement or to identify discrepancies. An explicit procedure or decision rule should be specified in the protocol for identifying and resolving disagreements. Most often, the source of the disagreement is an error by one of the extractors and is easily resolved. Thus, discussion among the authors is a sensible first step. More rarely, a disagreement may require arbitration by another person. Any disagreement that cannot be resolved should be addressed by contacting the study authors; if this is unsuccessful, the disagreement should be reported in the review.

The presence and resolution of disagreements should be carefully recorded. Maintaining a copy of the data ‘as extracted’ (in addition to the consensus data) allows assessment of reliability of coding. Examples of ways in which this can be achieved include the following:

Use one author’s (paper) data collection form and record changes after consensus in a different ink colour.

Enter consensus data onto an electronic form.

Record original data extracted and consensus data in separate forms (some online tools do this automatically).

Agreement of coded items before reaching consensus can be quantified, for example using kappa statistics (Orwin 1994), although this is not routinely done in Cochrane Reviews. If agreement is assessed, this should be done only for the most important data (e.g. key risk of bias assessments, or availability of key outcomes).

Throughout the review process informal consideration should be given to the reliability of data extraction. For example, if after reaching consensus on the first few studies, the authors note a frequent disagreement for specific data, then coding instructions may need modification. Furthermore, an author’s coding strategy may change over time, as the coding rules are forgotten, indicating a need for retraining and, possibly, some recoding.

5.5.6 Extracting data from clinical study reports

Clinical study reports (CSRs) obtained for a systematic review are likely to be in PDF format. Although CSRs can be thousands of pages in length and very time-consuming to review, they typically follow the content and format required by the International Conference on Harmonisation (ICH 1995). Information in CSRs is usually presented in a structured and logical way. For example, numerical data pertaining to important demographic, efficacy, and safety variables are placed within the main text in tables and figures. Because of the clarity and completeness of information provided in CSRs, data extraction from CSRs may be clearer and conducted more confidently than from journal articles or other short reports.

To extract data from CSRs efficiently, review authors should familiarize themselves with the structure of the CSRs. In practice, review authors may want to browse or create ‘bookmarks’ within a PDF document that record section headers and subheaders and search key words related to the data extraction (e.g. randomization). In addition, it may be useful to utilize optical character recognition software to convert tables of data in the PDF to an analysable format when additional analyses are required, saving time and minimizing transcription errors.

CSRs may contain many outcomes and present many results for a single outcome (due to different analyses) (Mayo-Wilson et al 2017b). We recommend review authors extract results only for outcomes of interest to the review (Section 5.3.6). With regard to different methods of analysis, review authors should have a plan and pre-specify preferred metrics in their protocol for extracting results pertaining to different populations (e.g. ‘all randomized’, ‘all participants taking at least one dose of medication’), methods for handling missing data (e.g. ‘complete case analysis’, ‘multiple imputation’), and adjustment (e.g. unadjusted, adjusted for baseline covariates). It may be important to record the range of analysis options available, even if not all are extracted in detail. In some cases it may be preferable to use metrics that are comparable across multiple included studies, which may not be clear until data collection for all studies is complete.

CSRs are particularly useful for identifying outcomes assessed but not presented to the public. For efficacy outcomes and systematically collected adverse events, review authors can compare what is described in the CSRs with what is reported in published reports to assess the risk of bias due to missing outcome data (Chapter 8, Section 8.5) and in selection of reported result (Chapter 8, Section 8.7). Note that non-systematically collected adverse events are not amenable to such comparisons because these adverse events may not be known ahead of time and thus not pre-specified in the protocol.

5.5.7 Extracting data from regulatory reviews

Data most relevant to systematic reviews can be found in the medical and statistical review sections of a regulatory review. Both of these are substantially longer than journal articles (Turner 2013). A list of all trials on a drug usually can be found in the medical review. Because trials are referenced by a combination of numbers and letters, it may be difficult for the review authors to link the trial with other reports of the same trial (Section 5.2.1).

Many of the documents downloaded from the US Food and Drug Administration’s website for older drugs are scanned copies and are not searchable because of redaction of confidential information (Turner 2013). Optical character recognition software can convert most of the text. Reviews for newer drugs have been redacted electronically; documents remain searchable as a result.

Compared to CSRs, regulatory reviews contain less information about trial design, execution, and results. They provide limited information for assessing the risk of bias. In terms of extracting outcomes and results, review authors should follow the guidance provided for CSRs (Section 5.5.6).

5.5.8 Extracting data from figures with software

Sometimes numerical data needed for systematic reviews are only presented in figures. Review authors may request the data from the study investigators, or alternatively, extract the data from the figures either manually (e.g. with a ruler) or by using software. Numerous tools are available, many of which are free. Those available at the time of writing include tools called Plot Digitizer, WebPlotDigitizer, Engauge, Dexter, ycasd, GetData Graph Digitizer. The software works by taking an image of a figure and then digitizing the data points off the figure using the axes and scales set by the users. The numbers exported can be used for systematic reviews, although additional calculations may be needed to obtain the summary statistics, such as calculation of means and standard deviations from individual-level data points (or conversion of time-to-event data presented on Kaplan-Meier plots to hazard ratios; see Chapter 6, Section 6.8.2).

It has been demonstrated that software is more convenient and accurate than visual estimation or use of a ruler (Gross et al 2014, Jelicic Kadic et al 2016). Review authors should consider using software for extracting numerical data from figures when the data are not available elsewhere.

5.5.9 Automating data extraction in systematic reviews

Because data extraction is time-consuming and error-prone, automating or semi-automating this step may make the extraction process more efficient and accurate. The state of science relevant to automating data extraction is summarized here (Jonnalagadda et al 2015).

At least 26 studies have tested various natural language processing and machine learning approaches for facilitating data extraction for systematic reviews.

· Each tool focuses on only a limited number of data elements (ranges from one to seven). Most of the existing tools focus on the PICO information (e.g. number of participants, their age, sex, country, recruiting centres, intervention groups, outcomes, and time points). A few are able to extract study design and results (e.g. objectives, study duration, participant flow), and two extract risk of bias information (Marshall et al 2016, Millard et al 2016). To date, well over half of the data elements needed for systematic reviews have not been explored for automated extraction.

Most tools highlight the sentence(s) that may contain the data elements as opposed to directly recording these data elements into a data collection form or a data system.

There is no gold standard or common dataset to evaluate the performance of these tools, limiting our ability to interpret the significance of the reported accuracy measures.

At the time of writing, we cannot recommend a specific tool for automating data extraction for routine systematic review production. There is a need for review authors to work with experts in informatics to refine these tools and evaluate them rigorously. Such investigations should address how the tool will fit into existing workflows. For example, the automated or semi-automated data extraction approaches may first act as checks for manual data extraction before they can replace it.

5.5.10 Suspicions of scientific misconduct

Systematic review authors can uncover suspected misconduct in the published literature. Misconduct includes fabrication or falsification of data or results, plagiarism, and research that does not adhere to ethical norms. Review authors need to be aware of scientific misconduct because the inclusion of fraudulent material could undermine the reliability of a review’s findings. Plagiarism of results data in the form of duplicated publication (either by the same or by different authors) may, if undetected, lead to study participants being double counted in a synthesis.

It is preferable to identify potential problems before, rather than after, publication of the systematic review, so that readers are not misled. However, empirical evidence indicates that the extent to which systematic review authors explore misconduct varies widely (Elia et al 2016). Text-matching software and systems such as CrossCheck may be helpful for detecting plagiarism, but they can detect only matching text, so data tables or figures need to be inspected by hand or using other systems (e.g. to detect image manipulation). Lists of data such as in a meta-analysis can be a useful means of detecting duplicated studies. Furthermore, examination of baseline data can lead to suspicions of misconduct for an individual randomized trial (Carlisle et al 2015). For example, Al-Marzouki and colleagues concluded that a trial report was fabricated or falsified on the basis of highly unlikely baseline differences between two randomized groups (Al-Marzouki et al 2005).

Cochrane Review authors are advised to consult with Cochrane editors if cases of suspected misconduct are identified. Searching for comments, letters or retractions may uncover additional information. Sensitivity analyses can be used to determine whether the studies arousing suspicion are influential in the conclusions of the review. Guidance for editors for addressing suspected misconduct will be available from Cochrane’s Editorial Publishing and Policy Resource (see community.cochrane.org). Further information is available from the Committee on Publication Ethics (COPE; publicationethics.org), including a series of flowcharts on how to proceed if various types of misconduct are suspected. Cases should be followed up, typically including an approach to the editors of the journals in which suspect reports were published. It may be useful to write first to the primary investigators to request clarification of apparent inconsistencies or unusual observations.

Because investigations may take time, and institutions may not always be responsive (Wager 2011), articles suspected of being fraudulent should be classified as ‘awaiting assessment’. If a misconduct investigation indicates that the publication is unreliable, or if a publication is retracted, it should not be included in the systematic review, and the reason should be noted in the ‘excluded studies’ section.

5.5.11 Key points in planning and reporting data extraction

In summary, the methods section of both the protocol and the review should detail:

the data categories that are to be extracted;

how extracted data from each report will be verified (e.g. extraction by two review authors, independently);

whether data extraction is undertaken by content area experts, methodologists, or both;

pilot testing, training and existence of coding instructions for the data collection form;

how data are extracted from multiple reports from the same study; and

how disagreements are handled when more than one author extracts data from each report.

5.6 Extracting study results and converting to the desired format

In most cases, it is desirable to collect summary data separately for each intervention group of interest and to enter these into software in which effect estimates can be calculated, such as RevMan. Sometimes the required data may be obtained only indirectly, and the relevant results may not be obvious. Chapter 6 provides many useful tips and techniques to deal with common situations. When summary data cannot be obtained from each intervention group, or where it is important to use results of adjusted analyses (for example to account for correlations in crossover or cluster-randomized trials) effect estimates may be available directly.

5.7 Managing and sharing data

When data have been collected for each individual study, it is helpful to organize them into a comprehensive electronic format, such as a database or spreadsheet, before entering data into a meta-analysis or other synthesis. When data are collated electronically, all or a subset of them can easily be exported for cleaning, consistency checks and analysis.

Tabulation of collected information about studies can facilitate classification of studies into appropriate comparisons and subgroups. It also allows identification of comparable outcome measures and statistics across studies. It will often be necessary to perform calculations to obtain the required statistics for presentation or synthesis. It is important through this process to retain clear information on the provenance of the data, with a clear distinction between data from a source document and data obtained through calculations. Statistical conversions, for example from standard errors to standard deviations, ideally should be undertaken with a computer rather than using a hand calculator to maintain a permanent record of the original and calculated numbers as well as the actual calculations used.

Ideally, data only need to be extracted once and should be stored in a secure and stable location for future updates of the review, regardless of whether the original review authors or a different group of authors update the review (Ip et al 2012). Standardizing and sharing data collection tools as well as data management systems among review authors working in similar topic areas can streamline systematic review production. Review authors have the opportunity to work with trialists, journal editors, funders, regulators, and other stakeholders to make study data (e.g. CSRs, IPD, and any other form of study data) publicly available, increasing the transparency of research. When legal and ethical to do so, we encourage review authors to share the data used in their systematic reviews to reduce waste and to allow verification and reanalysis because data will not have to be extracted again for future use (Mayo-Wilson et al 2018).

5.8 Chapter information

Editors: Tianjing Li, Julian PT Higgins, Jonathan J Deeks

Acknowledgements: This chapter builds on earlier versions of the Handbook. For details of previous authors and editors of the Handbook, see Preface. Andrew Herxheimer, Nicki Jackson, Yoon Loke, Deirdre Price and Helen Thomas contributed text. Stephanie Taylor and Sonja Hood contributed suggestions for designing data collection forms. We are grateful to Judith Anzures, Mike Clarke, Miranda Cumpston and Peter Gøtzsche for helpful comments.

Funding: JPTH is a member of the National Institute for Health Research (NIHR) Biomedical Research Centre at University Hospitals Bristol NHS Foundation Trust and the University of Bristol. JJD received support from the NIHR Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham. JPTH received funding from National Institute for Health Research Senior Investigator award NF-SI-0617-10145. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

5.9 References

Al-Marzouki S, Evans S, Marshall T, Roberts I. Are these data real? Statistical methods for the detection of data fabrication in clinical trials. BMJ 2005; 331: 267-270.

Allen EN, Mushi AK, Massawe IS, Vestergaard LS, Lemnge M, Staedke SG, Mehta U, Barnes KI, Chandler CI. How experiences become data: the process of eliciting adverse event, medical history and concomitant medication reports in antimalarial and antiretroviral interaction trials. BMC Medical Research Methodology 2013; 13: 140.

Baudard M, Yavchitz A, Ravaud P, Perrodeau E, Boutron I. Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ 2017; 356: j448.

Bent S, Padula A, Avins AL. Better ways to question patients about adverse medical events: a randomized, controlled trial. Annals of Internal Medicine 2006; 144: 257-261.

Berlin JA. Does blinding of readers affect the results of meta-analyses? University of Pennsylvania Meta-analysis Blinding Study Group. Lancet 1997; 350: 185-186.

Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. Journal of Clinical Epidemiology 2006; 59: 697-703.

Carlisle JB, Dexter F, Pandit JJ, Shafer SL, Yentis SM. Calculating the probability of random sampling for continuous variables in submitted or published randomised controlled trials. Anaesthesia 2015; 70: 848-858.

Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implementation Science 2007; 2: 40.

Carvajal A, Ortega PG, Sainz M, Velasco V, Salado I, Arias LHM, Eiros JM, Rubio AP, Castrodeza J. Adverse events associated with pandemic influenza vaccines: Comparison of the results of a follow-up study with those coming from spontaneous reporting. Vaccine 2011; 29: 519-522.

Chamberlain C, O'Mara-Eves A, Porter J, Coleman T, Perlen SM, Thomas J, McKenzie JE. Psychosocial interventions for supporting women to stop smoking in pregnancy. Cochrane Database of Systematic Reviews 2017; 2: CD001055.

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science 2009; 4: 50.

Davis AL, Miller JD. The European Medicines Agency and publication of clinical study reports: a challenge for the US FDA. JAMA 2017; 317: 905-906.

Denniston AK, Holland GN, Kidess A, Nussenblatt RB, Okada AA, Rosenbaum JT, Dick AD. Heterogeneity of primary outcome measures used in clinical trials of treatments for intermediate, posterior, and panuveitis. Orphanet Journal of Rare Diseases 2015; 10: 97.

Derry S, Loke YK. Risk of gastrointestinal haemorrhage with long term use of aspirin: meta-analysis. BMJ 2000; 321: 1183-1187.

Doshi P, Dickersin K, Healy D, Vedula SS, Jefferson T. Restoring invisible and abandoned trials: a call for people to publish the findings. BMJ 2013; 346: f2865.

Dusenbury L, Brannigan R, Falco M, Hansen WB. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Education Research 2003; 18: 237-256.

Dwan K, Altman DG, Clarke M, Gamble C, Higgins JPT, Sterne JAC, Williamson PR, Kirkham JJ. Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials. PLoS Medicine 2014; 11: e1001666.

Elia N, von Elm E, Chatagner A, Popping DM, Tramèr MR. How do authors of systematic reviews deal with research malpractice and misconduct in original studies? A cross-sectional analysis of systematic reviews and survey of their authors. BMJ Open 2016; 6: e010442.

Gøtzsche PC. Multiple publication of reports of drug trials. European Journal of Clinical Pharmacology 1989; 36: 429-432.

Gøtzsche PC, Hróbjartsson A, Maric K, Tendal B. Data extraction errors in meta-analyses that use standardized mean differences. JAMA 2007; 298: 430-437.

Gross A, Schirm S, Scholz M. Ycasd - a tool for capturing and scaling data from graphical representations. BMC Bioinformatics 2014; 15: 219.

Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, Altman DG, Barbour V, Macdonald H, Johnston M, Lamb SE, Dixon-Woods M, McCulloch P, Wyatt JC, Chan AW, Michie S. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014; 348: g1687.

ICH. ICH Harmonised tripartite guideline: Struture and content of clinical study reports E31995. ICH1995. www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E3/E3_Guideline.pdf.

Ioannidis JPA, Mulrow CD, Goodman SN. Adverse events: The more you search, the more you find. Annals of Internal Medicine 2006; 144: 298-300.

Ip S, Hadar N, Keefe S, Parkin C, Iovin R, Balk EM, Lau J. A web-based archive of systematic review data. Systematic Reviews 2012; 1: 15.

Ismail R, Azuara-Blanco A, Ramsay CR. Variation of clinical outcomes used in glaucoma randomised controlled trials: a systematic review. British Journal of Ophthalmology 2014; 98: 464-468.

Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, McQuay H. Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials 1996; 17: 1-12.

Jelicic Kadic A, Vucic K, Dosenovic S, Sapunar D, Puljak L. Extracting data from figures with software was faster, with higher interrater reliability than manual extraction. Journal of Clinical Epidemiology 2016; 74: 119-123.

Jones AP, Remmington T, Williamson PR, Ashby D, Smyth RL. High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. Journal of Clinical Epidemiology 2005; 58: 741-742.

Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Medicine 2015; 13: 282.

Jonnalagadda SR, Goyal P, Huffman MD. Automating data extraction in systematic reviews: a systematic review. Systematic Reviews 2015; 4: 78.

Lewin S, Hendry M, Chandler J, Oxman AD, Michie S, Shepperd S, Reeves BC, Tugwell P, Hannes K, Rehfuess EA, Welch V, McKenzie JE, Burford B, Petkovic J, Anderson LM, Harris J, Noyes J. Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR). BMC Medical Research Methodology 2017; 17: 76.

Li G, Abbade LPF, Nwosu I, Jin Y, Leenus A, Maaz M, Wang M, Bhatt M, Zielinski L, Sanger N, Bantoto B, Luo C, Shams I, Shahid H, Chang Y, Sun G, Mbuagbaw L, Samaan Z, Levine MAH, Adachi JD, Thabane L. A scoping review of comparisons between abstracts and full reports in primary biomedical research. BMC Medical Research Methodology 2017; 17: 181.

Li TJ, Vedula SS, Hadar N, Parkin C, Lau J, Dickersin K. Innovations in data collection, management, and archiving for systematic reviews. Annals of Internal Medicine 2015; 162: 287-294.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Medicine 2009; 6: e1000100.

Liu ZM, Saldanha IJ, Margolis D, Dumville JC, Cullum NA. Outcomes in Cochrane systematic reviews related to wound care: an investigation into prespecification. Wound Repair and Regeneration 2017; 25: 292-308.

Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. Journal of the American Medical Informatics Association 2016; 23: 193-201.

Mayo-Wilson E, Doshi P, Dickersin K. Are manufacturers sharing data as promised? BMJ 2015; 351: h4169.

Mayo-Wilson E, Li TJ, Fusco N, Bertizzolo L, Canner JK, Cowley T, Doshi P, Ehmsen J, Gresham G, Guo N, Haythomthwaite JA, Heyward J, Hong H, Pham D, Payne JL, Rosman L, Stuart EA, Suarez-Cuervo C, Tolbert E, Twose C, Vedula S, Dickersin K. Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy. Journal of Clinical Epidemiology 2017a; 91: 95-110.

Mayo-Wilson E, Fusco N, Li TJ, Hong H, Canner JK, Dickersin K, MUDS Investigators. Multiple outcomes and analyses in clinical trials create challenges for interpretation and research synthesis. Journal of Clinical Epidemiology 2017b; 86: 39-50.

Mayo-Wilson E, Li T, Fusco N, Dickersin K. Practical guidance for using multiple data sources in systematic reviews and meta-analyses (with examples from the MUDS study). Research Synthesis Methods 2018; 9: 2-12.

Meade MO, Richardson WS. Selecting and appraising studies for a systematic review. Annals of Internal Medicine 1997; 127: 531-537.

Meinert CL. Clinical trials dictionary: Terminology and usage recommendations. Hoboken (NJ): Wiley; 2012.

Millard LAC, Flach PA, Higgins JPT. Machine learning to assist risk-of-bias assessments in systematic reviews. International Journal of Epidemiology 2016; 45: 266-277.

Moher D, Schulz KF, Altman DG. The CONSORT Statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001; 357: 1191-1194.

Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, Elbourne D, Egger M, Altman DG. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010; 340: c869.

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O'Cathain A, Tinati T, Wight D, Baird J. Process evaluation of complex interventions: Medical Research Council guidance. BMJ 2015; 350: h1258.

Orwin RG. Evaluating coding decisions. In: Cooper H, Hedges LV, editors. The Handbook of Research Synthesis. New York (NY): Russell Sage Foundation; 1994. p. 139-162.

Page MJ, McKenzie JE, Kirkham J, Dwan K, Kramer S, Green S, Forbes A. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database of Systematic Reviews 2014; 10: MR000035.

Ross JS, Mulvey GK, Hines EM, Nissen SE, Krumholz HM. Trial publication after registration in ClinicalTrials.Gov: a cross-sectional analysis. PLoS Medicine 2009; 6.

Safer DJ. Design and reporting modifications in industry-sponsored comparative psychopharmacology trials. Journal of Nervous and Mental Disease 2002; 190: 583-592.

Saldanha IJ, Dickersin K, Wang X, Li TJ. Outcomes in Cochrane systematic reviews addressing four common eye conditions: an evaluation of completeness and comparability. PloS One 2014; 9: e109400.

Saldanha IJ, Li T, Yang C, Ugarte-Gil C, Rutherford GW, Dickersin K. Social network analysis identified central outcomes for core outcome sets using systematic reviews of HIV/AIDS. Journal of Clinical Epidemiology 2016; 70: 164-175.

Saldanha IJ, Lindsley K, Do DV, Chuck RS, Meyerle C, Jones LS, Coleman AL, Jampel HD, Dickersin K, Virgili G. Comparison of clinical trial and systematic review outcomes for the 4 most prevalent eye diseases. JAMA Ophthalmology 2017a; 135: 933-940.

Saldanha IJ, Li TJ, Yang C, Owczarzak J, Williamson PR, Dickersin K. Clinical trials and systematic reviews addressing similar interventions for the same condition do not consider similar outcomes to be important: a case study in HIV/AIDS. Journal of Clinical Epidemiology 2017b; 84: 85-94.

Stewart LA, Clarke M, Rovers M, Riley RD, Simmonds M, Stewart G, Tierney JF, PRISMA-IPD Development Group. Preferred reporting items for a systematic review and meta-analysis of individual participant data: the PRISMA-IPD statement. JAMA 2015; 313: 1657-1665.

Stock WA. Systematic coding for research synthesis. In: Cooper H, Hedges LV, editors. The Handbook of Research Synthesis. New York (NY): Russell Sage Foundation; 1994. p. 125-138.

Tramèr MR, Reynolds DJ, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis: a case study. BMJ 1997; 315: 635-640.

Turner EH. How to access and process FDA drug approval packages for use in research. BMJ 2013; 347.

von Elm E, Poglia G, Walder B, Tramèr MR. Different patterns of duplicate publication: an analysis of articles used in systematic reviews. JAMA 2004; 291: 974-980.

Wager E. Coping with scientific misconduct. BMJ 2011; 343: d6586.

Wieland LS, Rutkow L, Vedula SS, Kaufmann CN, Rosman LM, Twose C, Mahendraratnam N, Dickersin K. Who has used internal company documents for biomedical and public health research and where did they find them? PloS One 2014; 9.

Zanchetti A, Hansson L. Risk of major gastrointestinal bleeding with aspirin (Authors' reply). Lancet 1999; 353: 149-150.

Zarin DA, Tse T, Williams RJ, Califf RM, Ide NC. The ClinicalTrials.gov results database: update and key issues. New England Journal of Medicine 2011; 364: 852-860.

Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, Oxman AD, Moher D. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ 2008; 337: a2390.

For permission to re-use material from the Handbook (either academic or commercial), please see here for full details.

Sharefacebooktwitterwhatsappemail

Overview

Part 1: About Cochrane Reviews

Part 2: Core methodsChapter 1: Starting a review

Chapter 2: Determining the scope of the review and the questions it will address

Chapter 3: Defining the criteria for including studies and how they will be grouped for the synthesis

Chapter 4: Searching for and selecting studies4.S1 Supplementary material: Technical supplement

4.S2 Supplementary material: Appendix of resources

Chapter 5: Collecting data

Chapter 6: Choosing effect measures and computing estimates of effect

Chapter 7: Considering bias and conflicts of interest among the included studies

Chapter 8: Assessing risk of bias in a randomized trial

Chapter 9: Summarizing study characteristics and preparing for synthesis

Chapter 10: Analysing data and undertaking meta-analyses

Chapter 11: Undertaking network meta-analyses

Chapter 12: Synthesizing and presenting findings using other methods

Chapter 13: Assessing risk of bias due to missing results in a synthesis

Chapter 14: Completing ‘Summary of findings’ tables and grading the certainty of the evidence

Chapter 15: Interpreting results and drawing conclusions

Part 3: Specific perspectives in reviews

Part 4: Other topics

Online learning

Learning events

Guides and handbooks

Trainers' Hub

Cochrane

About Cochrane

Cochrane.org

Who we are

Get involved

Consumer Network

Partners

Colloquium

In the news

Publications

Cochrane Library

Cochrane Reviews (CDSR)

Trials (CENTRAL)

Cochrane Clinical Answers

Cochrane Evidence Synthesis and Methods

Community

Cochrane Account

Training

Support

Methods

Software

Jobs and opportunities

Cochrane Store

General enquiries

Cochrane Library support

Cochrane Groups

Media

Trusted evidence.

Informed decisions.

Better health.

Index | Terms & conditions | Disclaimer | Privacy | Cookie policy

We use cookies to improve your experience on our site. OK More information

What Is Data Collection: Methods, Types, Tools

All CoursesAll Courses Log inData Science & Business AnalyticsData Science & Business AnalyticsAI & Machine LearningProject ManagementCyber SecurityCloud ComputingDevOpsBusiness and LeadershipQuality ManagementSoftware DevelopmentAgile and ScrumIT Service and ArchitectureDigital MarketingBig DataCareer Fast-trackEnterpriseOther SegmentsArticlesEbooksFree Practice TestsOn-demand WebinarsTutorialsLive WebinarsExplore our curated learning milestones for you!Click here to close suggestions!HomeResourcesData Science & Business AnalyticsWhat Is Data Collection: Methods, Types, ToolsTable of ContentsWhat is Data Collection?Why Do We Need Data Collection?What Are the Different Data Collection Methods?Data Collection ToolsThe Importance of Ensuring Accurate and Appropriate Data CollectionIssues Related to Maintaining the Integrity of Data CollectionWhat are Common Challenges in Data Collection?What are the Key Steps in the Data Collection Process?Data Collection Considerations and Best PracticesFAQsChoose the Right Data Science ProgramAre You Interested in a Career in Data Science?What Is Data Collection: Methods, Types, ToolsBy SimplilearnShare This Article:Last updated on Sep 1, 2023887962Table of ContentsWhat is Data Collection?Why Do We Need Data Collection?What Are the Different Data Collection Methods?Data Collection ToolsThe Importance of Ensuring Accurate and Appropriate Data CollectionIssues Related to Maintaining the Integrity of Data CollectionWhat are Common Challenges in Data Collection?What are the Key Steps in the Data Collection Process?Data Collection Considerations and Best PracticesFAQsChoose the Right Data Science ProgramAre You Interested in a Career in Data Science?

The process of gathering and analyzing accurate data from various sources to find answers to research problems, trends and probabilities, etc., to evaluate possible outcomes is Known as Data Collection. Knowledge is power, information is knowledge, and data is information in digitized form, at least as defined in IT. Hence, data is power. But before you can leverage that data into a successful strategy for your organization or business, you need to gather it. That’s your first step.

So, to help you get the process started, we shine a spotlight on data collection. What exactly is it? Believe it or not, it’s more than just doing a Google search! Furthermore, what are the different types of data collection? And what kinds of data collection tools and data collection techniques exist?

If you want to get up to speed about what is data collection process, you’ve come to the right place.

Transform raw data into captivating visuals with Simplilearn's hands-on Data Visualization Courses and captivate your audience. Also, master the art of data management with Simplilearn's comprehensive data management courses - unlock new career opportunities today!

What is Data Collection?

Data collection is the process of collecting and evaluating information or data from multiple sources to find answers to research problems, answer questions, evaluate outcomes, and forecast trends and probabilities. It is an essential phase in all types of research, analysis, and decision-making, including that done in the social sciences, business, and healthcare.

Accurate data collection is necessary to make informed business decisions, ensure quality assurance, and keep research integrity.

During data collection, the researchers must identify the data types, the sources of data, and what methods are being used. We will soon see that there are many different data collection methods. There is heavy reliance on data collection in research, commercial, and government fields.

Before an analyst begins collecting data, they must answer three questions first:

What’s the goal or purpose of this research?

What kinds of data are they planning on gathering?

What methods and procedures will be used to collect, store, and process the information?

Additionally, we can break up data into qualitative and quantitative types. Qualitative data covers descriptions such as color, size, quality, and appearance. Quantitative data, unsurprisingly, deals with numbers, such as statistics, poll numbers, percentages, etc.

Why Do We Need Data Collection?

Before a judge makes a ruling in a court case or a general creates a plan of attack, they must have as many relevant facts as possible. The best courses of action come from informed decisions, and information and data are synonymous.

The concept of data collection isn’t a new one, as we’ll see later, but the world has changed. There is far more data available today, and it exists in forms that were unheard of a century ago. The data collection process has had to change and grow with the times, keeping pace with technology.

Whether you’re in the world of academia, trying to conduct research, or part of the commercial sector, thinking of how to promote a new product, you need data collection to help you make better choices.

Now that you know what is data collection and why we need it, let's take a look at the different methods of data collection. While the phrase “data collection” may sound all high-tech and digital, it doesn’t necessarily entail things like computers, big data, and the internet. Data collection could mean a telephone survey, a mail-in comment card, or even some guy with a clipboard asking passersby some questions. But let’s see if we can sort the different data collection methods into a semblance of organized categories.

What Are the Different Data Collection Methods?

Primary and secondary methods of data collection are two approaches used to gather information for research or analysis purposes. Let's explore each data collection method in detail:

1. Primary Data Collection:

Primary data collection involves the collection of original data directly from the source or through direct interaction with the respondents. This method allows researchers to obtain firsthand information specifically tailored to their research objectives. There are various techniques for primary data collection, including:

a. Surveys and Questionnaires: Researchers design structured questionnaires or surveys to collect data from individuals or groups. These can be conducted through face-to-face interviews, telephone calls, mail, or online platforms.

b. Interviews: Interviews involve direct interaction between the researcher and the respondent. They can be conducted in person, over the phone, or through video conferencing. Interviews can be structured (with predefined questions), semi-structured (allowing flexibility), or unstructured (more conversational).

c. Observations: Researchers observe and record behaviors, actions, or events in their natural setting. This method is useful for gathering data on human behavior, interactions, or phenomena without direct intervention.

d. Experiments: Experimental studies involve the manipulation of variables to observe their impact on the outcome. Researchers control the conditions and collect data to draw conclusions about cause-and-effect relationships.

e. Focus Groups: Focus groups bring together a small group of individuals who discuss specific topics in a moderated setting. This method helps in understanding opinions, perceptions, and experiences shared by the participants.

2. Secondary Data Collection:

Secondary data collection involves using existing data collected by someone else for a purpose different from the original intent. Researchers analyze and interpret this data to extract relevant information. Secondary data can be obtained from various sources, including:

a. Published Sources: Researchers refer to books, academic journals, magazines, newspapers, government reports, and other published materials that contain relevant data.

b. Online Databases: Numerous online databases provide access to a wide range of secondary data, such as research articles, statistical information, economic data, and social surveys.

c. Government and Institutional Records: Government agencies, research institutions, and organizations often maintain databases or records that can be used for research purposes.

d. Publicly Available Data: Data shared by individuals, organizations, or communities on public platforms, websites, or social media can be accessed and utilized for research.

e. Past Research Studies: Previous research studies and their findings can serve as valuable secondary data sources. Researchers can review and analyze the data to gain insights or build upon existing knowledge.

Data Collection Tools

Now that we’ve explained the various techniques, let’s narrow our focus even further by looking at some specific tools. For example, we mentioned interviews as a technique, but we can further break that down into different interview types (or “tools”).

Word Association

The researcher gives the respondent a set of words and asks them what comes to mind when they hear each word.

Sentence Completion

Researchers use sentence completion to understand what kind of ideas the respondent has. This tool involves giving an incomplete sentence and seeing how the interviewee finishes it.

Role-Playing

Respondents are presented with an imaginary situation and asked how they would act or react if it was real.

In-Person Surveys

The researcher asks questions in person.

Online/Web Surveys

These surveys are easy to accomplish, but some users may be unwilling to answer truthfully, if at all.

Mobile Surveys

These surveys take advantage of the increasing proliferation of mobile technology. Mobile collection surveys rely on mobile devices like tablets or smartphones to conduct surveys via SMS or mobile apps.

Phone Surveys

No researcher can call thousands of people at once, so they need a third party to handle the chore. However, many people have call screening and won’t answer.

Observation

Sometimes, the simplest method is the best. Researchers who make direct observations collect data quickly and easily, with little intrusion or third-party bias. Naturally, it’s only effective in small-scale situations.

The Importance of Ensuring Accurate and Appropriate Data Collection

Accurate data collecting is crucial to preserving the integrity of research, regardless of the subject of study or preferred method for defining data (quantitative, qualitative). Errors are less likely to occur when the right data gathering tools are used (whether they are brand-new ones, updated versions of them, or already available).

Among the effects of data collection done incorrectly, include the following -

Erroneous conclusions that squander resources

Decisions that compromise public policy

Incapacity to correctly respond to research inquiries

Bringing harm to participants who are humans or animals

Deceiving other researchers into pursuing futile research avenues

The study's inability to be replicated and validated

When these study findings are used to support recommendations for public policy, there is the potential to result in disproportionate harm, even if the degree of influence from flawed data collecting may vary by discipline and the type of investigation.

Let us now look at the various issues that we might face while maintaining the integrity of data collection.

Issues Related to Maintaining the Integrity of Data Collection

In order to assist the errors detection process in the data gathering process, whether they were done purposefully (deliberate falsifications) or not, maintaining data integrity is the main justification (systematic or random errors).

Quality assurance and quality control are two strategies that help protect data integrity and guarantee the scientific validity of study results.

Each strategy is used at various stages of the research timeline:

Quality control - tasks that are performed both after and during data collecting

Quality assurance - events that happen before data gathering starts

Let us explore each of them in more detail now.

Quality Assurance

As data collecting comes before quality assurance, its primary goal is "prevention" (i.e., forestalling problems with data collection). The best way to protect the accuracy of data collection is through prevention. The uniformity of protocol created in the thorough and exhaustive procedures manual for data collecting serves as the best example of this proactive step.

The likelihood of failing to spot issues and mistakes early in the research attempt increases when guides are written poorly. There are several ways to show these shortcomings:

Failure to determine the precise subjects and methods for retraining or training staff employees in data collecting

List of goods to be collected, in part

There isn't a system in place to track modifications to processes that may occur as the investigation continues.

Instead of detailed, step-by-step instructions on how to deliver tests, there is a vague description of the data gathering tools that will be employed.

Uncertainty regarding the date, procedure, and identity of the person or people in charge of examining the data

Incomprehensible guidelines for using, adjusting, and calibrating the data collection equipment.

Now, let us look at how to ensure Quality Control.

Become a Data Scientist With Real-World ExperienceData Scientist Master’s ProgramExplore ProgramQuality Control

Despite the fact that quality control actions (detection/monitoring and intervention) take place both after and during data collection, the specifics should be meticulously detailed in the procedures manual. Establishing monitoring systems requires a specific communication structure, which is a prerequisite. Following the discovery of data collection problems, there should be no ambiguity regarding the information flow between the primary investigators and staff personnel. A poorly designed communication system promotes slack oversight and reduces opportunities for error detection.

Direct staff observation conference calls, during site visits, or frequent or routine assessments of data reports to spot discrepancies, excessive numbers, or invalid codes can all be used as forms of detection or monitoring. Site visits might not be appropriate for all disciplines. Still, without routine auditing of records, whether qualitative or quantitative, it will be challenging for investigators to confirm that data gathering is taking place in accordance with the manual's defined methods. Additionally, quality control determines the appropriate solutions, or "actions," to fix flawed data gathering procedures and reduce recurrences.

Problems with data collection, for instance, that call for immediate action include:

Fraud or misbehavior

Systematic mistakes, procedure violations

Individual data items with errors

Issues with certain staff members or a site's performance

Researchers are trained to include one or more secondary measures that can be used to verify the quality of information being obtained from the human subject in the social and behavioral sciences where primary data collection entails using human subjects.

For instance, a researcher conducting a survey would be interested in learning more about the prevalence of risky behaviors among young adults as well as the social factors that influence these risky behaviors' propensity for and frequency. Let us now explore the common challenges with regard to data collection.

What are Common Challenges in Data Collection?

There are some prevalent challenges faced while collecting data, let us explore a few of them to understand them better and avoid them.

Data Quality Issues

The main threat to the broad and successful application of machine learning is poor data quality. Data quality must be your top priority if you want to make technologies like machine learning work for you. Let's talk about some of the most prevalent data quality problems in this blog article and how to fix them.

Inconsistent Data

When working with various data sources, it's conceivable that the same information will have discrepancies between sources. The differences could be in formats, units, or occasionally spellings. The introduction of inconsistent data might also occur during firm mergers or relocations. Inconsistencies in data have a tendency to accumulate and reduce the value of data if they are not continually resolved. Organizations that have heavily focused on data consistency do so because they only want reliable data to support their analytics.

Data Downtime

Data is the driving force behind the decisions and operations of data-driven businesses. However, there may be brief periods when their data is unreliable or not prepared. Customer complaints and subpar analytical outcomes are only two ways that this data unavailability can have a significant impact on businesses. A data engineer spends about 80% of their time updating, maintaining, and guaranteeing the integrity of the data pipeline. In order to ask the next business question, there is a high marginal cost due to the lengthy operational lead time from data capture to insight.

Schema modifications and migration problems are just two examples of the causes of data downtime. Data pipelines can be difficult due to their size and complexity. Data downtime must be continuously monitored, and it must be reduced through automation.

Ambiguous Data

Even with thorough oversight, some errors can still occur in massive databases or data lakes. For data streaming at a fast speed, the issue becomes more overwhelming. Spelling mistakes can go unnoticed, formatting difficulties can occur, and column heads might be deceptive. This unclear data might cause a number of problems for reporting and analytics.

Become a Data Science Expert & Get Your Dream JobCaltech Post Graduate Program in Data ScienceExplore ProgramDuplicate Data

Streaming data, local databases, and cloud data lakes are just a few of the sources of data that modern enterprises must contend with. They might also have application and system silos. These sources are likely to duplicate and overlap each other quite a bit. For instance, duplicate contact information has a substantial impact on customer experience. If certain prospects are ignored while others are engaged repeatedly, marketing campaigns suffer. The likelihood of biased analytical outcomes increases when duplicate data are present. It can also result in ML models with biased training data.

Too Much Data

While we emphasize data-driven analytics and its advantages, a data quality problem with excessive data exists. There is a risk of getting lost in an abundance of data when searching for information pertinent to your analytical efforts. Data scientists, data analysts, and business users devote 80% of their work to finding and organizing the appropriate data. With an increase in data volume, other problems with data quality become more serious, particularly when dealing with streaming data and big files or databases.

Inaccurate Data

For highly regulated businesses like healthcare, data accuracy is crucial. Given the current experience, it is more important than ever to increase the data quality for COVID-19 and later pandemics. Inaccurate information does not provide you with a true picture of the situation and cannot be used to plan the best course of action. Personalized customer experiences and marketing strategies underperform if your customer data is inaccurate.

Data inaccuracies can be attributed to a number of things, including data degradation, human mistake, and data drift. Worldwide data decay occurs at a rate of about 3% per month, which is quite concerning. Data integrity can be compromised while being transferred between different systems, and data quality might deteriorate with time.

Hidden Data

The majority of businesses only utilize a portion of their data, with the remainder sometimes being lost in data silos or discarded in data graveyards. For instance, the customer service team might not receive client data from sales, missing an opportunity to build more precise and comprehensive customer profiles. Missing out on possibilities to develop novel products, enhance services, and streamline procedures is caused by hidden data.

Finding Relevant Data

Finding relevant data is not so easy. There are several factors that we need to consider while trying to find relevant data, which include -

Relevant Domain

Relevant demographics

Relevant Time period and so many more factors that we need to consider while trying to find relevant data.

Data that is not relevant to our study in any of the factors render it obsolete and we cannot effectively proceed with its analysis. This could lead to incomplete research or analysis, re-collecting data again and again, or shutting down the study.

Deciding the Data to Collect

Determining what data to collect is one of the most important factors while collecting data and should be one of the first factors while collecting data. We must choose the subjects the data will cover, the sources we will be used to gather it, and the quantity of information we will require. Our responses to these queries will depend on our aims, or what we expect to achieve utilizing your data. As an illustration, we may choose to gather information on the categories of articles that website visitors between the ages of 20 and 50 most frequently access. We can also decide to compile data on the typical age of all the clients who made a purchase from your business over the previous month.

Not addressing this could lead to double work and collection of irrelevant data or ruining your study as a whole.

Dealing With Big Data

Big data refers to exceedingly massive data sets with more intricate and diversified structures. These traits typically result in increased challenges while storing, analyzing, and using additional methods of extracting results. Big data refers especially to data sets that are quite enormous or intricate that conventional data processing tools are insufficient. The overwhelming amount of data, both unstructured and structured, that a business faces on a daily basis.

The amount of data produced by healthcare applications, the internet, social networking sites social, sensor networks, and many other businesses are rapidly growing as a result of recent technological advancements. Big data refers to the vast volume of data created from numerous sources in a variety of formats at extremely fast rates. Dealing with this kind of data is one of the many challenges of Data Collection and is a crucial step toward collecting effective data.

Low Response and Other Research Issues

Poor design and low response rates were shown to be two issues with data collecting, particularly in health surveys that used questionnaires. This might lead to an insufficient or inadequate supply of data for the study. Creating an incentivized data collection program might be beneficial in this case to get more responses.

Now, let us look at the key steps in the data collection process.

Become a Data Science Expert & Get Your Dream JobCaltech Post Graduate Program in Data ScienceExplore Program

What are the Key Steps in the Data Collection Process?

In the Data Collection Process, there are 5 key steps. They are explained briefly below -

1. Decide What Data You Want to Gather

The first thing that we need to do is decide what information we want to gather. We must choose the subjects the data will cover, the sources we will use to gather it, and the quantity of information that we would require. For instance, we may choose to gather information on the categories of products that an average e-commerce website visitor between the ages of 30 and 45 most frequently searches for.

2. Establish a Deadline for Data Collection

The process of creating a strategy for data collection can now begin. We should set a deadline for our data collection at the outset of our planning phase. Some forms of data we might want to continuously collect. We might want to build up a technique for tracking transactional data and website visitor statistics over the long term, for instance. However, we will track the data throughout a certain time frame if we are tracking it for a particular campaign. In these situations, we will have a schedule for when we will begin and finish gathering data.

3. Select a Data Collection Approach

We will select the data collection technique that will serve as the foundation of our data gathering plan at this stage. We must take into account the type of information that we wish to gather, the time period during which we will receive it, and the other factors we decide on to choose the best gathering strategy.

4. Gather Information

Once our plan is complete, we can put our data collection plan into action and begin gathering data. In our DMP, we can store and arrange our data. We need to be careful to follow our plan and keep an eye on how it's doing. Especially if we are collecting data regularly, setting up a timetable for when we will be checking in on how our data gathering is going may be helpful. As circumstances alter and we learn new details, we might need to amend our plan.

5. Examine the Information and Apply Your Findings

It's time to examine our data and arrange our findings after we have gathered all of our information. The analysis stage is essential because it transforms unprocessed data into insightful knowledge that can be applied to better our marketing plans, goods, and business judgments. The analytics tools included in our DMP can be used to assist with this phase. We can put the discoveries to use to enhance our business once we have discovered the patterns and insights in our data.

Let us now look at some data collection considerations and best practices that one might follow.

Data Collection Considerations and Best Practices

We must carefully plan before spending time and money traveling to the field to gather data. While saving time and resources, effective data collection strategies can help us collect richer, more accurate, and richer data.

Below, we will be discussing some of the best practices that we can follow for the best results -

1. Take Into Account the Price of Each Extra Data Point

Once we have decided on the data we want to gather, we need to make sure to take the expense of doing so into account. Our surveyors and respondents will incur additional costs for each additional data point or survey question.

2. Plan How to Gather Each Data Piece

There is a dearth of freely accessible data. Sometimes the data is there, but we may not have access to it. For instance, unless we have a compelling cause, we cannot openly view another person's medical information. It could be challenging to measure several types of information.

Consider how time-consuming and difficult it will be to gather each piece of information while deciding what data to acquire.

3. Think About Your Choices for Data Collecting Using Mobile Devices

Mobile-based data collecting can be divided into three categories -

IVRS (interactive voice response technology) - Will call the respondents and ask them questions that have already been recorded.

SMS data collection - Will send a text message to the respondent, who can then respond to questions by text on their phone.

Field surveyors - Can directly enter data into an interactive questionnaire while speaking to each respondent, thanks to smartphone apps.

We need to make sure to select the appropriate tool for our survey and responders because each one has its own disadvantages and advantages.

4. Carefully Consider the Data You Need to Gather

It's all too easy to get information about anything and everything, but it's crucial to only gather the information that we require.

It is helpful to consider these 3 questions:

What details will be helpful?

What details are available?

What specific details do you require?

5. Remember to Consider Identifiers

Identifiers, or details describing the context and source of a survey response, are just as crucial as the information about the subject or program that we are actually researching.

In general, adding more identifiers will enable us to pinpoint our program's successes and failures with greater accuracy, but moderation is the key.

6. Data Collecting Through Mobile Devices is the Way to Go

Although collecting data on paper is still common, modern technology relies heavily on mobile devices. They enable us to gather many various types of data at relatively lower prices and are accurate as well as quick. There aren't many reasons not to pick mobile-based data collecting with the boom of low-cost Android devices that are available nowadays.

The Ultimate Ticket to Top Data Science Job RolesPost Graduate Program In Data ScienceExplore NowFAQs

1. What is data collection with example?

Data collection is the process of collecting and analyzing information on relevant variables in a predetermined, methodical way so that one can respond to specific research questions, test hypotheses, and assess results. Data collection can be either qualitative or quantitative. Example: A company collects customer feedback through online surveys and social media monitoring to improve their products and services.

2. What are the primary data collection methods?

As is well known, gathering primary data is costly and time intensive. The main techniques for gathering data are observation, interviews, questionnaires, schedules, and surveys.

3. What are data collection tools?

The term "data collecting tools" refers to the tools/devices used to gather data, such as a paper questionnaire or a system for computer-assisted interviews. Tools used to gather data include case studies, checklists, interviews, occasionally observation, surveys, and questionnaires.

4. What’s the difference between quantitative and qualitative methods?

While qualitative research focuses on words and meanings, quantitative research deals with figures and statistics. You can systematically measure variables and test hypotheses using quantitative methods. You can delve deeper into ideas and experiences using qualitative methodologies.

5. What are quantitative data collection methods?

While there are numerous other ways to get quantitative information, the methods indicated above—probability sampling, interviews, questionnaire observation, and document review—are the most typical and frequently employed, whether collecting information offline or online.

6. What is mixed methods research?

User research that includes both qualitative and quantitative techniques is known as mixed methods research. For deeper user insights, mixed methods research combines insightful user data with useful statistics.

7. What are the benefits of collecting data?

Collecting data offers several benefits, including:

Knowledge and Insight

Evidence-Based Decision Making

Problem Identification and Solution

Validation and Evaluation

Identifying Trends and Predictions

Support for Research and Development

Policy Development

Quality Improvement

Personalization and Targeting

Knowledge Sharing and Collaboration

8. What’s the difference between reliability and validity?

Reliability is about consistency and stability, while validity is about accuracy and appropriateness. Reliability focuses on the consistency of results, while validity focuses on whether the results are actually measuring what they are intended to measure. Both reliability and validity are crucial considerations in research to ensure the trustworthiness and meaningfulness of the collected data and measurements.

Choose the Right Data Science Program

Are you thinking about pursuing a career in the field of data science? Simplilearn's Data Science courses are designed to provide you with the necessary skills and expertise to excel in this rapidly changing field. Here's a detailed comparison for your reference:

Program Name

Data Scientist Master's Program

Post Graduate Program In Data Science

Geo

All Geos

Not Applicable in US

University

Simplilearn

Purdue

Caltech

Course Duration

11 Months

Coding Experience Required

Basic

Skills You Will Learn

10+ skills including data structure, data manipulation, NumPy, Scikit-Learn, Tableau and more

8+ skills includingExploratory Data Analysis, Descriptive Statistics, Inferential Statistics, and more

8+ skills includingSupervised & Unsupervised LearningDeep LearningData Visualization, and more

Additional Benefits

Applied Learning via Capstone and 25+ Data Science Projects

Purdue Alumni Association MembershipFree IIMJobs Pro-Membership of 6 monthsResume Building Assistance

Upto 14 CEU Credits Caltech CTME Circle Membership

Cost

$$$$

Explore Program

Are You Interested in a Career in Data Science?

We live in the Data Age, and if you want a career that fully takes advantage of this, you should consider a career in data science. Simplilearn offers a Caltech Post Graduate Program in Data Science that will train you in everything you need to know to secure the perfect position. This Data Science PG program is ideal for all working professionals, covering job-critical topics like R, Python programming, machine learning algorithms, NLP concepts, and data visualization with Tableau in great detail. This is all provided via our interactive learning model with live sessions by global practitioners, practical labs, and industry projects.

Data Science & Business Analytics Courses Duration and Fees Data Science & Business Analytics programs typically range from a few weeks to several months, with fees varying based on program and institution.Program NameDurationFeesApplied AI & Data ScienceCohort Starts: 12 Mar, 20243 Months$ 1,999Post Graduate Program in Data ScienceCohort Starts: 25 Mar, 202411 Months$ 2,790Post Graduate Program in Data AnalyticsCohort Starts: 25 Mar, 20248 Months$ 2,790Caltech Post Graduate Program in Data ScienceCohort Starts: 2 Apr, 202411 Months$ 3,790Data Scientist11 Months$ 1,299Data Analyst11 Months$ 1,299Recommended ReadsManaging Data28 Feb, 2020Capped Collection in MongoDB468327 Feb, 2023Software DevelopmentAn Ultimate One-Stop Solution Guide to Collections in C# Programming With Examples333207 Feb, 2023Data Science Career Guide: A Comprehensive Playbook To Becoming A Data Scientist12 Jun, 2023Software DevelopmentDifference Between Collection and Collections in Java2926522 Dec, 2022Software DevelopmentWhat Are Java Collections and How to Implement Them?12905326 Feb, 2024prevNextGet Affiliated Certifications with Live Class programsData ScientistAdd the IBM Advantage to your Learning25 Industry-relevant Projects and Integrated labs11 monthsView ProgramCaltech Data Sciences-BootcampExclusive visit to Caltech’s Robotics Lab6 monthsView ProgramCaltech Post Graduate Program in Data ScienceEarn a program completion certificate from Caltech CTMECurriculum delivered in live online sessions by industry experts11 monthsView Program

Data Collection: What It Is, Methods & Tools + Examples

Skip to main content Skip to primary sidebar Skip to footer

QuestionPro

Products

Survey softwareOur flagship survey solution. Sophisticated tools to get the answers you need.Research SuiteTuned for researchers. Get more insights. Response based pricing.CXExperiences change the world. Deliver the best with our CX management software.WorkforceEmpower your work leaders, make informed decisions and drive employee engagement.

Solutions

IndustriesGamingAutomotiveSports and eventsEducationGovernmentTravel & HospitalityFinancial ServicesHealthcareCannabisTechnologyUse CaseNPS+CommunitiesAudienceContactless surveysMobileLivePollsMember ExperienceGDPRPositive People Science360 Feedback Surveys

Resources

BlogeBooksSurvey TemplatesCase StudiesTrainingHelp center

Features

Pricing

Language

English

Español (Spanish)

Português (Portuguese (Brazil))

Nederlands (Dutch)

العربية (Arabic)

Français (French)

Italiano (Italian)

日本語 (Japanese)

Türkçe (Turkish)

Svenska (Swedish)

Hebrew IL (Hebrew)

Call Us

+1 800 531 0228

+1 (647) 956-1242

+52 999 402 4079

+49 301 663 5782

+44 20 3650 3166

+81-3-6869-1954

+61 2 8074 5080

+971 529 852 540

Home Market Research Data Collection: What It Is, Methods & Tools + Examples

Let’s face it, no one wants to make decisions based on guesswork or gut feelings. The most important objective of data collection is to ensure that the data gathered is reliable and packed to the brim with juicy insights that can be analyzed and turned into data-driven decisions. There’s nothing better than good statistical analysis.

LEARN ABOUT: Level of Analysis

Collecting high-quality data is essential for conducting market research, analyzing user behavior, or just trying to get a handle on business operations. With the right approach and a few handy tools, gathering reliable and informative data.

So, let’s get ready to collect some data because when it comes to data collection, it’s all about the details.

Content Index

What is Data Collection?

Data Collection Methods

Data Collection Examples

Reasons to Conduct Online Research and Data Collection

Conducting Customer Surveys for Data Collection to Multiply Sales

Steps to Effectively Conduct an Online Survey for Data Collection

Survey Design for Data Collection

What is Data Collection?

Data collection is the procedure of collecting, measuring, and analyzing accurate insights for research using standard validated techniques.

Put simply, data collection is the process of gathering information for a specific purpose. It can be used to answer research questions, make informed business decisions, or improve products and services.

To collect data, we must first identify what information we need and how we will collect it. We can also evaluate a hypothesis based on collected data. In most cases, data collection is the primary and most important step for research. The approach to data collection is different for different fields of study, depending on the required information.

LEARN ABOUT: Action Research

Data Collection Methods

There are many ways to collect information when doing research. The data collection methods that the researcher chooses will depend on the research question posed. Some data collection methods include surveys, interviews, tests, physiological evaluations, observations, reviews of existing records, and biological samples. Let’s explore them.

LEARN ABOUT: Best Data Collection Tools

Phone vs. Online vs. In-Person Interviews

Essentially there are four choices for data collection – in-person interviews, mail, phone, and online. There are pros and cons to each of these modes.

In-Person Interviews

Pros: In-depth and a high degree of confidence in the data

Cons: Time-consuming, expensive, and can be dismissed as anecdotal

Mail Surveys

Pros: Can reach anyone and everyone – no barrier

Cons: Expensive, data collection errors, lag time

Phone Surveys

Pros: High degree of confidence in the data collected, reach almost anyone

Cons: Expensive, cannot self-administer, need to hire an agency

Web/Online Surveys

Pros: Cheap, can self-administer, very low probability of data errors

Cons: Not all your customers might have an email address/be on the internet, customers may be wary of divulging information online.

In-person interviews always are better, but the big drawback is the trap you might fall into if you don’t do them regularly. It is expensive to regularly conduct interviews and not conducting enough interviews might give you false positives. Validating your research is almost as important as designing and conducting it.

We’ve seen many instances where after the research is conducted – if the results do not match up with the “gut-feel” of upper management, it has been dismissed off as anecdotal and a “one-time” phenomenon. To avoid such traps, we strongly recommend that data-collection be done on an “ongoing and regular” basis.

LEARN ABOUT: Research Process Steps

This will help you compare and analyze the change in perceptions according to marketing for your products/services. The other issue here is sample size. To be confident with your research, you must interview enough people to weed out the fringe elements.

A couple of years ago there was a lot of discussion about online surveys and their statistical analysis plan. The fact that not every customer had internet connectivity was one of the main concerns.

LEARN ABOUT: Statistical Analysis Methods

Although some of the discussions are still valid, the reach of the internet as a means of communication has become vital in the majority of customer interactions. According to the US Census Bureau, the number of households with computers has doubled between 1997 and 2001.

Learn more: Quantitative Market Research

In 2001 nearly 50% of households had a computer. Nearly 55% of all households with an income of more than 35,000 have internet access, which jumps to 70% for households with an annual income of 50,000. This data is from the US Census Bureau for 2001.

There are primarily three modes of data collection that can be employed to gather feedback – Mail, Phone, and Online. The method actually used for data collection is really a cost-benefit analysis. There is no slam-dunk solution but you can use the table below to understand the risks and advantages associated with each of the mediums:

Survey Medium Cost per ResponseData Quality/IntegrityReach (ALL US Households)Paper $20 – $30 Medium100%Phone$20 – $35High 95%Online / Email$1 – $5 Medium 50-70%

Keep in mind, the reach here is defined as “All U.S. Households.” In most cases, you need to look at how many of your customers are online and determine. If all your customers have email addresses, you have a 100% reach of your customers.

Another important thing to keep in mind is the ever-increasing dominance of cellular phones over landline phones. United States FCC rules prevent automated dialing and calling cellular phone numbers and there is a noticeable trend towards people having cellular phones as the only voice communication device.

This introduces the inability to reach cellular phone customers who are dropping home phone lines in favor of going entirely wireless. Even if automated dialing is not used, another FCC rule prohibits from phoning anyone who would have to pay for the call.

Learn more: Qualitative Market Research

Multi-Mode Surveys

Surveys, where the data is collected via different modes (online, paper, phone etc.), is also another way of going. It is fairly straightforward and easy to have an online survey and have data-entry operators to enter in data (from the phone as well as paper surveys) into the system. The same system can also be used to collect data directly from the respondents.

Learn more: Survey Research

Data Collection Examples

Data collection is an important aspect of research. Let’s consider an example of a mobile manufacturer, company X, which is launching a new product variant. To conduct research about features, price range, target market, competitor analysis, etc. data has to be collected from appropriate sources.

The marketing team can conduct various data collection activities such as online surveys or focus groups.

The survey should have all the right questions about features and pricing, such as “What are the top 3 features expected from an upcoming product?” or “How much are your likely to spend on this product?” or “Which competitors provide similar products?” etc.

For conducting a focus group, the marketing team should decide the participants and the mediator. The topic of discussion and objective behind conducting a focus group should be clarified beforehand to conduct a conclusive discussion.

Data collection methods are chosen depending on the available resources. For example, conducting questionnaires and surveys would require the least resources, while focus groups require moderately high resources.

Reasons to Conduct Online Research and Data Collection

Feedback is a vital part of any organization’s growth. Whether you conduct regular focus groups to elicit information from key players or, your account manager calls up all your marquee accounts to find out how things are going – essentially they are all processes to find out from your customers’ eyes – How are we doing? What can we do better?

Online surveys are just another medium to collect feedback from your customers, employees and anyone your business interacts with. With the advent of Do-It-Yourself tools for online surveys, data collection on the internet has become really easy, cheap and effective.

Learn more: Online Research

Conducting Customer Surveys for Data Collection to Multiply Sales

It is a well-established marketing fact that acquiring a new customer is 10 times more difficult and expensive than retaining an existing one. This is one of the fundamental driving forces behind the extensive adoption and interest in CRM and related customer retention tactics.

In a research study conducted by Rice University Professor Dr. Paul Dholakia and Dr. Vicki Morwitz, published in Harvard Business Review, the experiment inferred that the simple fact of asking customers how an organization was performing by itself to deliver results proved to be an effective customer retention strategy.

In the research study, conducted over the course of a year, one set of customers were sent out a satisfaction and opinion survey and the other set was not surveyed. In the next one year, the group that took the survey saw twice the number of people continuing and renewing their loyalty towards the organization data.

Learn more: Research Design

The research study provided a couple of interesting reasons on the basis of consumer psychology, behind this phenomenon:

Satisfaction surveys boost the customers’ desire to be coddled and induce positive feelings. This crops from a section of the human psychology that intends to “appreciate” a product or service they already like or prefer. The survey feedback collection method is solely a medium to convey this. The survey is a vehicle to “interact” with the company and reinforces the customer’s commitment to the company.

Surveys may increase awareness of auxiliary products and services. Surveys can be considered modes of both inbound as well as outbound communication. Surveys are generally considered to be a data collection and analysis source. Most people are unaware of the fact that consumer surveys can also serve as a medium for distributing data. It is important to note a few caveats here.

In most countries, including the US, “selling under the guise of research” is illegal.b. However, we all know that information is distributed while collecting information.c. Other disclaimers may be included in the survey to ensure users are aware of this fact. For example: “We will collect your opinion and inform you about products and services that have come online in the last year…”

Induced Judgments: The entire procedure of asking people for their feedback can prompt them to build an opinion on something they otherwise would not have thought about. This is a very underlying yet powerful argument that can be compared to the “Product Placement” strategy currently used for marketing products in mass media like movies and television shows. One example is the extensive and exclusive use of the “mini-Cooper” in the blockbuster movie “Italian Job.” This strategy is questionable and should be used with great caution.

Surveys should be considered as a critical tool in the customer journey dialog. The best thing about surveys is its ability to carry “bi-directional” information. The research conducted by Paul Dholakia and Vicki Morwitz shows that surveys not only get you the information that is critical for your business, but also enhances and builds upon the established relationship you have with your customers.

Recent technological advances have made it incredibly easy to conduct real-time surveys and opinion polls. Online tools make it easy to frame questions and answers and create surveys on the Web. Distributing surveys via email, website links or even integration with online CRM tools like Salesforce.com have made online surveying a quick-win solution.

Learn more: Survey Research

Steps to Effectively Conduct an Online Survey for Data Collection

So, you’ve decided to conduct an online survey. There are a few questions in your mind that you would like answered, and you are looking for a fast and inexpensive way to find out more about your customers, clients, etc.

First and foremost thing you need to decide what the smart objectives of the study are. Ensure that you can phrase these objectives as questions or measurements. If you can’t, you are better off looking at other data sources like focus groups and other qualitative methods. The data collected via online surveys is dominantly quantitative in nature.

Review the basic objectives of the study. What are you trying to discover? What actions do you want to take as a result of the survey? – Answers to these questions help in validating collected data. Online surveys are just one way of collecting and quantifying data.

Learn more: Qualitative Data & Qualitative Data Collection Methods

Visualize all of the relevant information items you would like to have. What will the output survey research report look like? What charts and graphs will be prepared? What information do you need to be assured that action is warranted?

Assign ranks to each topic (1 and 2) according to their priority, including the most important topics first. Revisit these items again to ensure that the objectives, topics, and information you need are appropriate. Remember, you can’t solve the research problem if you ask the wrong questions.

How easy or difficult is it for the respondent to provide information on each topic? If it is difficult, is there an alternative medium to gain insights by asking a different question? This is probably the most important step. Online surveys have to be Precise, Clear and Concise. Due to the nature of the internet and the fluctuations involved, if your questions are too difficult to understand, the survey dropout rate will be high.

Create a sequence for the topics that are unbiased. Make sure that the questions asked first do not bias the results of the next questions. Sometimes providing too much information, or disclosing purpose of the study can create bias. Once you have a series of decided topics, you can have a basic structure of a survey. It is always advisable to add an “Introductory” paragraph before the survey to explain the project objective and what is expected of the respondent. It is also sensible to have a “Thank You” text as well as information about where to find the results of the survey when they are published.

Decide the question type according to the requirement of the answers to meet analysis requirements. Choose from an array of question types such as open-ended text questions, dichotomous, multiple choice, rank order, scaled, or constant sum (ratio scale) questions. You have to consider an important aspect – Usually difficult analysis requirements will lead to an exponentially complicated survey design. However, there are a couple of tools available to make life easier:

Page Breaks – The attention span of respondents can be very low when it comes to a long scrolling survey. Add page breaks as wherever possible. Having said that, a single question per page can also hamper response rates as it increases the time to complete the survey as well as increases the chances for dropouts.

Branching – Create smart and effective surveys with the implementation of branching wherever required. Eliminate the use of text such as, “If you answered No to Q1 then Answer Q4” – this leads to annoyance amongst respondents which result in increase survey dropout rates. Design online surveys using the branching logic so that appropriate questions are automatically routed based on previous responses.

Write the questions. Initially, write a significant number of survey questions out of which you can use the one which is best suited for the survey. Divide the survey into sections so that respondents do not get confused seeing a long list of questions.

Sequence the questions so that they are unbiased.

Repeat all of the steps above to find any major holes. Are the questions really answered? Have someone review it for you.

Time the length of the survey. A survey should take less than five minutes. At three to four research questions per minute, you are limited to about 15 questions. One open end text question counts for three multiple choice questions. Most online software tools will record the time taken for the respondents to answer questions.

Pretest the survey to 20 or more people. Obtain their feedback in detail. What were they unsure about? Did they have questions? Did they have trouble understanding what you wanted? Did they take a point of view not covered in your answers or question?

Include a few open-ended survey questions that support your survey object. This will be a type of feedback survey.

Send an email to the project survey to your test group and then email the feedback survey afterward.

This way, you can have your test group provide their opinion about the functionality as well as usability of your project survey by using the feedback survey.

Make changes to your questionnaire based on the received feedback.

Send the survey out to all your respondents!

Online surveys have, over the course of time, evolved into an effective alternative to expensive mail or telephone surveys. However, you must be aware of a few conditions that need to be met for online surveys. If you are trying to survey a sample representing the target population, please remember that not everyone is online.

Moreover, not everyone is receptive to an online survey also. Generally, the demographic segmentation of younger individuals is inclined toward responding to an online survey.

Learn More: Examples of Qualitarive Data in Education

Survey Design for Data Collection

Good survey design is crucial for accurate data collection. From question-wording to response options, let’s explore how to create effective surveys that yield valuable insights with our tips to survey design.

Writing Great Questions for data collection

Writing great questions can be considered an art. Art always requires a significant amount of hard work, practice, and help from others.

The questions in a survey need to be clear, concise, and unbiased. A poorly worded question or a question with leading language can result in inaccurate or irrelevant responses, ultimately impacting the data’s validity.

Moreover, the questions should be relevant and specific to the research objectives. Questions that are irrelevant or do not capture the necessary information can lead to incomplete or inconsistent responses too.

Avoid loaded or leading words or questions

A small change in content can produce effective results. Words such as could, should and might are all used for almost the same purpose, but may produce a 20% difference in agreement to a question. For example, “The management could.. should.. might.. have shut the factory”.

Intense words such as – prohibit or action, representing control or action, produce similar results. For example, “Do you believe Donald Trump should prohibit insurance companies from raising rates?”.

Sometimes the content is just biased. For instance, “You wouldn’t want to go to Rudolpho’s Restaurant for the organization’s annual party, would you?”

Misplaced questions

Questions should always reference the intended context, and questions placed out of order or without its requirement should be avoided. Generally, a funnel approach should be implemented – generic questions should be included in the initial section of the questionnaire as a warm-up and specific ones should follow. Toward the end, demographic or geographic questions should be included.

Mutually non-overlapping response categories

Multiple-choice answers should be mutually unique to provide distinct choices. Overlapping answer options frustrate the respondent and make interpretation difficult at best. Also, the questions should always be precise.

For example: “Do you like water juice?”

This question is vague. In which terms is the liking for orange juice is to be rated? – Sweetness, texture, price, nutrition etc.

Avoid the use of confusing/unfamiliar words

Asking about industry-related terms such as caloric content, bits, bytes, MBS, as well as other terms and acronyms can confuse respondents. Ensure that the audience understands your language level, terminology, and, above all, the question you ask.

Non-directed questions give respondents excessive leeway

In survey design for data collection, non-directed questions can give respondents excessive leeway, which can lead to vague and unreliable data. These types of questions are also known as open-ended questions, and they do not provide any structure for the respondent to follow.

For instance, a non-directed question like “What suggestions do you have for improving our shoes?” can elicit a wide range of answers, some of which may not be relevant to the research objectives. Some respondents may give short answers, while others may provide lengthy and detailed responses, making comparing and analyzing the data challenging.

To avoid these issues, it’s essential to ask direct questions that are specific and have a clear structure. Closed-ended questions, for example, offer structured response options and can be easier to analyze as they provide a quantitative measure of respondents’ opinions.

Never force questions

There will always be certain questions that cross certain privacy rules. Since privacy is an important issue for most people, these questions should either be eliminated from the survey or not be kept as mandatory. Survey questions about income, family income, status, religious and political beliefs, etc., should always be avoided as they are considered to be intruding, and respondents can choose not to answer them.

Unbalanced answer options in scales

Unbalanced answer options in scales such as Likert Scale and Semantic Scale may be appropriate for some situations and biased in others. When analyzing a pattern in eating habits, a study used a quantity scale that made obese people appear in the middle of the scale with the polar ends reflecting a state where people starve and an irrational amount to consume. There are cases where we usually do not expect poor service, such as hospitals.

Questions that cover two points

In survey design for data collection, questions that cover two points can be problematic for several reasons. These types of questions are often called “double-barreled” questions and can cause confusion for respondents, leading to inaccurate or irrelevant data.

For instance, a question like “Do you like the food and the service at the restaurant?” covers two points, the food and the service, and it assumes that the respondent has the same opinion about both. If the respondent only liked the food, their opinion of the service could affect their answer.

It’s important to ask one question at a time to avoid confusion and ensure that the respondent’s answer is focused and accurate. This also applies to questions with multiple concepts or ideas. In these cases, it’s best to break down the question into multiple questions that address each concept or idea separately.

Dichotomous questions

Dichotomous questions are used in case you want a distinct answer, such as: Yes/No or Male/Female. For example, the question “Do you think this candidate will win the election?” can be Yes or No.

Avoid the use of long questions

The use of long questions will definitely increase the time taken for completion, which will generally lead to an increase in the survey dropout rate. Multiple-choice questions are the longest and most complex, and open-ended questions are the shortest and easiest to answer.

Conclusion

Data collection is an essential part of the research process, whether you’re conducting scientific experiments, market research, or surveys. The methods and tools used for data collection will vary depending on the research type, the sample size required, and the resources available.

Several data collection methods include surveys, observations, interviews, and focus groups. We learn each method has advantages and disadvantages, and choosing the one that best suits the research goals is important.

With the rise of technology, many tools are now available to facilitate data collection, including online survey software and data visualization tools. These tools can help researchers collect, store, and analyze data more efficiently, providing greater results and accuracy.

By understanding the various methods and tools available for data collection, we can develop a solid foundation for conducting research. With these research skills, we can make informed decisions, solve problems, and contribute to advancing our understanding of the world around us.

Analyze your survey data to gauge in-depth market drivers, including competitive intelligence, purchasing behavior, and price sensitivity, with QuestionPro.

You will obtain accurate insights with various techniques, including conjoint analysis, MaxDiff analysis, sentiment analysis, TURF analysis, heatmap analysis, etc. Export quality data to external in-depth analysis tools such as SPSS and R Software, and integrate your research with external business applications. Everything you need for your data collection. Start today for free!

LEARN MORE FREE TRIAL

SHARE THIS ARTICLE:

About the author

Adi Bhat

View all posts by Adi Bhat

Primary Sidebar

Gain insights with 80+ features for free

Create, Send and Analyze Your Online Survey in under 5 mins!

Create a Free Account

Learn from the Vodafone Customer Experience Strategy

Sep 15,2023

AI and Consumer Insights: How Technology is Changing Market Research

Jan 11,2023

Lost Customer Research: What it is & how to conduct it

Aug 11,2022

BROWSE BY CATEGORY

Academic

Academic Research

Artificial Intelligence

Assessments

Audience

Brand Awareness

Business

Case Studies

Communities

Consumer Insights

Customer effort score

Customer Engagement

Customer Experience

Customer Loyalty

Customer Research

Customer Satisfaction

Employee Benefits

Employee Engagement

Employee Retention

Enterprise

Events

Forms

Friday Five

General Data Protection Regulation

Guest Post

Insights Hub

Life@QuestionPro

LivePolls

Market Research

Marketing

Mobile

Mobile App

Mobile diaries

Mobile Surveys

New Features

non-profit

NPS

Online Communities

Polls

Question Types

Questionnaire

QuestionPro

QuestionPro Products

Release Notes

Research Tools and Apps

Revenue at Risk

Startups

Survey Templates

Surveys

TCXT

Tech News

Tips

Training

Training Tips

Trending

Uncategorized

Video Learning Series

VOC

Webinar

Webinars

What’s Coming Up

Workforce

Workforce Intelligence

FooterMORE LIKE THIS

AI in Healthcare: Exploring ClinicAI + FREE eBookMar 6, 2024HRIS Integration: What it is, Benefits & How to Approach It?Mar 4, 2024Top 10 Social Listening Tools for Brand ReputationMar 1, 202416 Best Knowledge Management Software 2024Feb 29, 2024

Other categories

Academic

Academic Research

Artificial Intelligence

Assessments

Audience

Brand Awareness

Business

Case Studies

Communities

Consumer Insights

Customer effort score

Customer Engagement

Customer Experience

Customer Loyalty

Customer Research

Customer Satisfaction

Employee Benefits

Employee Engagement

Employee Retention

Enterprise

Events

Forms

Friday Five

General Data Protection Regulation

Guest Post

Insights Hub

Life@QuestionPro

LivePolls

Market Research

Marketing

Mobile

Mobile App

Mobile diaries

Mobile Surveys

New Features

non-profit

NPS

Online Communities

Polls

Question Types

Questionnaire

QuestionPro

QuestionPro Products

Release Notes

Research Tools and Apps

Revenue at Risk

Startups

Survey Templates

Surveys

TCXT

Tech News

Tips

Training

Training Tips

Trending

Uncategorized

Video Learning Series

VOC

Webinar

Webinars

What’s Coming Up

Workforce

Workforce Intelligence

Help center

Live Chat

Tour

Sample questions

Sample reports

Survey logic

Branding

Integrations

Professional services

Security

Products

Survey Software

Customer Experience

Workforce

Communities

Audience

Polls

Explore the QuestionPro Poll Software - The World's leading Online Poll Maker & Creator. Create online polls, distribute them using email and multiple other options and start analyzing poll results.

Research Edition

LivePolls

InsightsHub

Resources

Blog

Articles

eBooks

Survey Templates

Case Studies

Training

Webinars

Coronavirus Resources

Pricing

All Plans

Nonprofit

Academic

Features Comparison

Qualtrics Alternative

Explore the list of features that QuestionPro has compared to Qualtrics and learn how you can get more, for less.

SurveyMonkey Alternative

VisionCritical Alternative

Medallia Alternative

Solutions

Likert Scale

Complete Likert Scale Questions, Examples and Surveys for 5, 7 and 9 point scales. Learn everything about Likert Scale with corresponding example for each question and survey demonstrations.

Conjoint Analysis

Net Promoter Score (NPS)

Learn everything about Net Promoter Score (NPS) and the Net Promoter Question. Get a clear view on the universal Net Promoter Score Formula, how to undertake Net Promoter Score Calculation followed by a simple Net Promoter Score Example.

Offline Surveys

Customer Satisfaction Surveys

Employee Survey Software

Employee survey software & tool to create, send and analyze employee surveys. Get real-time analysis for employee satisfaction, engagement, work culture and map your employee experience from onboarding to exit!

Market Research Survey Software

Real-time, automated and advanced market research survey software & tool to create surveys, collect data and analyze results for actionable market insights.

GDPR & EU Compliance

Employee Experience

Customer Journey

Company

About us

Executive Team

In the news

Testimonials

Advisory Board

Careers

Brand

Media Kit

QuestionPro in your language

English

Español (Spanish)

Português (Portuguese (Brazil))

Nederlands (Dutch)

العربية (Arabic)

Français (French)

Italiano (Italian)

日本語 (Japanese)

Türkçe (Turkish)

Svenska (Swedish)

Hebrew IL (Hebrew)

Awards & certificates

The Experience Journal

Find innovative ideas about Experience Management from the experts

Sitemap

Privacy Statement

Cookies Settings

Just a moment...

a moment...Enable JavaScript and cookies to continue

Data Collection in Research: Examples, Steps, and FAQs

Collection in Research: Examples, Steps, and FAQsJoin thousands of product people at Insight Out Conf on April 11. Register free.Try for freeDovetail logoProductInsights hub solutionsAnalyze dataUncover deep customer insights with fast, powerful featuresStore insightsCurate and manage insights in one searchable platformScale researchUnlock the potential of customer insights at enterprise scaleBy roleProduct managersDesignersResearchersPlatformSecurityAutomationIntegrations Contact salesView pricingJoin a live demoResourcesLearnBlogOutlierGuidesDovetail AcademyProposal builderHelp centerTrust centerChangelogCareers (10)Featured readsProduct updatesDovetail retro: our biggest releases from the past yearTips and tricksHow to affinity map using the canvasProduct updatesDovetail in the Details: 21 improvements to influence, transcribe, and storeEvents and videosUpcomingProduct webinarsInspiration Contact salesView pricingJoin a live demoCustomersCommunityEnterprisePricingGo to appLog inTry for freeMenuDovetail logoCloseWhat’s newA retrospective look at our biggest releases yetProduct Insights hub solutionsAnalyze dataUncover deep customer insights with fast, powerful featuresStore insightsCurate and manage insights in one searchable platformScale researchUnlock the potential of customer insights at enterprise scaleBy roleProduct managersDesignersResearchersPlatformSecurityAutomationIntegrationsResources LearnBlogOutlierGuidesDovetail AcademyProposal builderHelp centerTrust centerChangelogCareers (10)Featured readsProduct updatesDovetail retro: our biggest releases from the past yearTips and tricksHow to affinity map using the canvasProduct updatesDovetail in the Details: 21 improvements to influence, transcribe, and storeEvents and videosUpcomingProduct webinarsInspirationCustomersCommunityEnterprisePricingGo to appLog inTry for freeProductInsightsAnalysisAutomationIntegrationsEnterprisePricingLog inRolesProduct ManagersDesignersResearchersResourcesProposal builderGuidesBlogTips and tricksBest practicesContributorsProduct updatesLive demoRoadmapSolutionsCustomer analysis softwareQualitative data analysisQualitative research transcriptionSales enablement toolSentiment analysis softwareThematic analysis softwareUX research platformUX research repositoryCommunityCustomersTemplatesSlack communityEventsOutlierTopicsResearch methodsCustomer researchUser experience (UX)Product developmentMarket researchSurveysEmployee experiencePatient experienceCompanyAbout usCareers10LegalHelpHelp centerDovetail AcademyContact usChangelogTrust centerStatusProductInsightsAnalysisAutomationIntegrationsEnterprisePricingLog inResourcesProposal builderGuidesBlogTips and tricksBest practicesContributorsProduct updatesLive demoRoadmapRolesProduct ManagersDesignersResearchersCompanyAbout usCareers10LegalHelpHelp centerDovetail AcademyContact usChangelogTrust centerStatusCommunityCustomersTemplatesSlack communityEventsOutlierSolutionsCustomer analysis softwareQualitative data analysisQualitative research transcriptionSales enablement toolSentiment analysis softwareThematic analysis softwareUX research platformUX research repositoryTopicsResearch methodsCustomer researchUser experience (UX)Product developmentMarket researchSurveysEmployee experiencePatient experience© Dovetail Research Pty. Ltd.TermsPrivacy PolicyGuidesResearch methodsData collection in research: Your complete guideData collection in research: Your complete guideLast updated31 January 2023AuthorDovetail Editorial TeamReviewed byCathy HeathIn the late 16th century, Francis Bacon coined the phrase "knowledge is power," which implies that knowledge is a powerful force, like physical strength. In the 21st century, knowledge in the form of data is unquestionably powerful.But data isn't something you just have - you need to collect it. This means utilizing a data collection process and turning the collected data into knowledge that you can leverage into a successful strategy for your business or organization.Believe it or not, there's more to data collection than just conducting a Google search. In this complete guide, we shine a spotlight on data collection, outlining what it is, types of data collection methods, common challenges in data collection, data collection techniques, and the steps involved in data collection.Analyze all your data in one placeUncover hidden nuggets in all types of qualitative data when you analyze it in DovetailAnalyze with DovetailWhat is data collection?Data collection is the process of gathering information from various sources via different research methods and consolidating it into a single database or repository so researchers can use it for further analysis. Data collection aims to provide information that individuals, businesses, and organizations can use to solve problems, track progress, and make decisions.There are two specific data collection techniques: primary and secondary data collection. Primary data collection is the process of gathering data directly from sources. It's often considered the most reliable data collection method, as researchers can collect information directly from respondents.Secondary data collection is data that has already been collected by someone else and is readily available. This data is usually less expensive and quicker to obtain than primary data.What are the different methods of data collection?There are several data collection methods, which can be either manual or automated. Manual data collection involves collecting data manually, typically with pen and paper, while computerized data collection involves using software to collect data from online sources, such as social media, website data, transaction data, etc. Here are the five most popular methods of data collection:SurveysSurveys are a very popular method of data collection that organizations can use to gather information from many people. Researchers can conduct multi-mode surveys that reach respondents in different ways, including in person, by mail, over the phone, or online.As a method of data collection, surveys have several advantages. For instance, they are relatively quick and easy to administer, you can be flexible in what you ask, and they can be tailored to collect data on various topics or from certain demographics.However, surveys also have several disadvantages. For instance, they can be expensive to administer, and the results may not represent the population as a whole. Additionally, survey data can be challenging to interpret. It may also be subject to bias if the questions are not well-designed or if the sample of people surveyed is not representative of the population of interest.InterviewsInterviews are a common method of collecting data in social science research. You can conduct interviews in person, over the phone, or even via email or online chat.Interviews are a great way to collect qualitative and quantitative data. Qualitative interviews are likely your best option if you need to collect detailed information about your subjects' experiences or opinions. If you need to collect more generalized data about your subjects' demographics or attitudes, then quantitative interviews may be a better option.Interviews are relatively quick and very flexible, allowing you to ask follow-up questions and explore topics in more depth. The downside is that interviews can be time-consuming and expensive due to the amount of information to be analyzed. They are also prone to bias, as both the interviewer and the respondent may have certain expectations or preconceptions that may influence the data.Direct observationObservation is a direct way of collecting data. It can be structured (with a specific protocol to follow) or unstructured (simply observing without a particular plan).Organizations and businesses use observation as a data collection method to gather information about their target market, customers, or competition. Businesses can learn about consumer behavior, preferences, and trends by observing people using their products or service.There are two types of observation: participatory and non-participatory. In participatory observation, the researcher is actively involved in the observed activities. This type of observation is used in ethnographic research, where the researcher wants to understand a group's culture and social norms. Non-participatory observation is when researchers observe from a distance and do not interact with the people or environment they are studying.There are several advantages to using observation as a data collection method. It can provide insights that may not be apparent through other methods, such as surveys or interviews. Researchers can also observe behavior in a natural setting, which can provide a more accurate picture of what people do and how and why they behave in a certain context.There are some disadvantages to using observation as a method of data collection. It can be time-consuming, intrusive, and expensive to observe people for extended periods. Observations can also be tainted if the researcher is not careful to avoid personal biases or preconceptions.Automated data collectionBusiness applications and websites are increasingly collecting data electronically to improve the user experience or for marketing purposes.There are a few different ways that organizations can collect data automatically. One way is through cookies, which are small pieces of data stored on a user's computer. They track a user's browsing history and activity on a site, measuring levels of engagement with a business’s products or services, for example.Another way organizations can collect data automatically is through web beacons. Web beacons are small images embedded on a web page to track a user's activity.Finally, organizations can also collect data through mobile apps, which can track user location, device information, and app usage. This data can be used to improve the user experience and for marketing purposes.Automated data collection is a valuable tool for businesses, helping improve the user experience or target marketing efforts. Businesses should aim to be transparent about how they collect and use this data.Sourcing data through information service providersOrganizations need to be able to collect data from a variety of sources, including social media, weblogs, and sensors. The process to do this and then use the data for action needs to be efficient, targeted, and meaningful.In the era of big data, organizations are increasingly turning to information service providers (ISPs) and other external data sources to help them collect data to make crucial decisions. Information service providers help organizations collect data by offering personalized services that suit the specific needs of the organizations. These services can include data collection, analysis, management, and reporting. By partnering with an ISP, organizations can gain access to the newest technology and tools to help them to gather and manage data more effectively.There are also several tools and techniques that organizations can use to collect data from external sources, such as web scraping, which collects data from websites, and data mining, which involves using algorithms to extract data from large data sets. Organizations can also use APIs (application programming interface) to collect data from external sources. APIs allow organizations to access data stored in another system and share and integrate it into their own systems.Finally, organizations can also use manual methods to collect data from external sources. This can involve contacting companies or individuals directly to request data, by using the right tools and methods to get the insights they need.What are common challenges in data collection?There are many challenges that researchers face when collecting data. Here are five common examples:Big data environmentsData collection can be a challenge in big data environments for several reasons. It can be located in different places, such as archives, libraries, or online. The sheer volume of data can also make it difficult to identify the most relevant data sets.Second, the complexity of data sets can make it challenging to extract the desired information. Third, the distributed nature of big data environments can make it difficult to collect data promptly and efficiently.Therefore it is important to have a well-designed data collection strategy to consider the specific needs of the organization and what data sets are the most relevant. Alongside this, consideration should be made regarding the tools and resources available to support data collection and protect it from unintended use.Data biasData bias is a common challenge in data collection. It occurs when data is collected from a sample that is not representative of the population of interest. There are different types of data bias, but some common ones include selection bias, self-selection bias, and response bias. Selection bias can occur when the collected data does not represent the population being studied. For example, if a study only includes data from people who volunteer to participate, that data may not represent the general population.Self-selection bias can also occur when people self-select into a study, such as by taking part only if they think they will benefit from it. Response bias happens when people respond in a way that is not honest or accurate, such as by only answering questions that make them look good. These types of data bias present a challenge because they can lead to inaccurate results and conclusions about behaviors, perceptions, and trends. Data bias can be avoided by identifying potential sources or themes of bias and setting guidelines for eliminating them.Lack of quality assurance processesOne of the biggest challenges in data collection is the lack of quality assurance processes. This can lead to several problems, including incorrect data, missing data, and inconsistencies between data sets.Quality assurance is important because there are many data sources, and each source may have different levels of quality or corruption. There are also different ways of collecting data, and data quality may vary depending on the method used. Without quality assurance processes in place, it's difficult to ensure that data is accurate and complete. This can impact the validity of research findings, leading to decision-making based on faulty data.There are several ways to improve quality assurance in data collection. These include developing clear and consistent goals and guidelines for data collection, implementing quality control measures, using standardized procedures, and employing data validation techniques. By taking these steps, you can ensure that your data is of adequate quality to inform decision-making.Limited access to dataAnother challenge in data collection is limited access to data. This can be due to several reasons, including privacy concerns, the sensitive nature of the data, security concerns, or simply the fact that data is not readily available.Legal and compliance regulationsMost countries have regulations governing how data can be collected, used, and stored. In some cases, data collected in one country may not be used in another. This means gaining a global perspective can be a challenge. For example, if a company is required to comply with the EU General Data Protection Regulation (GDPR), it may not be able to collect data from individuals in the EU without their explicit consent. This can make it difficult to collect data from a target audience.Legal and compliance regulations can be complex, and it's important to ensure that all data collected is done so in a way that complies with the relevant regulations.What are the key steps in the data collection process?There are five steps involved in the data collection process. They are:1. Decide what data you want to gatherHave a clear understanding of the questions you are asking, and then consider where the answers might lie and how you might obtain them. This saves time and resources by avoiding the collection of irrelevant data, and helps maintain the quality of your datasets. 2. Establish a deadline for data collectionEstablishing a deadline for data collection helps you avoid collecting too much data, which can be costly and time-consuming to analyze. It also allows you to plan for data analysis and prompt interpretation. Finally, it helps you meet your research goals and objectives and allows you to move forward.3. Select a data collection approachThe data collection approach you choose will depend on different factors, including the type of data you need, available resources, and the project timeline. For instance, if you need qualitative data, you might choose a focus group or interview methodology. If you need quantitative data, then a survey or observational study may be the most appropriate form of collection.4. Gather informationWhen collecting data for your business, identify your business goals first. Once you know what you want to achieve, you can start collecting data to reach those goals. The most important thing is to ensure that the data you collect is reliable and valid. Otherwise, any decisions you make using the data could result in a negative outcome for your business.5. Examine the information and apply your findingsAs a researcher, it's important to examine the data you're collecting and analyzing before you apply your findings. This is because data can be misleading, leading to inaccurate conclusions. Ask yourself whether it is what you are expecting? Is it similar to other datasets you have looked at? There are many scientific ways to examine data, but some common methods include:looking at the distribution of data pointsexamining the relationships between variableslooking for outliersBy taking the time to examine your data and noticing any patterns, strange or otherwise, you can avoid making mistakes that could invalidate your research.How qualitative analysis software streamlines the data collection processKnowledge derived from data does indeed carry power. However, if you don't convert the knowledge into action, it will remain a resource of unexploited energy and wasted potential.Luckily, data collection tools enable organizations to streamline their data collection and analysis processes and leverage the derived knowledge to grow their businesses. For instance, qualitative analysis software can be highly advantageous in data collection by streamlining the process, making it more efficient and less time-consuming.Secondly, qualitative analysis software provides a structure for data collection and analysis, ensuring that data is of high quality. It can also help to uncover patterns and relationships that would otherwise be difficult to discern. Moreover, you can use it to replace more expensive data collection methods, such as focus groups or surveys.Overall, qualitative analysis software can be valuable for any researcher looking to collect and analyze data. By increasing efficiency, improving data quality, and providing greater insights, qualitative software can help to make the research process much more efficient and effective.Learn more about qualitative research data analysis softwareContentsWhat is data collection?What are the different methods of data collection?What are common challenges in data collection?What are the key steps in the data collection process?How qualitative analysis software streamlines the data collection processGet started todayGo from raw data to valuable insights with a flexible research platformTry for freeGet started todayGo from raw data to valuable insights with a flexible research platformTry for freeContact salesEditor’s picksUnderstanding the representativeness heuristic: A deep diveLast updated: 21 September 2023Qualitative data examples to ground your understandingLast updated: 14 February 2024How to write a strong conclusion for your research paperLast updated: 17 February 2024What is information bias in research?Last updated: 19 November 2023Research report guide: Definition, types, and tipsLast updated: 5 March 2024What is the Dunning–Kruger effect?Last updated: 5 February 2024What is informed consent in research?Last updated: 19 November 202380 fascinating psychology research questions for your next projectLast updated: 15 February 2024How to synthesize user research data for more actionable insightsLast updated: 12 October 202311 social psychology research topics to explore in 2024 Last updated: 6 March 2024What you need to know about research disseminationLast updated: 5 March 2024What is a research repository, and why do you need one?Last updated: 31 January 2024Diary study templatesLast updated: 10 April 2023Latest articles11 social psychology research topics to explore in 2024 Last updated: 6 March 2024Research report guide: Definition, types, and tipsLast updated: 5 March 2024What you need to know about research disseminationLast updated: 5 March 2024How to write a strong conclusion for your research paperLast updated: 17 February 202480 fascinating psychology research questions for your next projectLast updated: 15 February 2024Qualitative data examples to ground your understandingLast updated: 14 February 2024What is the Dunning–Kruger effect?Last updated: 5 February 2024What is a research repository, and why do you need one?Last updated: 31 January 2024What is informed consent in research?Last updated: 19 November 2023What is information bias in research?Last updated: 19 November 2023How to synthesize user research data for more actionable insightsLast updated: 12 October 2023Understanding the representativeness heuristic: A deep diveLast updated: 21 September 2023Diary study templatesLast updated: 10 April 2023Related topicsProduct developmentPatient experienceResearch methodsEmployee experienceSurveysMarket researchCustomer researchUser experience (UX)ContentsWhat is data collection?What are the different methods of data collection?What are common challenges in data collection?What are the key steps in the data collection process?How qualitative analysis software streamlines the data collection processYour customer insights hubTurn data into actionable insights. Bring your customer into every decision.Try for freeProductInsightsAnalysisAutomationIntegrationsEnterprisePricingLog inRolesProduct ManagersDesignersResearchersResourcesProposal builderGuidesBlogTips and tricksBest practicesContributorsProduct updatesLive demoRoadmapSolutionsCustomer analysis softwareQualitative data analysisQualitative research transcriptionSales enablement toolSentiment analysis softwareThematic analysis softwareUX research platformUX research repositoryCommunityCustomersTemplatesSlack communityEventsOutlierTopicsResearch methodsCustomer researchUser experience (UX)Product developmentMarket researchSurveysEmployee experiencePatient experienceCompanyAbout usCareers10LegalHelpHelp centerDovetail AcademyContact usChangelogTrust centerStatusProductInsightsAnalysisAutomationIntegrationsEnterprisePricingLog inResourcesProposal builderGuidesBlogTips and tricksBest practicesContributorsProduct updatesLive demoRoadmapRolesProduct ManagersDesignersResearchersCompanyAbout usCareers10LegalHelpHelp centerDovetail AcademyContact usChangelogTrust centerStatusCommunityCustomersTemplatesSlack communityEventsOutlierSolutionsCustomer analysis softwareQualitative data analysisQualitative research transcriptionSales enablement toolSentiment analysis softwareThematic analysis softwareUX research platformUX research repositoryTopicsResearch methodsCustomer researchUser experience (UX)Product developmentMarket researchSurveysEmployee experiencePatient experience© Dovetail Research Pty. Ltd.TermsPrivacy PolicyLog in or sign upGet started with a free trialContinue with GoogleorYour work emailContinue with emailBy clicking “Continue with Google / Email” you agree to our User Terms of Service and Privacy Policy

Research Methods | Definitions, Types, Examples

FAQ

About us

Our editors

Apply as editor

Team

Jobs

Contact

My account

Orders

Upload

Account details

Logout

My account

Overview

Availability

Information package

Account details

Logout

Admin

Proofreading & Editing

Thesis

Paper

AI Proofreader

Essay Checker

PhD dissertation

APA editing

Academic editing

College admissions essay

Personal statement

English proofreading

Spanish, French, or German

About our services

Proofreading services

Proofreading & editing example

Essay coaching example

Happiness guarantee

Plagiarism Checker

Citation Tools

Citation Generator

Check your Citations

Cite with Chrome

AI Writing

AI Proofreader

Paraphrasing Tool

Grammar Checker

Summarizer

AI Detector

Knowledge Base

Proofreading & Editing

Plagiarism Checker

Citation Tools

AI Writing

Knowledge Base

FAQ

About us

My account

Admin

Nederlands

English

Deutsch

Français

Italiano

Español

Svenska

Dansk

Suomi

Norwegian Bokmål

Back

Thesis

Paper

AI Proofreader

Essay Checker

PhD dissertation

APA editing

Academic editing

College admissions essay

Personal statement

English proofreading

Spanish, French, or German

About our services

Proofreading services

Proofreading & editing example

Essay coaching example

Happiness guarantee

Back

Citation Generator

Check your Citations

Cite with Chrome

Back

AI Proofreader

Paraphrasing Tool

Grammar Checker

Summarizer

AI Detector

Back

Our editors

Apply as editor

Team

Jobs

Contact

Back

Orders

Upload

Account details

Logout

Back

Overview

Availability

Information package

Account details

Logout

Have a language expert improve your writing

Proofreading Services

Run a free plagiarism check in 10 minutes

Plagiarism Checker

Generate accurate citations for free

Citation Generator

Home

Knowledge Base

Methodology

MethodologyAn introduction to research methodsResearch approachesInductive vs. deductiveInductive vs. deductiveInductive reasoningDeductive reasoningQualitative vs. quantitativeQualitative vs. quantitativeQuantitative researchQualitative researchMixed methods researchPrimary researchSecondary researchAction researchTypes of research comparedResearch designsResearch design step by stepDescriptiveCorrelationalExperimentalQuasi-experimentalCross-sectionalLongitudinalCase studyEthnographicExploratoryExplanatoryVariables and hypothesesTypes of variablesIndependent vs. dependentExplanatory vs. responseMediator vs. moderatorExtraneous variablesConfounding variablesControl variablesCorrelation vs. causationOperationalizationWriting hypothesesReliability & validityReliability vs. validityTypes of reliabilityTypes of validityTypes of validityInternal vs. externalInternal validityEcological validityExternal validityConstruct validityContent validityCriterion validityConcurrent validityDiscriminant validityFace validityConvergent validityPredictive validityReproducibility & replicabilityRandom & systematic errorTriangulationSamplingPopulation vs. sampleInclusion and exclusion criteriaSampling methodsProbability vs. non-probability samplingProbability samplingNon-probability samplingSimple random samplingSystematic samplingStratified samplingCluster samplingQuota samplingPurposive samplingConvenience samplingMultistage samplingSnowball samplingCollecting dataStep-by-stepSurveysDoing survey researchDesigning questionnairesLikert scalesDouble-barreled questionInterviewsTypes of interviews in researchStructured interviewSemi-structured interviewUnstructured interviewFocus groupExperimentsDesigning an experimentControlled experimentsControl groupsRandom assignmentBlindingBetween-subjects designWithin-subjects designObservational studiesStep-by-stepNaturalistic observationCase–control studyCohort studyRetrospective cohort studyProspective cohort studyParticipant observationQualitative observationQuantitative observationSystematic reviewPreparing dataData cleansingTranscribing interviewsAnalyzing dataStatistical analysisContent analysisDiscourse analysisThematic analysisTextual analysisWriting up methodsMethodology in a thesisMethods in APA styleResearch ethicsPeer review

Interesting topics

AMA style

Working with sources

IEEE

Commonly confused words

Commas

Definitions

UK vs. US English

Research bias

Nouns and pronouns

College essay

Parts of speech

Sentence structure

Verbs

Common mistakes

Effective communication

Using AI tools

Fallacies

Rhetoric

APA Style 6th edition

Applying to graduate school

Statistics

Chicago Style

Language rules

Methodology

MLA Style

Research paper

Academic writing

Starting the research process

Dissertation

Essay

Tips

APA Style 7th edition

APA citation examples

Citing sources

Plagiarism

Try our other services

Proofreading & Editing

Have a human editor polish your writing to ensure your arguments are judged on merit, not grammar errors.

Get expert writing help

AI Proofreader

Get unlimited proofreading for 30 days

Try for free

Plagiarism Checker

Compare your paper to billions of pages and articles with Scribbr’s Turnitin-powered plagiarism checker.

Run a free check

Citation Generator

Generate accurate APA, MLA, and Chicago citations for free with Scribbr's Citation Generator.

Start citing

Paraphraser

Rewrite and paraphrase texts instantly with our AI-powered paraphrasing tool.

Try for free

Grammar Checker

Eliminate grammar errors and improve your writing with our free AI-powered grammar checker.

Try for free

Research Methods | Definitions, Types, Examples

Research methods are specific procedures for collecting and analyzing data. Developing your research methods is an integral part of your research design. When planning your methods, there are two key decisions you will make.

First, decide how you will collect data. Your methods depend on what type of data you need to answer your research question:

Qualitative vs. quantitative: Will your data take the form of words or numbers?

Primary vs. secondary: Will you collect original data yourself, or will you use data that has already been collected by someone else?

Descriptive vs. experimental: Will you take measurements of something as it is, or will you perform an experiment?

Second, decide how you will analyze the data.

For quantitative data, you can use statistical analysis methods to test relationships between variables.

For qualitative data, you can use methods such as thematic analysis to interpret patterns and meanings in the data.

Table of contentsMethods for collecting dataExamples of data collection methodsMethods for analyzing dataExamples of data analysis methodsOther interesting articlesFrequently asked questions about research methods

Methods for collecting data

Data is the information that you collect for the purposes of answering your research question. The type of data you need depends on the aims of your research.

Qualitative vs. quantitative data

Your choice of qualitative or quantitative data collection depends on the type of knowledge you want to develop.

For questions about ideas, experiences and meanings, or to study something that can’t be described numerically, collect qualitative data.

If you want to develop a more mechanistic understanding of a topic, or your research involves hypothesis testing, collect quantitative data.

Pros

Cons

Qualitative

Flexible – you can often adjust your methods as you go to develop new knowledge.

Can be conducted with small samples.

Can’t be analyzed statistically, and not generalizable to broader populations.

Difficult to standardize research, at higher risk for research bias.

Quantitative

Can be used to systematically describe large collections of things.

Generates reproducible knowledge.

Requires statistical training to analyze data.

Requires larger samples.

You can also take a mixed methods approach, where you use both qualitative and quantitative research methods.

Primary vs. secondary research

Primary research is any original data that you collect yourself for the purposes of answering your research question (e.g. through surveys, observations and experiments). Secondary research is data that has already been collected by other researchers (e.g. in a government census or previous scientific studies).

If you are exploring a novel research question, you’ll probably need to collect primary data. But if you want to synthesize existing knowledge, analyze historical trends, or identify patterns on a large scale, secondary data might be a better choice.

Pros

Cons

Primary

Can be collected to answer your specific research question.

You have control over the sampling and measurement methods.

More expensive and time-consuming to collect.

Requires training in data collection methods.

Secondary

Easier and faster to access.

You can collect data that spans longer timescales and broader geographical locations.

No control over how data was generated.

Requires extra processing to make sure it works for your analysis.

Descriptive vs. experimental data

In descriptive research, you collect data about your study subject without intervening. The validity of your research will depend on your sampling method.

In experimental research, you systematically intervene in a process and measure the outcome. The validity of your research will depend on your experimental design.

To conduct an experiment, you need to be able to vary your independent variable, precisely measure your dependent variable, and control for confounding variables. If it’s practically and ethically possible, this method is the best choice for answering questions about cause and effect.

Pros

Cons

Descriptive

Allows you to describe your research subject without influencing it.

Accessible – you can gather more data on a larger scale.

No control over confounding variables.

Can’t establish causality.

Experimental

More control over confounding variables.

Can establish causality.

You might influence your research subject in unexpected ways.

Usually requires more expertise and resources to collect data.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

Academic style

Vague sentences

Grammar

Style consistency

See an example

Examples of data collection methods

Research methods for collecting data

Research method

Primary or secondary?

Qualitative or quantitative?

When to use

Experiment

Primary

Quantitative

To test cause-and-effect relationships.

Survey

Primary

Quantitative

To understand general characteristics of a population.

Interview/focus group

Primary

Qualitative

To gain more in-depth understanding of a topic.

Observation

Primary

Either

To understand how something occurs in its natural setting.

Literature review

Secondary

Either

To situate your research in an existing body of work, or to evaluate trends within a research topic.

Case study

Either

To gain an in-depth understanding of a specific group or context, or when you don’t have the resources for a large study.

Methods for analyzing data

Your data analysis methods will depend on the type of data you collect and how you prepare it for analysis.

Data can often be analyzed both quantitatively and qualitatively. For example, survey responses could be analyzed qualitatively by studying the meanings of responses or quantitatively by studying the frequencies of responses.

Qualitative analysis methods

Qualitative analysis is used to understand words, ideas, and experiences. You can use it to interpret data that was collected:

From open-ended surveys and interviews, literature reviews, case studies, ethnographies, and other sources that use text rather than numbers.

Using non-probability sampling methods.

Qualitative analysis tends to be quite flexible and relies on the researcher’s judgement, so you have to reflect carefully on your choices and assumptions and be careful to avoid research bias.

Quantitative analysis methods

Quantitative analysis uses numbers and statistics to understand frequencies, averages and correlations (in descriptive studies) or cause-and-effect relationships (in experiments).

You can use quantitative analysis to interpret data that was collected either:

During an experiment.

Using probability sampling methods.

Because the data is collected and analyzed in a statistically valid way, the results of quantitative analysis can be easily standardized and shared among researchers.

Examples of data analysis methods

Research methods for analyzing data

Research method

Qualitative or quantitative?

When to use

Statistical analysis

Quantitative

To analyze data collected in a statistically valid manner (e.g. from experiments, surveys, and observations).

Meta-analysis

Quantitative

To statistically analyze the results of a large collection of studies.

Can only be applied to studies that collected data in a statistically valid manner.

Thematic analysis

Qualitative

To analyze data collected from interviews, focus groups, or textual sources.

To understand general themes in the data and how they are communicated.

Content analysis

Either

To analyze large volumes of textual or visual data collected from surveys, literature reviews, or other sources.

Can be quantitative (i.e. frequencies of words) or qualitative (i.e. meanings of words).

Prevent plagiarism. Run a free check.

Try for free

解决idea debug的时候卡在Collecting data问题_waiting until last debugger command completes-CSDN博客

解决idea debug的时候卡在Collecting data问题

最新推荐文章于 2023-06-13 11:28:27 发布

独木桥向北

最新推荐文章于 2023-06-13 11:28:27 发布

阅读量7.4k

点赞数

文章标签：

java

windows

spring

debug

本文链接：https://blog.csdn.net/weixin_42425365/article/details/116134966

版权

IDEA在DEBUG时卡住问题

waiting until last debugger command completes.

Collecting data. . .

解决方案： File->Settings->Build,Execution,Deployment->Debugger->DataViews->Java

问题原因：未知，欢迎大佬留言告知！！！

优惠劵

独木桥向北

关注

踩

觉得还不错?

一键收藏

知道了

解决idea debug的时候卡在Collecting data问题

IDEA在DEBUG时卡住问题waiting until last debugger command completes.Collecting data. . .解决方案：File->Settings->Build,Execution,Deployment->Debugger->DataViews->Java问题原因：未知，欢迎大佬留言告知！！！...

复制链接

扫一扫

Web Scraping with Python_Collecting Data from the Modern Web

06-30

Web Scraping with Python_Collecting Data from the Modern Web,英文原版pdf

collecting_and_processing_web_data

04-01

“＃collection_and_processing_web_data”

5 条评论

您还未登录，请先

后发表或查看评论

web scraping with python collecting more data from the modern web 2nd

04-25

Learn web scraping and crawling techniques to access unlimited data from any web source in any format. With this practical guide, you’ll learn how to use Python scripts and web APIs to gather and process data from thousands—or even millions—of web pages at once.

Ideal for programmers, security professionals, and web administrators familiar with Python, this book not only teaches basic web scraping mechanics, but also delves into more advanced topics, such as analyzing raw data or using scrapers for frontend website testing. Code samples are available to help you understand the concepts in practice.

Learn how to parse complicated HTML pages

Traverse multiple pages and sites

Get a general overview of APIs and how they work

Learn several methods for storing the data you scrape

Download, read, and extract data from documents

Use tools and techniques to clean badly formatted data

Read and write natural languages

Crawl through forms and logins

Understand how to scrape JavaScript

Learn image processing and text recognition

《Mystery of Mobile Data Collecting AND Processing》PDF版本下载.txt

07-17

《Mystery of Mobile Data Collecting AND Processing》PDF版本下载

12C ORA-错误汇总1 ORA-00000 to ORA-00877

tokenpocket官网下载app苹果手机|collecting data

tokenpocket官网下载app苹果手机|collecting data

解决PyCharm调试查看变量时一直显示collecting data并报错Timeout waiting for response且看不到任何内容_pycharm collecting data-CSDN博客

Data Collection | Definition, Methods & Examples

7 Data Collection Methods in Business Analytics

Chapter 5: Collecting data | Cochrane Training

What Is Data Collection: Methods, Types, Tools

Data Collection: What It Is, Methods & Tools + Examples

Just a moment...

Data Collection in Research: Examples, Steps, and FAQs

Research Methods | Definitions, Types, Examples

解决idea debug的时候卡在Collecting data问题_waiting until last debugger command completes-CSDN博客

菜单

支持

Follow

tokenpocket官网下载app苹果手机|collecting data

tokenpocket官网下载app苹果手机|collecting data

解决PyCharm调试查看变量时一直显示collecting data并报错Timeout waiting for response且看不到任何内容_pycharm collecting data-CSDN博客

Data Collection | Definition, Methods & Examples

7 Data Collection Methods in Business Analytics

Chapter 5: Collecting data | Cochrane Training

What Is Data Collection: Methods, Types, Tools

Data Collection: What It Is, Methods & Tools + Examples

Just a moment...

Data Collection in Research: Examples, Steps, and FAQs

Research Methods | Definitions, Types, Examples

解决idea debug的时候卡在Collecting data问题_waiting until last debugger command completes-CSDN博客

最近的新闻

您可能喜欢的文章

imtoken钱包下载介绍|transportation research part c

比特派下载地址|eternal

比特派转错地址了怎么办

比特派二维码怎么找

bitpie比特派转不了账怎么办

比特派扫码被盗怎么办啊

比特派被攻击了怎么办啊