GB815 Unit 4 Study Quality Appraisal Example

Which Evidence to Trust About Franchise Survival: A Unit 4 Study Quality Appraisal of Four Sources

[Student Name]

Purdue University Global

GB815: Applied Research Topics

Unit 4 Study Quality Appraisal

[Instructor Name]

[Date]

The candidate's role is composite; the four appraised studies and the appraisal tool are real, and the ratings are the candidate's judgments.

Why This Tool

The four sources differ in design. One compares cohorts of businesses, one follows a cohort of franchisors, one models the timing of loan defaults and one uses administrative records with instrumental variables. A single appraisal tool that covers several designs allows them to be compared on one footing. The Mixed Methods Appraisal Tool does this: it asks two screening questions of every study, then five design-specific questions for its category, such as quantitative nonrandomized studies. Its developers advise against reducing the ratings to a single score, preferring that each criterion be reported, a point Hong et al. (2018) reinforced after interviewing researchers who had used the tool. This appraisal reports each criterion rated yes, no or cannot tell.

Source One: Bates (1995)

Bates (1995) used Census Bureau survey data on small businesses started between 1984 and 1987 and tracked their survival to late 1991, comparing franchise and independent startups. He found that about 62 percent of franchise startups survived against about 68 percent of independent startups, although the franchise units were larger and better capitalized.

Rating as a quantitative nonrandomized study. Participants representative of the target population: yes, since the Census sample covered small businesses broadly. Appropriate measurements of exposure and outcome: yes, with survival measured from Census records rather than self-report. Complete outcome data: mostly yes, though firms that could not be traced are a concern; cannot tell how many. Confounders accounted for: partly; the franchise units were larger and better capitalized, traits that should have favored their survival, yet they survived less often, but people who buy franchises may bring traits to the business that the survey never recorded. Exposure administered as intended: yes, since franchise status was recorded. Overall: strong for its period, limited by self-selection into franchising and by age.

Source Two: Shane (1996)

Shane (1996) followed firms that began franchising in one year and examined their growth and survival over the following decade. The study found that many new franchisors failed and that the arrangements firms chose, including how heavily they relied on franchised rather than company-owned outlets, were related to their survival.

Rating. Representative participants: yes, for new franchisors in that cohort. Appropriate measurements: yes, with survival defined as a system's continued franchising in later years. Complete outcome data: cannot tell, since systems that quietly stopped franchising may be hard to distinguish from those that failed. Confounders: partly. Exposure as intended: yes. Overall: moderate to strong, but it concerns franchisors, not franchisees, so it bears on the loan question only indirectly: a franchisee joining a young system takes on the system's risk.

Source Three: Glennon and Nigro (2005)

Glennon and Nigro (2005) used loan-level data on SBA-guaranteed small business loans to model when loans defaulted, using survival analysis that accounts for the timing of default over a loan's life. Their focus was on measuring default risk, not on comparing franchise and independent borrowers.

Rating. Representative participants: yes, for SBA-guaranteed borrowers, though these are not representative of all small businesses, since SBA guarantees go to borrowers who cannot obtain conventional credit on reasonable terms. Appropriate measurements: yes, with default drawn from administrative loan records. Complete outcome data: yes, since loans are tracked to resolution. Confounders: yes, for the variables the model includes. Exposure as intended: not applicable in the same sense, since the study models risk factors rather than a single exposure. Overall: strong for its purpose and directly useful as a method for the candidate's Direction A.

Source Four: Kalnins and Lafontaine (2013)

Kalnins and Lafontaine (2013) used records of more than 1.7 million Texas businesses that collect sales tax to show that a business located far from its owner's head office tended to close earlier, and used instrumental-variable and matched-pair analyses to argue that the effect is causal, reflecting the difficulty of monitoring distant units.

Rating. Representative participants: yes, for Texas sales-tax collectors. Appropriate measurements: yes. Complete outcome data: yes, from administrative records. Confounders: yes, with designs aimed at endogeneity. Exposure as intended: yes. Overall: strong, with relevance to franchising through the monitoring mechanism rather than through franchise status directly.

The strongest studies answer questions next to the lender's question, and the question itself, how franchise loans perform against independent ones, has not been asked well.

The Claim the Committee Repeats

The loan committee's belief that franchises succeed has no source I could find. Figures claiming that a large majority of franchises succeed circulate widely in marketing, but none discloses its sample, definitions or method. Against the appraisal tool's screening questions, such a claim fails at the first step: there is no clear research question or data to appraise. The appraisal cannot rate it lower than the other sources; it simply cannot rate it at all. That difference matters when a committee treats an unsourced figure and a Census-based study as if they were equal pieces of evidence, which is what happens whenever both are quoted from memory in the same meeting.

What a Stronger Source Would Look Like

Rating the four sources also reveals what the ideal source for the lender's question would contain. It would follow a defined cohort of new businesses, franchise and independent, from the date of their first loan. It would measure the outcome a lender cares about, whether the loan was repaid, from administrative records rather than owner reports. It would compare borrowers within the same industry, loan size and year, so that franchise status is not confused with the kinds of businesses franchises tend to be, such as restaurants. It would be recent enough to reflect current franchise systems and lending standards. And it would report how many borrowers could not be classified, so that readers could judge the completeness of the comparison. None of the four sources meets all of these conditions, because none was designed to answer this question. The SBA's public loan records contain most of the ingredients, which is why the next unit audits them.

What the Appraisal Implies

Two conclusions follow. First, the best direct evidence on franchise survival, Bates (1995), is old and points against the committee's belief, which should at least make the belief a hypothesis rather than a rule. Second, the method of Glennon and Nigro could be applied to compare franchise and independent borrowers in current SBA data, which would answer the lender's question directly. That is the strongest reason to favor Direction A.

References

Bates, T. (1995). A comparison of franchise and independent small business survival rates. Small Business Economics, 7(5), 377-388. https://doi.org/10.1007/BF01302738

Glennon, D., & Nigro, P. (2005). Measuring the default risk of small business loans: A survival analysis approach. Journal of Money, Credit and Banking, 37(5), 923-947. https://doi.org/10.1353/mcb.2005.0051

Hong, Q. N., Gonzalez-Reyes, A., & Pluye, P. (2018). Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). Journal of Evaluation in Clinical Practice, 24(3), 459-467. https://doi.org/10.1111/jep.12884

Kalnins, A., & Lafontaine, F. (2013). Too far away? The effect of distance to headquarters on business establishment performance. American Economic Journal: Microeconomics, 5(3), 157-179. https://doi.org/10.1257/mic.5.3.157

Shane, S. A. (1996). Hybrid organizational arrangements and their implications for firm growth and survival: A study of new franchisors. Academy of Management Journal, 39(1), 216-234. https://doi.org/10.2307/256637