Better Models for Education

A cautionary tale

Four years ago leading universities jumped into the bandwagon of massive open online courses. They didn’t get much more attention since then:

C98510C2-08E0-494A-9922-F7DC114F39FE
Google Trends

This is international data. In the US, interest in MOOCs declined, despite respectable institutions kept offering new courses on various topics. Is it a marketing failure, which best universities would be proud of, or a bad educational technology?

Let’s see. A typical MOOC consists of

  • lecture slides and exercises
  • a talking head that reads the slides
  • a discussion board, barely alive
  • an optional certificate

Despite many professors having good presentation skills, this technology is not different from a textbook. In fact, ten years before MOOCs, the MIT offered a much better solution: OpenCourseWare — a guideline how to study like an MIT student. It wasn’t tied to particular enrollment dates, pace, or lecturer. Instead, it showed what a diligent student should complete in one semester.

MOOCs became popular after Sebastian Thrun and Peter Norvig had released their open AI course. More than 100,000 students had enrolled, and universities decided to supply more courses. But the AI course was backed by new exciting technologies like self-driving cars and text recognition, while a standard university course covered boring rudiments available in any textbook.

The quality of online courses didn’t improve over time. Each professor appreciated his own brand and didn’t collaborate with colleagues from other universities. So each one had his own course, that is, slides and exercises. For example, a large MOOC provider offers 609 “data science” courses. Students enroll in just a dozen of them, when the lecturer already has a very good reputation. Like Andrew Ng and his machine learning course based on Stanford’s CS229 and available online since 1999.

The history of MOOCs shows how a lot of smart people keep making things that don’t work. Interestingly, it has to do with their core competencies and not online education itself.

Because someone else did better.

Y Combinator: Engaging educators

University professors have little motivation to work with students. Richard Feynman described teaching as “something [to do] so that when I don’t have any ideas and I’m not getting anywhere I can say to myself, ‘At least I’m living; at least I’m doing something; I’m making some contribution’—it’s just psychological.” So when it comes to research vs teaching, many professors choose research.

Anyway, most universities teach future workers, not researchers or educators. Normally, you expect workers teaching workers. Workers raised by professors are like Tarzan raised by gorillas. An innocent problem in a primary school, but the difference in interests increases as education progresses.

How to align the interests of educators and students? By involving the educator in the student’s real passion. That’s what startup accelerators do.

Y Combinator, the most prestigious of accelerators, invests in early-stage startups and puts their founders through a 3-month training program. The 5% stake that Y Combinator acquires for $120K ensures that the mentor’s wellbeing depends on the performance of his students.

Mentorship and apprenticeship are old business practices, of course. Startup accelerators add a social component by bringing many founders to one place. They also escape the research lab hierarchy, when a senior faculty member secures funding and employs graduate students as cheap labor force.

The MIT Media Lab is perhaps the most famous academic lab that operates like a startup accelerator. Professors join the companies founded by their graduates. That’s not a general practice in other universities, in which offering a stake for better mentoring sounds like an insult.

Khan Academy: Engaging students

Engaging students is the second most important task of an educator after engaging himself. This task takes time, so schools and colleges prefer to get rid of the least motivated troublemakers, instead. Many leave college because they see better options. How can educators decrease attrition?

Khan Academy was a one-man project done by a hedge fund analyst in his spare time. The founder taught math on YouTube years before universities started publishing videos of their own classes.

But arguably the best part of Khan Academy appeared later, when students started solving exercises online and getting immediate feedback. Happened before, but Khan Academy polished this technology with data:

In brief, Khan Academy sets the sequence of exercises such that students are not discouraged by frequent failures. It’s part of Khan Academy’s gamification mechanism, which keeps learners motivated throughout K-12.

Stack Exchange: Asking and answering questions

Good educators teach the Socratic way, by asking leading questions. This technique does not scale well in a class with 100+ students. A good alternative is a Q&A website, like StackExchange or Quora.

StackExchange covers many academic subjects up to the graduate level. Its community encourages good questions and punishes for ill-prepared ones. Over time, a motivated person learns how to do preliminary research and ask right questions.

Answering these questions makes more sense than standardized tests or oral exams. Other advantages? Real problems, clear rewards, faster feedback.

Wikipedia: Accumulating knowledge

Wikipedia is fifteen year old, but the education system integrated only one half of it: students copy-paste Wikipedia content into their essays. It should be the other way around! Instead of assigning essays that no one reads, university professors could assign editing Wikipedia articles.

That’s a real contribution. Wikipedia editors check changes and reject the bad ones. It’s easy to track these edits. The Wikimedia Foundation always look for new editors and broader coverage. The content goes straight onto the front page of Google Search.

Despite all the advantages, I saw very few professors who practice this. That’s again about engaging educators, rather than students.

GitHub: Offering creative assignments

GitHub became a Wikipedia for code. Anyone can contribute to a project of interest. The list of open issues suggests possible contributions.

Like Wikipedia and StackExchange, GitHub addresses genuine problems, not synthetic exercises. Software engineers dominate, but any STEM project suits this platform.

Kaggle: Encouraging competition

Though the idea of 3,500 statisticians competing for $50,000 may seem irrational, Kaggle attracted thousands of math-savvy folks to practical problem solving. “Practical” is Kaggle’s key innovation. Competitive problem solving existed before in international olympiads and websites like Hacker Rank. Kaggle made such competitions useful, massive, and scalable.

Some CS departments encourage students to take part in Kaggle competitions. Why here and not on Wikipedia or GitHub? Kaggle challenges look much more like a standardized testing with clear-cut ranking. No need to evaluate whether the student made a useful contribution or just cheated.

Code4Startup: Learning for doing

Learning by doing is an old, popular, and effective technique. But task assignment is a trap. Stupid tasks kill motivation, and the rest dies by itself.

The simplest way to improve motivation is to increase the reward. Startup success stories turned to be a very effective one. More importantly, they are free.

Code4Startup turned this idea into a service. They offer courses showing users how to make a clone of a successful startup. Unlike MOOCs, these courses show how to turn coding and marketing skills into a useful product.

Code School and treehouse take a similar approach.

A honorary mention goes to McDonald’s and Walmart. These companies employ and train the people which top universities would never admit (and other universities get rid of these people after admission). Those who complain about students paying them $50K a year must try to teach a person working for the minimum wage.

Code Review: Giving feedback

Feedback prevents bad habits. In music, you want someone to hear you playing and to fix your techniques before you mastered them. Because one hundred thousand repetitions later you still may do it wrong. And music is complex enough to require a dedicated person sitting next to you and giving tips.

In other areas, technology provides a medium. Kind strangers from Stack Exchange Code Review help developers write better code. Duolingo fixes pronunciation. Show HN let the developer know if his MVP is good enough to keep on going. And elsewhere, video calls connect students with any teachers they can afford.

The calls are the best. Feedback is too complex for technology. Humans have to do it. Skillful people, really. So while lots of apps offer to teach you math or music for $20 a month, they sell new problems, not solutions. And any buck saved on teachers turns into hours of wasted time.

A comment

The services I mentioned have nothing to do with the formal education system. Many of them are not even labeled as educational. But they do what colleges are supposed to do, and do it better.

Three more things. (1) These services never associated themselves with colleges. More importantly, none attempted to reform the formal educational system. That’d be an interesting waste of time, as it was for John Dewey and other reformers. (2) These services scale and depend less and less on the limited supply of really good professors. (3) These services specialize. They don’t teach everything; they make narrow tools to improve specific skills.

Comparing their popularity with that of top universities (the MIT is much more popular outside the US; other terms are insensitive to geography):

1D02B960-8D95-4D2F-8596-EBEDFB5B8C98
Google Trends: The United States

Selected services (the two plots have different vertical scales and only trends are comparable; for more, check the links):

5D4EB9D7-AE45-4E01-A987-9AE9C160FA80
Google Trends

So if education is changing, it it’s changing outside traditional institutions.

Russia Growth Diagnostics (4): Human Capital

< Part 3: Finance

I find it difficult to diagnose human capital in Russia because the literature on topic is scarce and the market differs from what economists know well. Like with the rest of economics, economists know much about human capital in the United States and almost nothing about human capital in other countries.

Still, a couple of points.

First, being careful with interpreting the data. The relations between education and output became an instant classic after Mankiw, Romer, and Weil (1992):

gdc_gdp_humca

While Russia is clearly under the regression line and underperforms for its stock of human capital, such cross-country comparisons have the following limitation. Acemoglu, Gallego, and Robinson (2014) remind us that cross-country regressions overestimate the contribution of human capital, at least, its traditional proxies. If you measure the impact of education on wages in mico, the reported effect is like five times smaller than cross-country analysis implies.

For Russia, the wage premium for each year of formal education varies from 5 to 9%. Denisova and Kartseva (2007) show occupational premia for being an engineer, lawyer, and (drum roll!) economist.

But these premia don’t mean that the Russian economy needs more of these types. Depending on the model chosen by authors, these evidences leave questions. How much does positive selection contribute to the premia? What are the costs of switching to another profession? Is physical relocation of workers expensive? We have to answer these questions to make robust policy recommendations.

The second point to mention, education in Russia is a state-owned enterprise, much more centralized than banks mentioned in the previous post. Heavy centralization creates distortions. When government sets policies incorrectly, this lever just moves the entire education system in a wrong direction. Such a system either needs more discretion at low levels or evidence-based public policies. Better to have both, but government doesn’t tolerate the discretion. Then you need evidences for policies.

Few are available even for changes in aggregate variables. Like, how would GDP respond to one more year of education for each citizen? In terms of the HRV framework, GDP may be insensitive to changes in raw human capital. Also accounting is tricky when you compare the gains from more education against (1) full costs of the respective programs, including foregone earnings, (2) benefits of investing the same amount in physical capital or technology.

Careful policymaking could equalize returns to investments, which are now distorted. Government discourages investments in human capital by taxing labor more than financial capital. A person prefers investing in stocks and real estate to education because his wage-related tax rates are about four times higher than taxes on income from financial assets. This is sort of a good example, because we can understand the magnitude of the distortion. Some other distortions have no clear estimates.

Where to move from here? I’d say, randomized evaluations for effective policymaking, plus James Heckman and Stefanie Stantcheva for rigorous thinking about human capital. For any mid-income country, not just Russia.

Next Post

It was supposed to be about infrastructure, but infrastructure shares many conceptual problems with human capital. I’m not even sure that with all cost overruns infrastructure investments are more predictable than money put in human capital. Perhaps, here benevolent stakeholders should do very local analysis, when specific infrastructure projects are compared against each other.

Attention to opportunity costs. Governments like to spend on construction because it creates opportunities for corruption. First, costs have no direct market pricing. Second, stealing inputs in a capital-intensive industry is easier, since in labor-intensive industries workers monitor their managers. That makes wages an unlikely source for corrupted officials.

So instead, there will be a post on government failures.

Government Resistance to Human Capital

James Heckman has a great paper called “Policies to foster human capital“. Apart from excellent integration of economics into government policies, this paper relentlessly reminds the reader about the lifetime aspects of investments in human capital:

Heckman and Carneiro - 2003 - Human Capital Policy
Heckman and Carneiro – 2003 – Human Capital Policy

Heckman refers to the evidences that late investments do not pay off. Late learning costs money, including foregone earnings, but it only modestly increases wage rates and generates less human capital, since investments are made closer to retirement.

Heckman also criticizes the excessive attention to formal education, which is all about a few narrow cognitive abilities and in any case just one of many ways to acquire skills.

This paper was published in 2000. To count how many of its insights made it into government policies, I looked into the 2013 Economic Report of the President. This report includes a chapter on human capital.

The chapter discusses three things. First, labor inputs of women and immigrants. Second, rising debt and enrollment bias in college education. Third, educational opportunities for adults. So, formal education and adult learning?

Why does this influential publication promote the exact policies that Heckman criticized fifteen years ago? Alan Krueger — who edited the 2013 report — is a brilliant labor economist himself. The entire Council of Economic Advisors is well staffed. So it’s not about competence.

The problem seems to be in the complexity of the policies that Heckman proposes. When economists remind the Congress about student debt, they have better chances of being heard than economists who suggest targeted programs in preschool education. Student debt is something that even US presidents had. Preschool education? TL;DR.

But rephrasing Neil deGrasse Tyson, science is true whether you read it or not. The complex stuff still explains well why policies fail when consensus is reached.

For the United States, this complex stuff is about fine-tuning the system that works well. For the middle-income countries, on the other hand, ignorance is costlier, not least because these countries tend to aggravate mistakes with more government intervention. Policy makers in these countries often frame human capital accumulation as another industrialization. Like, if we could raise savings-investments to 50% of GDP, we can accumulate human capital in the same way. Obviously, it doesn’t work this way; and for BRICS, it’s an important challenge.

As for the least developed countries, the major innovators out there are international organizations, which recently got an open letter from Chris Blattman asking to “stop hurting” poor people with skill training programs. For the same reason: the programs don’t work as intended; even if being accumulated, this human capital solves no important problem.

“Stop hurting” is a good suggestion because it encourages policy makers to do less, which is the idea they like. The second part is more difficult: stop hurting and accept better policies. Some of the better policies look unconventional but remain as simple as training programs. Their main disadvantage is that they are not invented here, that is, in government. And before anything happens, someone has to market these policies as if they were government’s genuine invention.

Free Cheese, not in the Mousetrap

OECD has a nice cost-benefit analysis of returns to education. First, what a high school gives to students:

Source
Source

Okay, huge net benefits for a degree. Even more interesting is the “unemployment effect.” Here lies a monetary value of higher employment security. The degree holder spends less time unemployed due to job security during crises and later retirement. This component is especially high in Slovak and Czech Republic. These countries have one of the lowest wealth inequality in the world, but it looks like the effect of relatively low incomes in the top quantiles. Their labor markets need more highly educated employees, as the market quickly absorbs high skilled candidates and low-skilled workers remain jobless (unemployment rates of 14 and 7% for Slovakia and Czech Republic, respectively). You can compare it to Korea, which has a more balanced labor market: a degree holder earns more but not because she gets jobs faster.

Net lifetime gains from having Bachelor’s, Master’s, or PhD:

screenshot

Eastern Europe could do a lot better given its middle-income status. Slovenia and Czech Republic would greatly benefit from more educated workers. A Hungarian with a tertiary degree creates more benefits for others than for herself, which makes a case for government support. It’s not necessarily support for education spending in this particular case. Free labor migration within the EU creates difficulties for public spending on education. Political support for these subsidies is low because students who get free education may migrate to high-income Germany and United Kingdom. High returns to tertiary education in Eastern Europe discourage this move, but they cannot fully offset the income gap between the West and the East.

So, it’s a case of the European Union without unity. Countries still have independent budgets (with exception of “stability and growth” rules), collect and spend their public revenues, but have to distort policies in response to other members stealing employment, demand, capital, or workforce. So, hypothetically, net beneficiaries from the brain drain should compensate losers for free public education.

But the point is, countries with fewer emigrants keep the returns to education and should invest in it more. Both by increasing public spending and by facilitating student loans. It’s not rocket science, just more care about people’s future.

Deskilling Attacks

The WSJ frightens its readers with computers making humans dumb. About automation:

This philosophy traps people in a vicious cycle of de-skilling. By isolating them from hard work, it dulls their skills and increases the odds that they will make mistakes. When those mistakes happen, designers respond by seeking to further restrict people’s responsibilities—spurring a new round of de-skilling.

By “automation,” the author means CAD, decision support, and plane navigation systems. Thanks to them, doctors and pilots become dumber.

Maybe that article was written on paper and with ink, but most people would prefer keyboard and text processor. Why? Because keyboard and software reduce routine work, just like the other technologies mentioned there. Computers can’t steal creativity they don’t have.

Deskilling is a hypothesis from classical economics. Adam Smith, The Wealth of Nations, c. 1776:

The man whose whole life is spent in performing a few simple operations, of which the effects too are, perhaps, always the same, or very nearly the same, has no occasion to exert his understanding, or to exercise his invention in finding out expedients for removing difficulties which never occur. He naturally loses, therefore, the habit of such exertion, and generally becomes as stupid and ignorant as it is possible for a human creature to become.

Compare to the WSJ piece:

As software improves, the people using it become less likely to sharpen their own know-how. Applications that offer lots of prompts and tips are often to blame; simpler, less solicitous programs push people harder to think, act and learn.

(The master of the deskilling genre was, of course, Karl Marx. Is anyone surprised that Rupert Murdoch publishes Marxists?)

But the 19th century technologies replaced artisans because machines plus humans did better than humans alone. Engineers and workers got the jobs that artisans had lost, with a huge surplus in productivity. Workers have never been deskilled because most of them came from agriculture, which is dull, hard labor all over the day. You had to know tricks in agriculture, but these tricks didn’t make a skilled farmer by definition.

The present. No deskilling on horizon. Jobs demand more skills, actually. Degree premia in the US are increasing, and they’re positive in other countries. Wherever you can learn more skills, they pay off. Technologies replace middle-skill jobs by reducing routine operations, but it has implications opposite to those the WSJ implies. We need more, not less, education as workers look for more skilled jobs.

When ROI Hits the Roof

rct_nyc_schools

The Coalition for Evidence-Based Policy has a nice compilation of low-cost program evaluations. Example #2 tells us about a $75 million education program that improved nothing. The cost of finding this out was $50K. The simple math says that returns on money invested in evaluation reached 150,000%. It kinda outperforms S&P500.

What’s the trick? First, as mentioned before, maybe the same program completes something else. The program had aimed at improving student results and attendance, and it didn’t improve them. But the teachers got $3K more each and bought themselves useful things. Nothing wrong with that, but we need other ideas to improve education.

Second, so-called unconditional money transfers rarely motivate better performance, though it may seem counterintuitive. Not only in education. Public services just happened to be in full view of everybody. Then ROI in evaluation depends on how much the government or business puts into unchecked programs. This time it was $75 million, next time it’s $750 million. Big policies promise big returns, either due to better selection or faster rejection.

Third, such opportunities exist because big organizations evaluate execution, not impact. Execution is easier to monitor, so public corporations have to have independent auditors who ensure that employees don’t steal. In contrast, efficiency audit requires management’s genuine interest in rigorous evaluation, but there’s no incentives for that. After all, stealing is everywhere a crime, while incompetence is not (despite incompetence being more wasteful).

With that said, ROI of 150,000% is a fact. If you spend on a policy doing X and the policy does nothing to X, you can just leave $75M on the table. Without that $50K evaluation, you’d lose them.

Not Reinventing Education

startups_ts_by_market

Education seems like an attractive market to entrepreneurs. It’s huge. It has low competition. Its core technology comes straight from the Stone Age. Why won’t you create something cool here?

Education is different. Not only because of ideology, but because of the very Stone-Age technology that looks replaceable. Universities didn’t change much since they had come out of monasteries a thousand years ago. And they are highly competitive despite that. You won’t find another industry in which a company remains on top for nine centuries.

Most startups created thus far compete with textbooks, not education. New textbooks now work in a browser and interact with the reader. Online courses offer lectures and materials from the best teachers. But it’s not the university experience, as the teachers themselves agree. Books remain books even online. Best books were in libraries for centuries. It never withheld the learners.

Education is cooperation. Cooperation happens in groups, and groups are limited by definition. Harvard might have a million students (after all, Walmart has 245 million customers weekly), but then it would be an ordinary place. Until groups are small and carefully selected, its members may learn from each other and the faculty. In other words, education is all about one limited resource: people’s attention. IT can’t scale up people’s attention yet. It does routine stuff and does it well, like crawling petabytes of data daily. For this, it’s more likely to create the next Google worth $300 bn. than to create a $10 bn. business in education.

PS: Investors agree:

funding_ts_by_market

Learning obedience

Back in the 90s, growth economists explained income differences among countries with human capital (Mankiw, Romer, and Weil, “A Contribution to the Empirics of Economic Growth”). Formal education was a typical proxy, with either fraction of educated population or years of education entering regressions.

The human capital story had to integrate the institutional component since then, but formal education remained correlated with national output. One channel or another, nations have to have good education to develop.

Universal formal education came from governments in the 19th century. Peoples actually thought it was a bad idea. They cared more about what is now called social security, unemployment, and inequality.

“Universal,” however, typically meant “compulsory.” The reason is suggested in the first graph (taken from Alesina and Reich, “Nation Building”). Each time the French rioted, European rulers reformed education in their territories. The education system helped keeping children busy with the “right” thoughts and reduced the risk of insurrections. It didn’t help European monarchs for long, but the habit remained. Say, “education” is still the most popular issue in the US Congress bills.

While many are concerned, no one started asking children what they would like to learn. Sparse surveys (like the one by NYC or Gates Foundation) ask typical satisfaction questions—the ones that ensure you don’t protest much in the classroom. Authors don’t question the curriculum. Questions about biased history classes would be outer space, but no survey even asks “Do you want to learn biology?”

So, education is really sacred, unlike any other services on the market. You are supposed to spend ten plus years on being good at things you don’t care about. It’s like a 20-year mortgage, but here they also take your time.

Any hope? If 100 years ago the factory-like training in the school produced suitable workforce for real factories, now employers expect more flexibility from hires. Mainstream technology becomes skill-biased and requires from employees to be “quick learners.” Employers in competitive industries have to respond. And maybe this new demand for quick learning will change the education system more than democracy did in the 20th century.