Results of HN poll: Half think bootcamp grads as good as fresh CS majors

We wanted to get a sense of how well code bootcamps prepare their graduates for work. So we asked Hacker News readers who had worked with a graduate to evaluate them.(See methodology)

Most importantly we asked, “Would you rather work with the graduate, or the median fresh-out-of-college Computer Science-major you’ve worked with?”

Average: 3.6 (95% confidence interval of 3.0 to 4.3) % of respondents rating 4 or higher: 57% (95% confidence interval of 43% to 71%) Sample size of 47

Average: 3.6 (95% confidence interval of 3.0 to 4.3)
% of respondents rating 4 or higher: 57% (95% confidence interval of 43% to 71%)
Sample size of 47

 While on average engineers leaned towards the median college CS graduate, about half said they would just as soon work with that code bootcamp grad as the college grad. That’s pretty amazing if you consider that bootcamps tend to cost $10k-$15k and take a few months, quite a bit less than a 4-year college degree.

This isn’t quite apples-to-apples. Some of these folks had graduated months ago, and there was a trend towards higher ratings after more months in the workforce. (There are other caveats, too, if you’re of an analytical bent.)

Correlation is (barely) statistically significant. Click through for the stats, and to play with the dataset generally.

Correlation is (barely) statistically significant. Click through for the stats, and to play with the dataset generally.

And it’s not all great news for bootcamps. There’s quite a bit of variation in respondents’ estimation of the technical abilities of graduates.

Click the image to get stats on average, confidence interval, etc., and to play with the rest of the dataset.

Click the image to get stats on average, confidence interval, etc., and to play with the rest of the dataset.

So the takeaway shouldn’t be “Skip college, do a bootcamp.” It’s probably more like “While there is significant variation in graduate quality, code bootcamps are reasonably effective in preparing their graduates for work.”

See the findings for typical size of company of graduates, pleasantness of working with grads (technical ability aside), and play with the data.
Or download an Excel file of results.
See methodology and measures taken against fraud

 

Open-ended comments

We also left a space for open-ended comments. You should also check out the conversation that ensued on the survey post itself, which was full of useful and interesting advice. Here’s the survey comments, preceded by the associated score on the 1 to 7 scale of preference towards graduates:

7) Had such a good experience with the bootcamp graduate we hired, that we hired 3 more from the very next cohort.

6) Overall he learned quickly, was incredibly effective, and managed to go from “still learning” to “getting shit done” in a short period of time. We also ended up hiring another bootcamp grad after this and it worked out very well too.

1) They are good at selling themselves (as are the bootcamps), but those without engineering backgrounds at all are not good coders.

6) We were pleasantly surprised by our bootcamp graduate. He is super smart and motivated, and we all have a great time working with him.

1) Lacked understanding of basics and was significantly tripped up by very minor errors and unable to “just figure it out.”

5) Experienced professional in an analytic but non-computer related field before doing bootcamp.

1) It’s essential that you have some background in order to get something useful out of a bootcamp. They are great for getting someone up to speed with a particular technology, but without any grounding or context that specific knowledge is of limited value.

4) In addition to his bootcamp I know that my team member also did a lot of hacking outside of school and is very good at educating himself on how to do things before raising questions. It takes time, dedication and hard work to excel at anything you do. I do know some grads out of bootcamps have difficulty finding jobs but this is generally due to the demand of qualified engineers in Vancouver. They generally have better luck at larger companies.

6) My experience with bootcamp grads is the experience greatly changes based on the type of person. A more motivated, interested-in-everything type of person generally comes out a good programmer and an enjoyable person to work with. Someone with an average to below average motivation and work ethic may struggle more.

5) Don’t stop at rails.

3) Potential attendees should be open to learning new technologies, even if they were hired primarily for Rails, JS, etc. Instead of sticking to what you know, taking initiative to learn something new is an extremely valuable skill.

2) Make sure you talk with a couple graduates of the program before you sign up. Their experience may not be as pleasant as the quotes on the website imply. Make sure the school takes care of all their students, no matter what skill level they come in with.

 

Methodology

We submitted this prompt to HN, with a link to this survey. And, to reiterate, one of us does in fact have a close friend making this decision, and she was very curious to hear the results, we weren’t lying about our motivations.

For whatever reason we failed to stop some people from taking the survey multiple times. This was probably a blessing in disguise, as it allowed us to later go back through IP addresses and clean out multiple responses (i.e., they didn’t get blocked then switch to faking IPs, which would have been hard to trace)

We were only on the front page of HN for a few hours because the discussion triggered the slightly trigger-happy HN flame war detector (there were too many comments). Sadly, this meant we got too few responses for many of our other questions to get good data; we were initially pretty darn interested in which bootcamp would fare best, which we basically can’t answer with such limited data.

We did a lot to prevent fraudulent entries:

  • We excluded results from duplicate IPs.
  • One IP from App Academy was responsible for a quite a few fraudulent completes. The rest of the App Academy responses look fine, but are slightly above average. We excluded the clearly fraudulent ones, and if you exclude them, the total number of respondents giving a 4 or higher is still more than 50%
  • We excluded results that came in after our post left the front page of HN. 
  • We analyzed a variation where we weighted the results so that multiple evaluations for one bootcamp only counted as one total. The results for this were fairly similar, about 50% of respondents rating a 4 or higher (very small sample size of 17).
  • There was no upward trend over time in survey responses. You’d expect that if people were gaming it, someone would email other people to respond, and since email is asynchronous, there’d be a delay and then more positive answers would come in later, which was not the case.

Is it possible that there was still fraud? Sure. But given that many companies do keep hiring from bootcamps, and given the tenor of the HN discussion, the simpler possibility is that these findings are in fact legitimate. Furthermore, 27 engineers rated grads a 4 or higher; even if you removed every respondent who gave a 7, it’d still be a 20 to 20 tie.

Sorry, comments are closed for this post.