Thus, when I switched from finance to healthcare in 2001, my title was Statistician. There just wasn’t any job called Data Analyst or Data Scientist at the time. It was all basic descriptive statistics and as far as predictive modelling, it was Ordinary Least Squares (but you might know this today as “Machine Learning” LOL).
Marmi Maramot Le
I was told there was no future in being a Statistician
That was 1979. I had been a consultant in computational statistics for 7 years. It was a great job as I worked at a university in the role of “senior teaching fellow” assisting staff and research students with their statistical design.
Once I’d sorted out the day’s challenges I was free to do whatever else I wanted. That was “academic life” and I was tenured. Mostly, I coded up interesting solutions and taught myself new programming languages. I volunteered to regularly man the Computer Centre help desk, and I taught a few service courses in statistics as the students were fun.
Before that job, I’d worked for three years as a scientific programmer in the “Computational Laboratory” of another university — a part of the Department of Statistics. It was funded by grants to one of the professors, and once I had completed his work he didn’t mind what else I did. He allowed me to use his name to gain unlimited access to the university computer centre — everything was batch in those days. I completed my degree in mathematical statistics part-time, and applied it daily.
University life in a different era
I also completed — part-time — an MSc in Computing Science by thesis. My supervisor had the room next to me and was a nice human being. I asked him once why he’d never made professor, as he seemed as smart as the others I dealt with regularly.
His answer did and didn’t surprise me. It kind of helped me suddenly understand the natural milieu of the university — walking the corridors and wondering what was going on behind the closed doors.
It turned out — not much. My supervisor explained to me that he had “retired” on the day that he was appointed as an Associate Professor. “But you worked with all these famous people at the University of Manchester, don’t you want to do more”, I asked enthusiastically.
Out here, I just have to worry about my fencing, was his reply.
He rarely came in to the university, and then only to roll out his 10-year old lecture notes. I drove out to his farm to do my thesis “reviews”. His one substantial comment was on the final draft when he told me that my conclusion was rubbish, and to fix it or get rid of it. I rewrote it.
They’ll never know — zombie hypotheses roaming freely
The most interesting part of the job — statistically speaking — was the number of PhD students and researchers who came to me to assist them to validate a pre-defined outcome — their hypothesis.
Inevitably, after we worked through the “experimental design” (in quotes because most were not designed but were just carried out) I hinted that things might be a bit messy.
I’d break the bad news — their experimental design was unlikely to deliver any reliable insight into their hypothesis. The reasons were the usual ones — the design violated the underlying assumptions upon which the proposed statistical tests are based, and the variables were all intertwined and not independent.
This was always a freaky moment.
The PhD students were already in the fourth year of their three years of full-time work, or in the 7th or 8th year of their part-time research. They had to deliver results — there was no time nor resources to design a proper experiment and redo the work.
Sometimes we’d go non-parametric
At this point, I’d ask them if they would like a second opinion. It so happened that at this university one of the Professors of Computing Science had previously been a Professor of Statistics — a guy named John Burr. He was a likeable Disney-style professor, and I had developed an easy-going relationship with him.
There was also Vic Bofinger, a die-hard true academic statistician whose wife was also an academic — in the Mathematics Department. With Vic, you didn’t waste time, and he was tremendously helpful.
If my “client” was resisting my advice I would take them to Vic, without briefing him beforehand on my thoughts. He would tell them very directly why they were between a rock and a hard place, and then give some brusk statistical suggestions.
They almost never understood his suggestions, and so it was back to me to explain the options.
If my “client” was thoughtful and wondering what might be salvaged from their years of work, I would go see Professor Burr and discuss the case. He was an expert in non-parametric statistics and what we now called unsupervised machine learning. Then, we simply called it cluster analysis.
Sometimes, I’d have to write large slabs of new code and test it running for 24-hours-plus on the university computer — SPSS wasn’t up to the task. We’d end up with results that got the job done, and everyone was happy.
Close the door — I’ll tell you a secret
How about the ones who resisted their data? What was my prognosis for them?
Simple. I’d add a bit of drama. “Would you mind just closing the door for a moment”.
Guess what, just go ahead and apply the analysis you were proposing. But… how will that work, they’d respond. Look at it this way, do you think that any of your supervisors would know the difference?
Their uncertainty about this proposition was always overtaken by their fear of having to go back and do a whole lot of work again. Besides, I told them, what do you think has happened with all the other PhD students in your faculty over the last 5 years?
No-one ever missed out on their award — at least, not for anything related to the statistical analysis.
The jobs are all taken
As for jobs for statisticians — all taken.
Hospitals had one, government research institutions had many and government agencies had a few — all lifers. Companies were starting to employ economists but not statisticians.
There’s no future in it — was what I read and observed.
In 1980 I jumped at the opportunity to join Australia’s largest public company and progressed through Audit, IT, corporate planning, marketing, strategic planning and had a great time.
I rarely used my knowledge of statistics again — except today I claim to be a “citizen data scientist”. A little statistics helps, but of course, being a data scientist today means a lot more than being a statistician.