Today, I want to discuss some of the common mistakes that programmers make when starting to learn machine learning.
But first, let me speak about why software engineers should start looking into ML.
First, let’s see why programmers should start ML?
Today, from what I’ve seen, people coming with a strong software engineering background, good programmers are highly sought out for cream roles in large product companies and startups.
Because, as ML sees increased adoption with companies and growing [use cases], ML engineers are not only expected to build ML models, but to productize it as a software solution.
Yes, streamlit, dash, rshiny etc exist to make quick dashboards that use ML in the backend.
But for more serious / heavy weight applications, and may be development of mobile apps for your ML solutions (which is getting increasingly common these days), proper software developers with knowledge of ML are preferred.
Alright Let’s now go over the common mistakes programmers make.
1. Placing machine learning on a pedestal
You don’t want to start off with a wrong mindset. If you think something is hard/tough, chances are, it is going to be.
Yes, machine learning requires your commitment and effort. But it is not something that is out-of-the world difficult.
Here is what usually happens.
Programmers initially get excited by the hype, applications and salary of ML and decide to start. Then you try learning some concept in ML, but nothing seems to make sense. As you read books and learn more, it gets more and more complex.
Whereas, as programmers / developers, things were so simple. You loved building things and simply make it work.
But it ML, you need to understand concepts.
That can be intimidating initially. Yes.
The thing is, when you find some idea or concept difficult to understand, the chances are it was not explained well enough in the first place.
As Albert Einstein famously said:
“If you can’t explain it simply, you don’t understand it well enough.”
So, if you are not able to understand some concept completely, seek out a better resource or teacher who can put it simple english for you.
In several cases, they seem to explain everything, but you still really don’t get it. So, ask all the questions til you are completely satisfied yourself.
A good test of the level of your understanding of any concept is this:
(i) You are now able to explain it simply to another person (or interviewer).
(ii) You won’t have to ‘remember’ anything. Because once you simply understand, that’s it.
2. Not focusing enough on basic numpy and pandas
Python is the primary language for machine learning and deep learning apps (followed by R programming language).
If you primarily use Python for Data science, then having a good command of the two main data wrangling packages – NumPy and Pandas is important.
Because, for you to be hands-on with data science / ML, you are doing to be working with data. You need to be able to wrangle – transform, reshape and do anything you want with the data.
Knowing pandas and numpy really well is important for this.
I’d highly recommend reading the book ‘Python for Data Analysis’ – by the author of Pandas Wes McKinney.
Personally, I’ve read it twice, it’s a very good book.
If you like videos better, enroll to the NumPy for Data Science course and the Pandas for Data Science Course (paid courses) from the ML+ Complete ML mastery courses set. Covers advanced packages and techniques.
3. Fascination with minimizing an error metric
A common mistake early career folks make is Machine learning is to do everything possible to lower a specific error metric.
When you overdo this, it is very much possible that it may lead to overfitting, that is the model does not perform as good on an unseen dataset, or worse when scored on live real world data.
Besides, not all problems require a small error rate. Make sure to keep in context what is and acceptable error rate required for success and productionize it. It’s important to set sight on the larger objectives as well for the success of the project.
4. Relying on Deep learning for everything
While deep learning is a great tool, use it appropriately. Building a deep neural network with several layers and leaving it to the mercy of the machines to give a solution can be concerning considering the long term health of the project, resource costs, large training times etc.
Besides, make sure the models are making the right predictions for right reasons (features). Else the model is still junk even if it makes right predictions.
5. Falling in the data wrangling trap
When your data are scattered across different sources, or is highly unstructured, contains missing, outliers etc. You need to organize so that you spend the minimum possible time and mind energy on wrangling data.
Several data science folks complain 70-80% of the time is wasted in data munging etc. This can be a problem and cause inefficiencies.
Create standard data views that you can reuse. Consider setting up feature stores if needed.
You ideally want to spend your efforts on building the ML and solving the problem.
6. Jumping around algorithms too much
When you are working in ML project, chances are you are not going to hit jackpot results in your first shot.
If you did get unbelievably accurate results, chances are there could be data leakage or some sort of over fitting could happen. Just make sure everything is Ok, before announcing your proud results.
Now, when some algorithm does not give good results, consider inspecting the model results more closely before jumping off to another algorithm.
Often it’s not always about the model.
Some questions you might want to consider before jumping off to next algo:
- Is the data you have sufficient?
- Is there a chance the data is inconsistent? (duplicate entries, changed historic values etc)
- Does not have sufficient explanatory power to model / explain the Y?
- Can I create new meaningful features that could help with predictions?
- Could data transformations (log, sq.root, box cox) help?
- Could scaling help?
- Would a encoding method method help?
- Examine what predictions went wrong? and why? Is there anything you could do about it without changing the algorithm?
- Can you source new data that are publicly available? consider google trends, macro economic factors, weather, holidays data etc.
Yes, think through the various aspects that could help improve the quality of the data, thereby the model, before jumping from one algorithm to next.
7. Ignoring Exploratory data analysis
It can be a terrible mistake to ignore exploratory data analysis and jump straight to build models.
Often you can valuable insights that itself justifies the project / initiative you are working on. Your stakeholders will thank you for it, helps build trust in your abilities.
8. Focusing too much or too little on the math
This is quite self explanatory. Math behind machine learning is important. But don’t get stuck into one idea that it stops your learning progress.
At the same time, avoid completely skipping the math and just relying on automl / python packages.
If you want to become awesome in Data Science, join the Complete ML Mastery Path and take the courses in the learning paths.
You will have everything you need for a successful career in Data Science. The industry project courses gives you the expertise, comes with the fully solved code notebooks and datasets for hands-on practice.