Yes, EU has the 4 Freedoms as part of its principles, but that do not mean everyone can just travel to EU and get in without any issues. Far from it, for most people in the world EU is one of the hardest places to get into.
The above mentioned 4 Freedoms only apply to members of EU. They are the reason to join EU in the first place. EU is closed club with huge benefits to members; free travel is one of them. It's the same how you do not need passport to move between states of the US. You still need passport and visa to get into US and so is to get into EU. EU avoids country like terminology because certain people still hold out towards their national identity and not willing to embrace union completely. However, especially when it comes to economy, EU operates like a single federal country.
EU members also get other benefits too, for example they trade within EU without any border checks tariffs or paperwork. (To remove border checks you need to join Schengen Agreement after joining the EU.) In contrast nonmembers are burdened with hefty tariffs, loads of bureaucracy, complex requirements for Schengen visa to allow travel to EU and more. EU is an exclusive club with lots of members-only privileges.
All that gives clear insensitive to join as those who stay out miss out by a lot. That is why Brexit was such a stupid idea. From no paperwork and tariffs to all trade to EU they went towards these tariffs, border checks and paperwork. I destroyed the entire industries that used to export into EU but after Brexit could no longer do so due to paperwork and tariffs.
Courtiers that aspire to join the EU, like Ukraine or Moldova get a trial version of these benefits: removal or some tariffs as well as free travel for 3 month every half year. You can come and see for yourself how awesome it is but then have to go back to finish the accession process. OECD countries like the US, Canada or Japan get above mentioned 3-month travel as well. That is reciprocal as EU citizens also get visa free travel to these countries.
For most of the rest of the world however, entry to EU is only through complex Schengen Visa process where you have to bring a lot of documents to convince EU officials that you should be allowed to enter. If you have criminal record, did not bought return ticket or your bank account does not have sufficient money to spend in EU, you will not get the visa. It is business and they only want those who have enough money to spend on their Cote d'Azur vacay.
There are also refugees, but they are governed by UN rules and get processed by institutions to determine if they qualify for refugee status. Smugglers often misuse these UN rules as they are the more permissive and laxer compared to national polices. However, UN rules are only guidelines rather than laws and nations can choose to ignore them.
EU admission policy is rather strict. Probably only Australia has even tougher visa system. It is far from come anyone. Do not misunderstand.
