40 is not late, unless you're hoping to make it to the Indy 500 Depending on what class you pick, you might be one of the young ones!

Formula Vees are quirky, old and cheap Lots of them to race with and it's a pretty fun group.

I recommend renting a car (or two) before you buy, that way you can try a couple of classes, get a feel for it before committing to a class!

Check http://unionracing.ca/ for FV rentals, and the forum here for closed wheel rentals, there are a few options out there!

Here is a post for closed wheel that might be helpful
http://www.sccbc.net/forum/YaBB.pl?num=1347952624

Never2late2race
Very helpful link (and I say that without bias of course!).
It does cover just about all the questions.

Definitely rent before you commit to buying, Renting a cars until you finish the novice program has many advantages, one of which is that you will not be preoccupied with getting the car race ready, most novices have enough on their plate already. The other advantage is many that rent cars can also coach your driving skills, that alone is probably worth the price of renting a car. By the time you are through the novice program you will probably have observed enough to get a feel for what class you ultimately want to run in. Or you may find that racing with the Vintage Racing Club is closer to your liking, You mentioned liking quirky older cars, you can't get much quirkier than vintage racers. Attend a few meetings as well, the politics at the meeting may be a bit overwhelming, but it is a good opportunity to get to know members. Several of us also get together for coffee or beer after the meetings, this is a prime opportunity to get advice (we can all talk racing and cars until the end of time, so always welcome new blood).

Thanks for all the great advice. I am considering to rent a car to go around the track a few times.

That information is sanctioning body specific, so for CACC you need to go to: http://www.caccautosport.org/rules/
and for ICSCC you need to look at: http://www.icscc.com/references.php

We only run one ICSCC weekend at Mission now, and the rest are CACC, so that's probably a better place to start unless you want to run a lot of race south of the border.