Not in theory. It's not the television that turns brains into swiss cheese, it's the television programs. Of course kids would be better off reading a book than watching reruns of some reality TV show, but there are plenty of good stations. For example, there was a study to find out how much television really affected kids. The great majority of evidence pointed towards the "swiss cheese brains" end of the spectrum, but kids who watched certain educational shows (e.g. Sesame Street) actually developed positively.
So basically I hate it when people say things like "don't watch television, it'll turn your brain into mush and has no value". While most television and too much television will probably do that, not all of the TV programs are bad. The problem with TV is that it takes no effort to watch it, so people are content to watch any old crap. When I'm feeling depressed, I turn on the TV and tune out, so that my brain is occupied and amused wihout much effort. But there are educational shows, such as the history channel, or shows that demand attention and create excitement, such as thriller/mystery/dramas like 24 or Lost.
As long as you actually have some care in choosing what you watch, instead of just settling back on the couch and channel surfing lazily when you don't feel like thinking, there's no reason to avoid the boob tube like the plague.