Security and privacy. Well, this post could have ended here, but since I'm a bit talkative (just a bit), I'll try to argument.
First of all, Internet of Thing
[...] is the network of physical objects or "things" embedded with electronics, software, sensors and connectivity to enable it to achieve greater value and service by exchanging data with the manufacturer, operator and/or other connected devices.
This short description should be enough to make you understand why privacy is a serious issue. If you don't feel the risk, just think that smart TVs send everything you say to a remote data center - and some of them don't even encrypt it.
The security risk needs some more explanations. Since embedded devices are usually not so powerful (mainly for a matter of cost), it's not possible to apply all the best practices needed to ensure a top level of security.
In addition, as new exploits and security flaws come out, those devices should be updated, but the producer may decide to not provide patches for devices considered too old. This is what happens today with many smartphone producers that don't distribute new Android versions for relatively old devices.
Don't get me wrong, I'm sure that IoT can improve our houses and be wealthy for the environment. But, this should not make us forget the other aspects.
Suggested read: An Internet of Things that do what they’re told