This paper investigates the feasibility of practical backscatter communication using visible light for battery-free IoT applications. Based on the idea of modulating the light retroreflection with a commercial LCD shutter, we effectively synthesize these off-the-shelf optical components into a sub-mW low power visible light passive transmitter along with a retroreflecting uplink design dedicated for power constrained mobile/IoT devices. On top of that, we design, implement and evaluate PassiveVLC, a novel visible light backscatter communication system. PassiveVLC system enables a battery-free tag device to perform passive communication with the illuminating LEDs over the same light carrier and thus offers several favorable features including battery-free, sniff-proof, and biologically friendly for human-centric use cases. Experimental results from our prototyped system show that PassiveVLC is flexible with tag orientation, robust to ambient lighting conditions, and can achieve up to 1 kbps uplink speed. Link budget analysis and two proof-of-concept applications are developed to demonstrate PassiveVLC's efficacy and practicality.