An important open question in visual neuroscience is how visual information is represented in cortex, and ultimately used to form perceptions and shape behaviors. To address this question, we developed the Allen Brain Observatory, a 2-photon calcium imaging platform that enables us to collect systematic data from the awake mouse visual cortex. Using a standard set of visual stimuli that include gratings, noise, natural images, and natural movies, we created an open dataset by imaging physiological activity from neurons in 6 cortical areas, 4 cortical layers, and 14 different transgenic cell types, including over 63,000 neurons in total. We found that visual responses throughout the mouse cortex are highly variable. Using the joint reliabilities of responses to multiple stimuli, we classify neurons into functional classes and validate this classification with models of visual responses. Only 10% of neurons in the mouse visual cortex show reliable responses to all of the stimuli used, and are reasonably well predicted by linear-nonlinear models. The remaining neurons fall into classes characterized by responses to specific subsets of the stimuli and the neurons in the largest class do not reliably responsive to any of the stimuli. These classes reveal a functional organization within the mouse visual cortex wherein putative dorsal areas show specialization for visual motion signals. Further, we found that the greatest diversity among cell types was among the inhibitory interneurons, namely the VIP and SST, suggesting that they have distinct functions in the cortical circuit. We built upon this initial dataset, collecting additional data that examined the contrast tuning of neurons in a subset of areas and cell types. While SST interneurons respond strongly to high contrast stimuli, the VIP interneurons only respond to low contrast, specifically for front-to-back motion. Using a Stabilized Supralinear Network model, we show that the VIP interneurons function to enhance these weak, but behaviorally relevant, stimuli in the superficial layers of cortex, while the SST interneurons provide network stability.