1 Introduction -- 2 Quantum Theory of Gravitational-Wave Detectors -- 3 Modifying Input Optics: Double Squeezed-input -- 4 Modifying Test-Mass Dynamics: Double Optical Spring -- 5 Measuring a Conserved Quantity: Variational Quadrature Readout -- 6 MQM with Three-Mode Optomechanical Interactions -- 7 Achieving the Ground State and Enhancing Optomechanical Entanglement -- 8 Universal Entanglement Between an Oscillator and Continuous Fields -- 9 Nonlinear Optomechanical System for Probing Mechanical Energy Quantization -- 10 State Preparation: Non-Gaussian Quantum State -- 11 Probing Macroscopic Quantum States -- 12 Conclusions and Future Work -- 13 List of Publications -- Bibliography.

Recent state-of-the-art technologies in fabricating low-loss optical and mechanical components have significantly motivated the study of quantum-limited measurements with optomechanical devices. Such research is the main subject of this thesis. In the first part, the author considers various approaches for surpassing the standard quantum limit for force measurements. In the second part, the author proposes different experimental protocols for using optomechanical interactions to explore quantum behaviors of macroscopic mechanical objects. Even though this thesis mostly focuses on large-scale laser interferometer gravitational-wave detectors and related experiments, the general approaches apply equally well for studying small-scale optomechanical devices. The author is the winner of the 2010 Thesis prize awarded by the Gravitational Wave International Committee.