Ipilimumab, Nivolumab, and Radiation Therapy in Treating Patients With HPV Positive Advanced Oropharyngeal Squamous Cell Carcinoma

Inclusion Criteria: * Histologically or cytologically newly confirmed diagnosis of squamous cell carcinoma (including the histological variants of papillary or basaloid) of the oropharynx (tonsil, base of tongue, soft palate, or oropharyngeal walls) * Clinical American Joint Committee on Cancer (AJCC) 7th edition stage T1N2a-N2CM0, T2N1-N2CM0, T3N0-N2CM0, equivalent to AJCC 8th edition stage 1 and 2 (T1 N2, T2 N1-N2, T3 N0-N2) excluding T1N0-N1 and T2N0 (Brian O'Sullivan et al. 2016) * Tumor positive for p16 immunohistochemistry (IHC) (defined as greater than 70% strong nuclear or nuclear and cytoplasmic staining of tumor cells) and HPV DNA in situ hybridization or HPV messenger ribonucleic acid (mRNA) RNAScope. Repeat samples may be required if adequate diagnostic tissue is unavailable for testing * Zubrod Performance Status of 0-1 * Patients must have radiographically evident measurable disease at the primary site or at nodal stations per response evaluation criteria in solid tumors (Response Evaluation Criteria in Solid Tumors \[RECIST\]) 1.1 documented by diagnostic quality CT or magnetic resonance imaging (MRI) of the neck with contrast within 28 days prior to registration; a FDG-PET/CT of the neck performed for the purposes of radiation planning is acceptable as a substitute if the CT is of diagnostic quality * Diagnostic quality cross sectional imaging of the thorax within 28 days prior to registration. A 18-FDG-PET/CT or conventional CT are acceptable * FDG-PET/CT of the neck is required within 28 days prior to registration for comparison to post treatment FDG-PET/CT. Note: Repeat imaging for variability within 96 hours of this time frame should be allowed to avoid unnecessary re-imaging and its financial and potential physical consequences for patients * History and physical exam within 1 month prior to registration * Fiberoptic exam with laryngopharyngoscopy (mirror and/or fiberoptic and/or direct procedure) by radiation oncologist, medical oncologist or ear, nose, throat (ENT)/head and neck surgeons within 28 days prior to registration * Absolute neutrophil count (ANC) \>= 1,500 cells/mm\^3 (within 2 weeks prior to registration) * Platelets \>= 100,000 cells/mm\^3 (within 2 weeks prior to registration) * Hemoglobin \>= 8.0 g/dl (within 2 weeks prior to registration); Note: The use of transfusion or other intervention to achieve Hgb \>= 8.0 g/dl is acceptable * Serum creatinine \< 1.5 mg/dl or creatinine clearance (CC) \>= 50 ml/min determined by 24-hour collection or estimated by Cockcroft-Gault formula (within 2 weeks prior to registration) * Bilirubin \< 2 mg/dl (within 2 weeks prior to registration) * Aspartate aminotransferase (AST) or alanine aminotransferase (ALT) \< 3 x the upper limit of normal (within 2 weeks prior to registration) * Sodium (Na), potassium (K), chloride (Cl), glucose, calcium (Ca), magnesium (Mg), albumin, amylase, lipase, thyroid stimulating hormone (TSH) within 2 weeks prior to registration * Negative serum pregnancy test within 2 weeks prior to registration for women of childbearing potential * Seronegative for active hepatitis-B or hepatitis-C infection and seronegative for human immunodeficiency virus (HIV) * Mandatory submission of hematoxylin and eosin (H\&E) and paraffin-embedded tumor block or unstained slides Exclusion Criteria: * Cancers of the oral cavity (oral tongue, floor mouth, alveolar ridge, buccal or lip), or the nasopharynx, hypopharynx, or larynx, even if p16 positive * Carcinoma of the neck of unknown primary site origin (even if p16 positive) * Definitive clinical or radiologic evidence of metastatic disease or adenopathy below the clavicles * Gross total excision of both primary and nodal disease with curative intent; this includes tonsillectomy, local excision of primary site, and nodal excision that removes all clinically and radiographically evident disease * Simultaneous primary cancers or separate bilateral primary tumor sites * Prior invasive malignancy (except non-melanomatous skin cancer) unless disease free for a minimum of 1095 days (3 years) (for example, carcinoma in situ of the breast, oral cavity, or cervix are all permissible) * Prior systemic chemotherapy for the study cancer; note that prior chemotherapy for a different cancer is allowable; patients who have received PD-1/PD-L1 or CTLA4 therapy for a previous malignancy are not eligible * Prior RT to the region of the study cancer that would result in overlap of radiation therapy fields * Severe, active co-morbidity defined as any of the following: Unstable angina and/or congestive heart failure requiring hospitalization within the last 6 months; acute bacterial or fungal infection requiring intravenous antibiotics at the time of registration; chronic obstructive pulmonary disease exacerbation or other respiratory illness requiring hospitalization or precluding study therapy within 30 days of registration; hepatic insufficiency resulting in clinical jaundice and/or coagulation defects; a diagnosis of immunodeficiency or use of any form of systemic immunosuppressive therapy. The use of physiologic doses of corticosteroids may be approved after consultation with the sponsor * Active autoimmune disease that has required systemic treatment in past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment. Steroid premedications for contrast allergy allowed * Has evidence of active, non-infectious pneumonitis * Has received a live vaccine within 30 days of planned start of study therapy * Pregnancy or breast-feeding; this exclusion is necessary because the treatment involved in this study may be significantly teratogenic * History of severe hypersensitivity or contraindication to CT or PET contrast material uncontrolled with pre-medications (steroids, antihistamines)