Durvalumab With or Without Tremelimumab in Treating Participants With Stage II-IVA Oropharyngeal Squamous Cell Cancer

Inclusion Criteria: * Written informed consent to participate in the study according to the investigational review board (IRB) * Suspected or histologically/cytologically confirmed oropharyngeal squamous cell carcinoma (OPSCC), stage II, III, or IVA (according to the American Joint Committee on Cancer \[AJCC\] 7th edition), or patients with loco-regional recurrence from an OPSCC primary, if time of recurrence is at least 6 months after completion of initial curative intent treatment (surgery or radiotherapy +/- chemotherapy or cetuximab). Patients with a suspected lesion may be enrolled and a baseline biopsy will be obtained as part of the study. If squamous cell histology is not confirmed, patients will be discontinued from the study * Patients must have surgically resectable disease in the opinion of the treating physician. For patients with a primary OPSCC, patients must be eligible for TORS (transoral robotic surgery) * Available tissue from prior biopsy (minimum of 10 unstained slides), or willing to undergo core biopsy to obtain tumor material. Biopsy will be mandatory for patients with recurrent disease * Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1 * Hemoglobin \>= 9.0 g/dL * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9/L (\> 1500 per mm\^3) * Platelet count \>= 100 x 10\^9/L (\> 100,000 per mm\^3) * Serum bilirubin =\< 1.5 x institutional upper limit of normal (ULN). This will not apply to subjects with confirmed Gilbert's syndrome (persistent or recurrent hyperbilirubinemia that is predominantly unconjugated in the absence of hemolysis or hepatic pathology), who will be allowed only in consultation with their physician * Aspartate aminotransferase (AST)/alanine aminotransferase (ALT) =\< 2.5 x institutional upper limit of normal * Serum creatinine clearance (CL) \> 40 mL/min by the Cockcroft-Gault formula * Patients with reproductive potential (e.g., females menopausal for less than 1 year and not surgically sterilized) must practice two highly effective contraceptive measures for the duration of study drug therapy and for at least 90 days after completion of durvalumab monotherapy or for at least 180 days after completion of durvalumab/tremelimumab combination therapy. Female patients of childbearing potential must provide a negative pregnancy test (urine) prior to treatment initiation * Subject is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations including follow up Exclusion Criteria: * Histology other than squamous cell carcinoma * Primary site other than oropharynx * Prior systemic therapy (chemotherapy, biologic, or immunotherapy) for the same OPSCC. Prior chemotherapy, biologic therapy, and radiotherapy is allowed in patients with loco-regional recurrent disease, if administered at least 6 months prior to study enrolment * Previous treatment with Anti-CTLA-4 including tremelimumab or PD1/PD-L1 inhibitor, including durvalumab * History of another primary malignancy except for: (i) OPSCC with loco-regional recurrence after 6 months of curative-intent treatment and amenable to salvage surgery; (ii) malignancy treated with curative intent and with no evidence of active disease \>= 2 years before the first dose of study drug and of low potential risk for recurrence; (iii) non-melanoma, skin cancer or lentigo maligna; in situ cervical, breast, prostate or bladder carcinoma * Any concurrent anticancer therapy * Mean QT interval corrected for heart rate (QTc) \>= 470 ms calculated from a baseline electrocardiogram using Fridericia's correction * Current or prior use of immunosuppressive medication within 28 days before the first dose of durvalumab or tremelimumab, with the exceptions of intranasal and inhaled corticosteroids or systemic corticosteroids at physiological doses, which are not to exceed 10 mg/day of prednisone, or an equivalent corticosteroid, and single dose of dexamethasone of up to 20 mg (or equivalent corticosteroid) administered prior to diagnostic or baseline study biopsy of the oropharynx, as per institution standard of care * Active or prior documented autoimmune disease within the past 2 years NOTE: Subjects with vitiligo, Grave's disease, or psoriasis not requiring systemic treatment (within the past 2 years) are not excluded * Active or prior documented inflammatory bowel disease (e.g., Crohn's disease, ulcerative colitis) or pneumonitis * History of primary immunodeficiency * History of allogeneic organ transplant * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, seizures, symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, cardiac arrhythmia, active peptic ulcer disease or gastritis, active bleeding diatheses including any subject known to have evidence of acute or chronic hepatitis B, hepatitis C or human immunodeficiency virus (HIV), or psychiatric illness/social situations that would limit compliance with study requirements or compromise the ability of the subject to give written informed consent * Known history of active tuberculosis * Receipt of live attenuated vaccination within 30 days prior to study entry or within 30 days of receiving durvalumab or tremelimumab * Female subjects who are pregnant, breast-feeding or male or female patients of reproductive potential who are not employing two highly effective methods of birth control * Any medical or psychosocial condition that will interfere with evaluation of study treatment or interpretation of patient safety or study results * Known allergy or hypersensitivity to investigational product (IP) or any excipient * Any unresolved grade 2 or higher toxicity from previous anti-cancer therapy * Primary tumor not amenable to TORS