Stereotactic Body Radiation Therapy and Durvalumab With or Without Tremelimumab Before Surgery in Treating Participants With Human Papillomavirus Positive Oropharyngeal Squamous Cell Caner

Inclusion Criteria: * Written informed consent from the patient/legal representative prior to performing any protocol-related procedures, including screening evaluation. * Pathologically proven diagnosis of HPV-positive squamous cell carcinoma of the oropharynx. HPV-positivity will be defined as tumors that are p16-positive by immunohistochemistry. * Patients must have T0-3 disease with all gross disease amenable to R0 resection (reviewed by multidisciplinary study team) and is eligible for TORS in the opinion of the treating physician. * N0-N2b with all cervical disease confined to 2 cervical lymph node levels if the involved nodal levels are adjacent. * Karnofsky performance status \>= 70. * Body weight \>= 50 kg. * Patients who are medically operable, without pre-existing medical conditions that could inhibit surgery following neoadjuvant therapy, and do not refuse surgery. * Patients with smoking history (\< 20 pack year history) is allowed. * Patients must have MRI neck with and without contrast and a diagnostic positron emission tomography (PET), computed tomography (CT) or PET/CT skull base to mid-thigh with contrast within 30 days prior to SBRT. * Hemoglobin \>= 9.0 g/dL. * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9/L (\>= 1500 per mm\^3). * Platelet count \>= 100 (or 75) x 10\^9/L (\>= 75,000 per mm\^3). * Serum bilirubin =\< 1.5 x institutional upper limit of normal (ULN). This will not apply to subjects with confirmed Gilbert?s syndrome (persistent or recurrent hyperbilirubinemia that is predominantly unconjugated in the absence of hemolysis or hepatic pathology), who will be allowed only in consultation with their physician. * Aspartate aminotransferase (AST)/alanine aminotransferase (ALT) =\< 2.5 x institutional upper limit of normal. * Serum creatinine \> 40 mL/min. * Female patients with reproductive potential (e.g., females menopausal for less than 1 year and not surgically sterilized) must agree to practice two highly effective contraceptive measures for the duration of study drug therapy and for at least 90 days after completion of durvalumab monotherapy or for at least 180 days after completion of durvalumab/tremelimumab combination therapy. Female patients of childbearing potential must provide a negative pregnancy test (urine) prior to treatment initiation. Male patients, whose partners are women with reproductive potential, and the women themselves also must agree to practice two effective contraceptive measures. * Female patients with evidence of post-menopausal status if they have been amenorrheic for 12 months without an alternative medical cause. The following age-specific requirements apply: * Women \< 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of exogenous hormonal treatments and if they have luteinizing hormone and follicle-stimulating hormone levels in the post-menopausal range for the institution or underwent surgical sterilization (bilateral oophorectomy or hysterectomy). * Women \>= 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of all exogenous hormonal treatments, had radiation-induced menopause with last menses \> 1 year. * Subject is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations including follow up. Exclusion Criteria: * Karnofsky performance status (KPS) \< 70. * Body weight \< 50 kg. * Histology other than squamous cell carcinoma. * Primary site other than oropharynx (i.e. oral cavity, salivary glands, nasal cavity, paranasal sinuses, larynx, or nasopharynx) or of unknown primary. * Patients with synchronous or bilateral disease. * Patient with N3 nodal disease, N2c nodal disease, and N2b disease with gross disease in more than 2 neck levels (levels not adjacent to each others are also not allowed). * Patients with recurrent head and neck cancer. * Patients with metastatic disease on initial staging study. * Patients who underwent attempted resection rather than the diagnostic biopsy of the primary site or nodal sampling of the neck disease. * Prior systemic therapy (chemotherapy, biologic, or immunotherapy) for the same OPSCC. * Previous treatment with Anti-CTLA-4 (cytotoxic T-lymphocyte-associated protein 4) including tremelimumab or PD1/PD-L1 inhibitor (programmed cell death) , including durvalumab. * Concurrent enrollment in another clinical trial. * Patients that received prior radiotherapy that would result in overlap of radiation therapy fields. * History of another primary malignancy except for: * Malignancy treated with curative intent and with no evidence of active disease \>= 5 years before the first dose of study drug and of low potential risk for recurrence * Adequately treated non-melanoma skin cancer or lentigo maligna without evidence of disease * Adequately treated carcinoma in situ without evidence of disease. * Any concurrent anticancer therapy. * Mean QT interval corrected for heart rate (QTc) \>= 470 ms calculated from a baseline electrocardiogram using Fredericia?s correction. * Current or prior use of immunosuppressive medication within 14 days before the first dose of durvalumab or tremelimumab, with the exceptions of intranasal and inhaled corticosteroids or systemic corticosteroids at physiological doses, which are not to exceed 10 mg/day of prednisone, or an equivalent corticosteroid. * Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease \[e.g., colitis or Crohn's disease\], diverticulitis \[with the exception of diverticulosis\], systemic lupus erythematosus, Sarcoidosis syndrome, or Wegener syndrome \[granulomatosis with polyangiitis, Graves' disease, rheumatoid arthritis, hypophysitis, uveitis, etc\]). The following are exceptions to this criterion: * Patients with vitiligo or alopecia * Patients with hypothyroidism (e.g., following Hashimoto syndrome) stable on hormone replacement * Any chronic skin condition that does not require systemic therapy * Patients without active disease in the last 5 years may be included but only after consultation with the study physician * Patients with celiac disease controlled by diet alone. * History of allogeneic organ transplant. * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, seizures, symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, cardiac arrhythmia, interstitial lung disease, active peptic ulcer disease or gastritis, serious chronic gastrointestinal (GI) conditions associated with diarrhea, active bleeding diatheses including any subject known to have evidence of acute or chronic hepatitis B, hepatitis C or human immunodeficiency virus (HIV), or psychiatric illness/social situations that would limit compliance with study requirements or compromise the ability of the subject to give written informed consent. * Receipt of live attenuated vaccination within 30 days prior to study entry or within 30 days of receiving durvalumab or tremelimumab. * Female patients who are pregnant or breastfeeding or male or female patients of reproductive potential who are not willing to employ two highly effective birth control methods from screening to 90 days after the last dose of durvalumab monotherapy or 180 days after the last dose of durvalumab + tremelimumab combination therapy. * Major surgical procedure (as defined by the investigator) within 28 days prior to the first dose of investigational product (IP). Note: Local surgery of isolated lesions for palliative intent is acceptable. * Any unresolved toxicity National Cancer Institute (NCI) CTCAE grade \>= 2 from previous anticancer therapy with the exception of alopecia, vitiligo, and the laboratory values defined in the inclusion criteria * Patients with grade \>= 2 neuropathy will be evaluated on a case-by-case basis after consultation with the study physician. * Patients with irreversible toxicity not reasonably expected to be exacerbated by treatment with durvalumab or tremelimumab may be included only after consultation with the study physician. * Any condition that, in the opinion of the investigator, would interfere with evaluation of study treatment or interpretation of patient safety or study results. * Known allergy or hypersensitivity to IP or any excipient. * Subjects with uncontrolled seizures. * History of leptomeningeal carcinomatosis. * History of active primary immunodeficiency. * Clinical documentation of active infection including tuberculosis (clinical evaluation that includes clinical history, physical examination and radiographic findings, and tuberculosis (TB) testing in line with local practice), hepatitis B (known positive hepatitis B virus \[HBV\] surface antigen (HBsAg) result), hepatitis C, or human immunodeficiency virus (positive HIV 1/2 antibodies). Patients with a past or resolved HBV infection (defined as the presence of hepatitis B core antibody \[anti-HBc\] and absence of HBsAg) are eligible. Patients positive for hepatitis C (HCV) antibody are eligible only if polymerase chain reaction is negative for HCV ribonucleic acid (RNA).