Radiation Therapy With Durvalumab or Cetuximab in Treating Patients With Locoregionally Advanced Head and Neck Cancer Who Cannot Take Cisplatin

Inclusion Criteria: * PRIOR TO STEP 1 REGISTRATION INCLUSION CRITERIA * Patients must have pathologically confirmed, previously untreated, unresected squamous cell carcinoma of the larynx, hypopharynx, oropharynx, oral cavity, or carcinoma of unknown head/neck primary prior to step 1 registration; submission of hematoxylin and eosin (H\&E) stained slides and formalin-fixed and paraffin-embedded (FFPE) tissue block (or punch biopsy of FFPE block) to the biospecimen bank at University of California, San Francisco (UCSF) for central review for oropharyngeal and unknown primaries and for p16 analysis for all other non-oropharyngeal primaries is mandatory for all patients; investigators should check with their pathology department regarding release of biospecimens before approaching patients about participation in the trial; for oropharyngeal and unknown primaries, submission of H\&E and p16 stained slides (with the required block for PD-L1) to the biospecimen bank at UCSF for central review is also required prior to step 2 registration * Note: fine needle aspirates (FNA) samples are not acceptable since they do not provide enough material for PD-L1 and p16 testing; however, if a cell block derived from the FNA is available, it is allowable if there are sufficient cells present in the block for PD-L1 testing; Dr. Jordan will determine this upon receipt; for sites submitting FNA cell blocks for ALL patients they must do so within 7-10 business days from registering the patient; sites must confirm with their cytology/pathology labs to make sure they can provide the required material as the bank must be able to retain these samples for the mandatory testing * Patients must have locoregionally advanced head and neck squamous cell carcinoma (HNSCC) * For p16-positive oropharyngeal/unknown primaries, American Joint Committee on Cancer \[AJCC\] 8th edition stage III and selected stage I-II based on smoking status in pack-years * For laryngeal, hypopharyngeal, and oral cavity primaries and p16-negative oropharyngeal/unknown primaries, AJCC 8th edition stage III-IVB * Based on the following minimum diagnostic workup within 60 days prior to step 1 registration: * General history and physical examination by a radiation oncologist or medical oncologist or ear, nose and throat (ENT) or head \& neck surgeon * For larynx, hypopharynx, and base of tongue primaries, a laryngopharyngoscopy (mirror and/or fiberoptic and/or direct procedure) is required, unless the patient cannot tolerate or refuses * Imaging of the head and neck with a neck CT or magnetic resonance imaging (MRI) (with contrast, unless contraindicated) or PET/CT; note that the CT portion of the PET/CT must be of diagnostic quality, including contrast administration unless contraindicated. If the CT portion of the PET/CT study is low-dose (non-diagnostic), then an additional CT or MRI study with contrast (unless contraindicated) is required * Chest imaging: chest CT with and without contrast (unless contraindicated) or PET/CT * Patients must have a contraindication to cisplatin as defined in the following bullet points; sites must complete the online tool at comogram.org prior to step 1 registration to determine if the patient is eligible; the scores must be recorded on a case report form (CRF) * Age \>= 70 with moderate to severe comorbidity or vulnerability to cisplatin, defined as having one or more of the following conditions within 30 days prior to step 1 registration: * Modified Charlson Comorbidity Index \>= 1 * Adult Comorbidity Evaluation (ACE)-27 Index \>= 1 * Generalized Competing Event Model for Cancer Risk (GCE) omega PFS score \< 0.80 * Geriatric screening (G-8) score =\< 14 * Cancer and Aging Research Group (CARG) toxicity score \>= 30% * Cumulative Illness Rating scale for Geriatrics (CIRS-G) score \>= 4 OR * Age \< 70 with severe comorbidity or vulnerability to cisplatin, defined as having two or more of the following conditions within 30 days prior to step 1 registration * Modified Charlson Comorbidity Index \>= 1 * ACE-27 Index \>= 1 * GCE omega PFS-score \< 0.80 * G-8 score =\< 14 * CARG Toxicity score \>= 30% * CIRS-G score \>= 4 OR * Age \>= 18 with an absolute or relative contraindication to cisplatin, defined as one or more of the following within 30 days prior to step 1 registration: * Creatinine clearance (CC) \> 30 and \< 60 cc/min; for this calculation, use the Cockcroft-Gault formula * Zubrod performance status 2 prior to step 1 registration * Pre-existing peripheral neuropathy grade \>= 1 * History of hearing loss, defined as either: * Existing need of a hearing aid OR * \>= 25 decibel shift over 2 contiguous frequencies on a pretreatment hearing test as clinically indicated * Absolute neutrophil count (ANC) \>= 1,000 cells/mm\^3 (within 14 days prior to step 1 registration) * Platelets \>= 100,000 cells/mm\^3 (within 14 days prior to step 1 registration) * Hemoglobin \>= 9.0 g/dl (Note: the use of transfusion or other intervention to achieve hemoglobin \[Hgb\] \>= 9.0 g/dl is acceptable) (within 14 days prior to step 1 registration) * Aspartate aminotransferase (AST) or alanine aminotransferase (ALT) =\< 2.5 times institutional upper limit of normal (within 14 days prior to step 1 registration) * Serum bilirubin =\< 1.5 x institutional upper limit of normal (within 14 days prior to step 1 registration) * Measured creatinine clearance (CL) \> 30 mL/min or calculated creatinine CL \> 30 mL/min by the Cockcroft-Gault formula (Cockcroft and Gault 1976) or by 24-hour urine collection for determination of creatinine clearance (within 14 days prior to step 1 registration) * For women of childbearing potential, a negative serum or urine pregnancy test within 14 days prior to step 1 registration; Note: women will be considered post-menopausal if they have been amenorrheic for 12 months without an alternative medical cause; the following age-specific requirements apply: * Women \< 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of exogenous hormonal treatments and if they have luteinizing hormone and follicle-stimulating hormone levels in the post-menopausal range for the institution or underwent surgical sterilization (bilateral oophorectomy or hysterectomy) * Women \>= 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of all exogenous hormonal treatments, had radiation-induced menopause with last menses \> 1 year ago, had chemotherapy-induced menopause with last menses \> 1 year ago, or underwent surgical sterilization (bilateral oophorectomy, bilateral salpingectomy or hysterectomy) * The patient or a legally authorized representative must provide study-specific informed consent prior to step 1 registration * PRIOR TO STEP 2 REGISTRATION INCLUSION CRITERIA * For patients with oropharyngeal or unknown primaries: p16 determination by immunohistochemistry (defined as greater than 70% strong nuclear or nuclear and cytoplasmic staining of tumor cells), confirmed by central pathology review * Note: for patients with oral cavity, laryngeal, and hypopharyngeal primaries, analysis of p16 status prior to step 2 registration/randomization is not required (p16 status will be analyzed centrally post-hoc); step 2 registration for these patients can be completed after step 1 registration Exclusion Criteria: * PRIOR TO STEP 1 REGISTRATION EXCLUSION CRITERIA * Prior invasive malignancy within the past 3 years (except for non-melanomatous skin cancer, and early stage treated prostate cancer); synchronous head and neck primaries are ineligible * Prior radiotherapy to the region of the study cancer that would result in overlap of radiation therapy fields * Note: Prior external beam radiotherapy is excluded, but iodine 131 is allowed * Prior immunotherapy * Prior systemic therapy, including cytotoxic chemotherapy, biologic/targeted therapy, or immune therapy for the study cancer * Major surgery within 28 days prior to step 1 registration * Proven evidence of distant metastases * If both of the following conditions are present, the patient is ineligible: * =\< 10 pack-year smoking history * p16-positive carcinoma of the oropharynx or unknown primary that are T0-3, N0-1 (AJCC 8th Edition) * Note: in the event that a registered patient with =\< 10 pack-years has a p16-positive result on central review with the tumor and nodal stage T0-3, N0-1 (AJCC 8th Edition), then the site will be notified that the patient is ineligible * Zubrod performance status \>= 3 * Body weight =\< 30 kg * Patients with oral cavity cancer are excluded from participation if the patient is medically operable and resection of the primary tumor is considered technically feasible by an oral or head and neck cancers surgical subspecialist;(please consult the surgical oncology co-principal investigator \[PI\], Steven Chang, Doctor of Medicine \[MD\], if clarification is needed on an individual case) * Sodium \< 130 mmol/L or \> 155 mmol/L (within 14 days of step 1 registration, unless corrected prior to step 1 registration) * Potassium \< 3.5 mmol/L or \> 6 mmol/L (within 14 days of step 1 registration, unless corrected prior to step 1 registration) * Fasting glucose \< 40 mg/dl or \> 400 mg/dl (within 14 days of step 1 registration, unless corrected prior to step 1 registration) * Serum calcium (ionized or adjusted for albumin) \< 7 mg/dl or \> 12.5 mg/dl (within 14 days of step 1 registration, unless corrected prior to step 1 registration) * Magnesium \< 0.9 mg/dl or \> 3 mg/dl (within 14 days of step 1 registration, unless corrected prior to step 1 registration) * Unstable angina and/or congestive heart failure requiring hospitalization within 3 months prior to step 1 registration * Transmural myocardial infarction within 3 months prior to step 1 registration * Respiratory illness requiring hospitalization at the time of step 1 registration * Note: if the respiratory illness is resolved and the patient meets the eligibility status above, then the patient can be considered for the trial * Idiopathic pulmonary fibrosis or other severe interstitial lung disease that requires oxygen therapy or is thought to require oxygen therapy within 1 year prior to step 1 registration * History of (non-infectious) pneumonitis that required steroids or current pneumonitis * Clinically apparent jaundice and/or known coagulation defects * Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease \[e.g., colitis or Crohn's disease\], diverticulitis \[with the exception of diverticulosis\], systemic lupus erythematosus, Sarcoidosis syndrome, or Wegener syndrome \[granulomatosis with polyangiitis\], Graves' disease, rheumatoid arthritis, hypophysitis, uveitis, etc.) * The following are exceptions to this criterion: * Patients with vitiligo or alopecia; * Patients with hypothyroidism (e.g., following Hashimoto syndrome) stable on hormone replacement; * Any chronic skin condition that does not require systemic therapy; * Patients without active disease in the last 5 years may be included but only after consultation with the medical oncology study chair; * Patients with celiac disease controlled by diet alone * History of active primary immunodeficiency including, but not limited to acquired immune deficiency syndrome (AIDS) based upon current Centers for Disease Control and Prevention (CDC) definition; Note: human immunodeficiency virus (HIV) testing is not required for entry into this protocol; the need to exclude patients with AIDS from this protocol is necessary because the treatment involved in this protocol may be immunosuppressive; patients with known HIV, CD4 counts \>= 200/uL, and undetectable viral loads who are stable on an antiretroviral regimen may be included * Current or prior use of immunosuppressive medication within 14 days before step 1 registration, with the exceptions of intranasal and inhaled corticosteroids or systemic corticosteroids at physiological doses, which are not to exceed 10 mg/day of prednisone, or an equivalent corticosteroid * Receipt of live attenuated vaccination within 30 days prior to step 1 registration * Medical or psychiatric illness which would compromise the patient's ability to tolerate treatment or limit compliance with study requirements * Pregnancy or women of childbearing potential and men who are sexually active and not willing/able to use medically acceptable forms of contraception during treatment and for 6 months after the last dose of cetuximab or MEDI14736 (durvalumab); this exclusion is necessary because the treatment involved in this study may be significantly teratogenic; women who are breastfeeding are also excluded * Prior allergic reaction or hypersensitivity to cetuximab or MEDI4736 (durvalumab) or any of study drug excipients * History of allogenic organ transplantation * Uncontrolled hypertension * Uncontrolled cardiac arrhythmia * Uncontrolled serious chronic gastrointestinal condition associated with diarrhea * Active infection including tuberculosis (clinical evaluation that includes clinical history, physical examination and radiographic findings, and tuberculosis \[TB\] testing in line with local practice), hepatitis B (known positive hepatitis B virus \[HBV\] surface antigen \[HBsAg\] result), hepatitis C; patients with a past or resolved HBV infection (defined as the presence of hepatitis B core antibody \[anti-HBc\] and absence of HBsAg) are eligible; patients positive for hepatitis C (hepatitis C virus \[HCV\]) antibody are eligible only if polymerase chain reaction is negative for HCV ribonucleic acid (RNA)