Durvalumab, Tremelimumab and Hypofractionated Radiation Therapy in Treating Patients With Recurrent or Metastatic Head and Neck Squamous Cell Carcinoma

Inclusion Criteria: * Histologically proven recurrent/metastatic squamous cell carcinoma arising from a previous head and neck primary site, and located within the head and neck region, lung mediastinum, lymph nodes, soft tissue metastases or bone, and who are not candidates for curative intent therapy * An actual body weight \> 40kg * Demonstrated disease progression during, or after discontinuation, of the most recent line of systemic therapy * Have received any number lines of prior systemic therapy (including systemic therapy in the curative intent setting, and including a platinum containing regimen) * Have received an anti-PD1 or anti PDL1 monoclonal antibody * Have a target lesion/s deemed suitable by the treating physicians for hypofractionated radiation therapy (HIGRT or SBRT) with the intent of palliation or prevention of symptoms; this lesion must be: a) 1-3 non overlapping sites in the head and neck region OR b) metastatic lesions outside the head and neck (H\&N) region in the lung mediastinum, soft tissue metastases, lymph nodes or bone (a minimum of 1 and a maximum 5 lesions will be irradiated), provided there is no significant overlap between the lesions; patients should have RECIST 1.1 criteria measurable disease in addition to the lesion/s treated with radiation; if the site/s of radiation were previously radiated to high dose RT (\> 50 Gy), there should be \> 6 month time interval between the last dose of radiation and the start of radiation * Have the ability to tolerate required radiotherapy-related procedures (e.g.: lie flat and hold position for treatment) as determined by the treating physician * Be willing and able to provide written informed consent for the trial and comply with the study visit requirements * Have measurable disease based on RECIST 1.1. (in addition to the lesion/s that will be treated with hypofractionated radiation therapy) * Have provided tissue from an archival tissue sample or newly obtained core or excisional biopsy of a tumor lesion * Have a performance status of 0 or 1 on the Eastern Cooperative Oncology Group (ECOG) performance scale * Hemoglobin \>= 9.0 g/dL (should be performed within 10 days of treatment initiation) * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9/L (\>= 1500 per mm\^3) (should be performed within 10 days of treatment initiation) * Platelet count \>= 100 x 10\^9/L (\>= 100,000 per mm\^3) (should be performed within 10 days of treatment initiation) * Serum bilirubin =\< 1.5 x institutional upper limit of normal (ULN) (should be performed within 10 days of treatment initiation); this will not apply to subjects with confirmed Gilbert's syndrome (persistent or recurrent hyperbilirubinemia that is predominantly unconjugated in the absence of hemolysis or hepatic pathology), who will be allowed only in consultation with their physician * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x institutional upper limit of normal unless liver metastases are present, in which case it must be =\< 5 x ULN (should be performed within 10 days of treatment initiation) * Serum creatinine clearance (CL) \> 60 mL/min by the Cockcroft-Gault formula (Cockcroft and Gault 1976) or by 24-hour urine collection for determination of creatinine clearance (should be performed within 10 days of treatment initiation) * Evidence of post-menopausal status OR negative urinary or serum pregnancy test for female pre-menopausal patients; women will be considered post-menopausal if they have been amenorrheic for 12 months without an alternative medical cause; the following age-specific requirements apply: * Women \< 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of exogenous hormonal treatments and if they have luteinizing hormone and follicle-stimulating hormone levels in the post-menopausal range for the institution or underwent surgical sterilization (bilateral oophorectomy, or hysterectomy) * Women \>= 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of all exogenous hormonal treatments, had radiation-induced menopause with last menses \> 1 year ago, had chemotherapy-induced menopause with last menses \> 1 year ago, or underwent surgical sterilization (bilateral oophorectomy, bilateral salpingectomy or hysterectomy). * Female subjects of childbearing potential should have a negative urine or serum pregnancy within 72 hours prior to receiving the first dose of study medication; if the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required. * Female subjects of childbearing potential should be willing to use 1 method of highly effective birth control or be surgically sterile, or abstain from heterosexual activity for the course of the study through 180 days after the last dose of study medication; subjects of childbearing potential are those who have not been surgically sterilized or have not been free from menses for \> 1 year * Male subjects should agree to use an adequate method of contraception starting with the first dose of study therapy through 180 days after the last dose of study therapy * Patient is \>= 5 years free of another primary malignancy, except: a) if the other malignancy is basal cell carcinoma or cervical carcinoma in situ or b) if the other primary malignancy is not considered clinically significant and is requiring no active intervention Exclusion Criteria: * Has a body weight =\< 40kg at the time of enrollment * Is currently participating in or has participated in a study of an investigational agent or using an investigational device within 4 weeks of the first dose of treatment * Has a target lesion/s for radiotherapy that is \> 5 cm (\> 50 cc) in greatest dimension * Has a target lesion/s in a region that previously received high dose radiation therapy (RT) (\> 50 Gy) demonstrating any of the following: * carotid artery encasement (\> 180 degrees) * unprotected carotid artery (i.e. skin is directly over the carotid without intervening soft tissue, especially after prior neck dissection without a vascularized free flap) (a\&b due to risk of carotid blow out) * skin infiltration by tumor (due to risk of fistula) * located in the larynx/hypopharynx primaries (due airway threat) * treated with high dose radiation therapy (\> 50 Gy) within 6 months or less of trial enrollment * Any prior grade \>= 3 immune-related adverse event (irAE) while receiving a prior immunotherapy agent, or any unresolved irAE \> grade 1 * Current or prior use of immunosuppressive medication within 14 days before the first dose of durvalumab or tremelimumab; the following are exceptions to this criterion: * Intranasal, inhaled, topical steroids, or local steroid injections (e.g., intra articular injection) * Systemic corticosteroids at physiologic doses not to exceed 10 mg/day of prednisone or its equivalent * Steroids as premedication for hypersensitivity reactions (e.g., computed tomography \[CT\] scan premedication) * Has received a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =\< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier * Has received prior chemotherapy, targeted small molecule therapy, or radiation therapy within 2 weeks prior to study day 1 or who has not recovered (i.e., =\< grade 1 or at baseline) from adverse events due to a previously administered agent * Note: Subjects with =\< grade 2 neuropathy are an exception to this criterion and may qualify for the study * Note: If subject received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting therapy * Has a known additional malignancy that is progressing or requires active treatment; exceptions include basal cell carcinoma of the skin, squamous cell carcinoma of the skin, or in situ cervical cancer that has undergone potentially curative therapy * Has known brain metastases or spinal cord compression unless the patient is stable (asymptomatic; no evidence of new or emerging brain metastases; and stable and off steroids for at least 14 days prior to start of study treatment); following radiotherapy and/or surgery of the brain metastases patients must wait 4 weeks following the intervention and before initiating study treatment with imaging to confirm stability * Has an active autoimmune disease requiring systemic treatment within the past 2 years or a documented history of clinically severe autoimmune disease, or a syndrome that requires systemic steroids or immunosuppressive agents; subjects with vitiligo or resolved childhood asthma/atopy would be an exception to this rule; subjects that require intermittent use of bronchodilators or local steroid injections would not be excluded from the study; subjects with hypothyroidism stable on hormone replacement will not be excluded from the study * Has evidence of current interstitial lung disease (ILD) or pneumonitis or a prior history of ILD or pneumonitis requiring oral or intravenous glucocorticoids * Has an active infection requiring systemic therapy * Requires therapeutic anticoagulation or has known active bleeding diathesis * Has a history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the trial, interfere with the subject's participation for the full duration of the trial, or is not in the best interest of the subject to participate, in the opinion of the treating investigator * Has known psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the trial * Is pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the trial, starting with the pre-screening or screening visit through 180 days after the last dose of trial treatment * Has received prior therapy with an anti-Cytotoxic T-lymphocyte-associated antigen-4 (CTLA-4) antibody (including ipilimumab or any other antibody or drug specifically targeting T-cell co-stimulation or checkpoint pathways) * Has a known history of human immunodeficiency virus (HIV) (HIV 1/2 antibodies) * Has evidence of acute or chronic hepatitis B or hepatitis C * Has received a live vaccine within 30 days prior to the first dose of trial treatment * Has a mean QT interval corrected for heart rate (QTc) \>= 470ms calculated from 3 electrocardiograms (ECGs) using Fridericia's Correction * Has a history of primary immunodeficiency or an allogeneic organ transplant * Has a history of hypersensitivity to durvalumab or tremelimumab excipient * Known history of previous clinical diagnosis of tuberculosis * Uncontrolled intercurrent illness including, but not limited to symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, cardiac arrhythmia, active peptic ulcer disease or gastritis, seizures