Stereotactic Body Radiation Therapy, Tremelimumab and Durvalumab in Treating Participants With Recurrent or Metastatic Cervical, Vaginal, or Vulvar Cancers

Inclusion Criteria: * Written informed consent obtained from subject prior to any protocol related procedures * Performance status Eastern Cooperative Oncology Group (ECOG) 0-2. * Body weight \> 30 kg * Must have an average life expectancy of 6 months * Patient is able and willing to comply with protocol and study procedures for the duration of the study including undergoing treatment and scheduled visits and examinations including follow-up visits * Histological diagnosis of recurrent or metastatic cervical, vaginal, or vulvar cancer or an unknown pelvic malignancy most likely to have arisen from these sites as determined clinically by the treating physicians (i.e. squamous cell carcinoma or adenocarcinoma that is high risk \[HR\] human papillomavirus positive \[HPV\]+, but not limited to this example) * Metastatic disease in at least two distinct lesions (including the index lesion to be treated) with at least one site outside of the radiation field and evaluable by Response Evaluation Criteria in Solid Tumors (RECIST) criteria for evaluation of response * At least one index lesion to be treated measuring 1 cm amenable to hypofractionated radiation therapy * Staging computed tomography (CT) scans done prior to enrollment * Have had at least one line of prior platinum-based systemic chemotherapy once diagnosed with recurrence or metastatic disease if primary cervical cancer. If a patient has primary vulvar or vaginal cancer, there is not a requirement. * May have received 1 prior biologic regimen (i.e. avastin) but not within 4 weeks of enrollment * Full recovery from the acute effects of prior treatments, defined as effects having resolved to National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) version (v) 4.03 grade 0 or 1 with the exception of alopecia and certain laboratory values as outlined below; subjects with irreversible toxicity that is not reasonably expected to be exacerbated by durvalumab and tremelimumab may be included (e.g., hearing loss, neuropathy) upon approval of the principal investigator (PI) * In patients with central nervous system (CNS) metastases, metastases must be asymptomatic at the time of day 1 of the study and meet the following criteria: * Brain metastases should have been treated with either whole brain radiation therapy (WBRT), stereotactic radiosurgery (SRS)/gamma-knife, or surgical resection; * At least 28 days without progression of CNS metastases as evidenced by magnetic resonance imaging (MRI) or CT from last day of treatment with radiation to the CNS metastases; * At least 3 months from surgical resection (if had surgery) with stability on MRI brain at enrollment; * At least 14 days since last dose of corticosteroids; * Must not have leptomeningeal disease or cord compression * Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 2.5 x upper limit of normal (ULN); for subjects with liver metastasis, ALT and AST =\< 5 x ULN within 3 weeks prior to initial treatment * Total bilirubin =\< 1.5 x ULN except in patients with documented Gilbert's syndrome or liver metastasis, who must have a baseline total bilirubin =\< 3.0 mg/dl within 3 weeks prior to initial treatment * Serum creatinine clearance (CL) \> 40 mL/min by the Cockcroft-Gault formula or by 24-hour urine collection for determination of creatinine clearance within 3 weeks prior to initial treatment * Negative screening test results for hepatitis B, hepatitis C, and human immunodeficiency virus * Absolute neutrophil count (ANC) \>= 1,500 cells/ul without growth factor support prior to initial treatment * Hemoglobin \>= 9 g/dL prior to initial treatment * Platelet count \>= 100,000 platelets/ul prior to initial treatment * Subjects must either be of non-reproductive potential (ie, post-menopausal by history; OR history of hysterectomy, OR history of bilateral tubal ligation, OR history of bilateral oophorectomy) or must have a negative serum or urine pregnancy test upon study entry; women will be considered post-menopausal if they have been amenorrheic for 12 months without an alternative medical cause; the following age-specific requirements apply: * Women \< 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of exogenous hormonal treatments and if they have luteinizing hormone and follicle-stimulating hormone levels in the post-menopausal range for the institution or underwent surgical sterilization (bilateral oophorectomy or hysterectomy); they would also be considered had radiation-induced or chemotherapy-induced menopause with last menses \> 1 year ago * Women \>= 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of all exogenous hormonal treatments, had radiation-induced menopause with last menses \> 1 year ago, had chemotherapy-induced menopause with last menses \> 1 year ago, or underwent surgical sterilization (bilateral oophorectomy, bilateral salpingectomy or hysterectomy) * Women of any age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of all exogenous hormonal treatments, had radiation-induced or chemotherapy-induced menopause * Females of childbearing potential who are sexually active with a nonsterilized male partner must use highly effective method of contraception from the time of screening, and must agree to continue using such precautions for 180 days after the final dose of durvalumab and tremelimumab; cessation of contraception after this point should be discussed with a responsible physician; they must also refrain from egg cell donation for 180 days after the final dose of durvalumab and tremelimumab * Note: A highly effective method of contraception is defined as one that results in a low failure rate (i.e., less than 1% per year) when used consistently and correctly; the acceptable methods of contraception include barrier methods (male condom plus spermicide, copper T intrauterine device, levonorgesterel-releasing intrauterine system) or hormonal methods (implants, hormone shot or injection, combined pill, minipill, patch) Exclusion Criteria: * Involvement in the planning and/or conduct of the study * Previous enrollment in the present study * Female patients who are pregnant or breastfeeding or male or female patients of reproductive potential who are not willing to employ effective birth control from screening to 90 days after the last dose of durvalumab monotherapy or 180 days after the last dose of durvalumab + tremelimumab combination therapy * Prior treatment with an anti-CTLA-4, including tremelimumab PD-1 or PD-L1 inhibitor, including durvalumab * Prior oncology vaccine therapy * Prior radiation treatment to the index lesion to be treated * Prior radiation which overlaps and precludes hypofractionated treatment to the index lesion * Treatment with other investigational agent within 4 weeks to the first dose of tremelimumab or durvalumab * Concomitant therapy with any of the following: interleukin-2 (IL-2), interferon, or other non-study immunotherapy regimens; cytotoxic chemotherapy; immunosuppressive agents; other investigational therapies; all such therapies must have been discontinued \> 4 weeks * Any unresolved toxicity (CTCAE grade \< 2) from previous anti-cancer therapy; (subjects with irreversible toxicity that is not reasonably expected to be exacerbated by the investigational product may be included (e.g., hearing loss, peripherally neuropathy) * Any prior grade \>= 3 immune-related adverse event (imAE) while receiving any previous immunotherapy agent, or any unresolved imAE \> grade 1 * Treatment with a vascular endothelial growth factor (VEGF) inhibitor within the last 6 weeks * Major surgical procedure (as defined by the treating physician) within 28 days prior to the first dose of durvalumab and tremelimumab or still recovering from prior surgery * Active cardiac disease defined as unstable angina, uncontrolled hypertension, myocardial infarction in the last six months (unless successfully treated with coronary artery bypass grafting \[CABG\] or percutaneous transluminal coronary angioplasty \[PTCA\]), uncontrolled arrhythmia, or symptomatic congestive heart failure; \> 3 heart-related hospitalizations in the past year * Mean QT interval corrected for heart rate (QTc) \>= 470 ms calculated from 3 electrocardiogram (ECGs) using Fridericia's correction from triplicate ECGs in those patients who have an initial abnormal EKG on screening * Severe chronic obstructive pulmonary disease (COPD) requiring \> 3 hospitalizations in the past year * Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease \[eg, colitis or Crohn's disease\], diverticulitis \[with the exception of diverticulosis\], systemic lupus erythematosus, sarcoidosis syndrome, or Wegener syndrome \[granulomatosis with polyangiitis, Graves' disease, rheumatoid arthritis, hypophysitis, uveitis, etc\]); the following are exceptions to this criterion: * Patients with vitiligo or alopecia; * Patients with hypothyroidism (eg, following Hashimoto syndrome) stable on hormone replacement; * Any chronic skin condition that does not require systemic therapy; * Patients without active disease in the last 5 years may be included but only after consultation with the study physician; * Patients with celiac disease controlled by diet alone * Active or prior documented interstitial lung disease * Current or prior use of immunosuppressive medication within 14 days before the first dose of durvalumab and tremelimumab with the exceptions of intranasal and inhaled corticosteroids, systemic corticosteroids at physiologic doses not to exceed 10 mg/day of prednisone or equivalent, or steroids used transiently to control contrast agent allergies for radiographic studies * History of allogeneic organ transplant * History of hypersensitivity to durvalumab or tremelimumab or any CTLA4, PD1, or PDL-1 inhibitor * Receipt of live attenuated vaccination within 30 days prior to study entry or within 30 days of receiving durvalumab or tremelimumab * Active infection including tuberculosis (clinical evaluation that includes clinical history, physical examination and radiographic findings, and tuberculosis \[TB\] testing in line with local practice), hepatitis B (known positive hepatitis B virus \[HBV\] surface antigen \[HBsAg\] result), hepatitis C, or human immunodeficiency virus (positive HIV 1/2 antibodies); patients with a past or resolved HBV infection (defined as the presence of hepatitis B core antibody \[anti-HBc\] and absence of HBsAg) are eligible; patients positive for hepatitis C virus (HCV) antibody are eligible only if polymerase chain reaction is negative for HCV ribonucleic acid (RNA) * History of prior bowel fistula, ulcerations, or perforations * Evidence of progressive or symptomatic central nervous system (CNS) metastases or leptomeningeal disease * Uncontrolled inter-current illness, including, but not limited to, ongoing or active infection requiring systemic treatment, current or history of prior radiation induced pneumonitis, interstitial lung disease, or psychiatric illness/social situations that would limit compliance with study requirement or compromise the ability of the subject to give written informed consent * Other active invasive malignancy; history of non-invasive malignancies such as ductal carcinoma in situ of the breast, non-melanomatous carcinoma of the skin, is allowed, as is history of other invasive malignancy that is in remission for \>= 5 years after treatment with curative intent * Any medical, psychological, or social condition that, in the opinion of the treating physician would interfere with evaluation of the investigational product or interpretation of subject safety or study results