Vudalimab for the Treatment of Locally Advanced or Metastatic Anaplastic Thyroid Cancer or Hurthle Cell Thyroid Cancer

Inclusion Criteria: * MAIN COHORT (ANAPLASTIC THYROID CANCER \[ATC\]): Subjects enrolling in the Main Cohort must have a histologically confirmed diagnosis of anaplastic thyroid cancer (ATC). This includes cases with pathologic findings supporting the clinical impression of anaplastic thyroid carcinoma. Diagnosis may include terminology consistent with or suggestive of: anaplastic thyroid carcinoma, undifferentiated carcinoma, squamous carcinoma; carcinoma with spindled, giant cell, or epithelial features; or poorly differentiated carcinoma with pleomorphism, extensive necrosis with tumor cells present. (This study will not conduct central pathology review. Documentation on a standard pathology report is sufficient for determination of eligibility.) * MAIN COHORT (ATC): Subject's anaplastic thyroid cancer (ATC) must be either 1) metastatic ATC, or 2) incurable, locally-advanced ATC * MAIN COHORT (ATC): Subjects whose ATC carries a known BRAF V600E mutation must previously have received and failed, or be intolerant to, a BRAF/MEK inhibitor (e.g., dabrafenib or trametinib). Note: BRAF V600E mutation is not required to enroll in the trial, but if known to be present, this criterion must be met * MAIN COHORT (ATC): Subjects also must meet all general inclusion criteria * EXPLORATORY COHORT HURTHLE CELL THYROID CANCER \[HCC\]): Subjects enrolling in the Exploratory Cohort must have a histologically confirmed diagnosis of Hurthle cell thyroid cancer (HCC). (This study will not conduct central pathology review. Documentation on the pathology report is sufficient for determination of eligibility.) * EXPLORATORY COHORT (HCC): Subject's Hurthle cell thyroid cancer (HCC) must be incurable, metastatic HCC * EXPLORATORY COHORT (HCC): Subjects with HCC must previously have received and failed, or be intolerant to, at least one line of primarily anti-VEGFR tyrosine kinase inhibitor therapy (e.g., lenvatinib) * EXPLORATORY COHORT (HCC): Subjects with HCC must have demonstrated radiographic disease progression within =\< 14 months prior to enrollment * EXPLORATORY COHORT (HCC): Subjects also must meet all general inclusion criteria * GENERAL INCLUSION CRITERIA: * Subject's cancer must have progressed after treatment with all potentially curative standard therapies, or there must be no appropriate therapies available * Subjects must have measurable disease, as assessed by RECIST 1.1 * Subjects must be \>= 18 years of age * Subjects must exhibit an Eastern Cooperative Oncology Group (ECOG) performance status of =\< 2 or a Karnofsky performance status of \>= 60% * Tissue requirements for participation in correlative studies: * Prior to enrolling in the study, it must be determined if the subject has available adequate archival formalin-fixed paraffin embedded (FFPE) block(s)/slides containing tumor for correlative studies * If adequate archival tissue is available, collection of the tissue for use in correlative studies is mandatory. (Documentation of sufficient archival tissue availability should be noted in the study record prior to enrollment. Archival tissue need not be submitted until after the subject has been fully registered to the study. or * If adequate archival tissue is not available, subjects may still participate in the study. (Documentation of insufficient archival tissue availability should be noted in the study record.) In these cases, a pre-treatment fresh biopsy may be conducted per treating investigator discretion, if indicated; a portion of such biopsy tissue may be submitted for use in correlative studies, if sufficient biopsy material is available * Hemoglobin \> 9.0 g/dL * Without receipt of growth factor support and/or blood products/transfusion within =\< 28 days prior * Absolute neutrophil count (ANC) \>= 1.5 x 10\^9/L * Without receipt of growth factor support and/or blood products/transfusion within =\< 28 days prior * Platelet count \>= 100 x 10\^9/L * Without receipt of growth factor support and/or blood products/transfusion within =\< 28 days prior * Creatinine clearance \>= 50 mL/min * Creatinine clearance should be calculated using the Cockcroft-Gault formula * Serum total bilirubin \< 1.5 x upper limit of normal (ULN) (unless prior diagnosis and documentation of ongoing hemolysis or Gilbert's syndrome has been made, in which case there is no upper limit) * Aspartate aminotransferase/serum glutamic oxaloacetic transaminase (AST/SGOT) =\< 3.0 x ULN for subjects without known liver tumor involvement or =\< 5.0 x ULN for subjects with known liver tumor involvement * Hepatitis B screening requirements: * Subjects must undergo a hepatitis B surface antigen (HBsAg) test during screening, and the HBsAg test result must be negative; and * Subjects also must undergo a total hepatitis B core antibody (HBcAb) test during screening, and the total HBcAb test result must be negative; alternatively, a positive total HBcAb result is acceptable, provided the subject meets both of the below two criteria: * Subject with a positive total HBcAb test has a subsequent negative hepatitis B virus (HBV) deoxyribonucleic acid (DNA) test at screening. (The HBV DNA test only needs to be performed for subjects who have a negative HBsAg test and a positive HBcAb test.); and * Subject with a positive total HBcAb test must either receive effective suppressive HBV therapy while receiving trial therapy or agree to be re-tested for HBsAg and HBV DNA every 2 months (+/- 3 days) (wherein continuing receipt of trial therapy is contingent upon having negative monthly HBsAg and HBV DNA tests.) * Human immunodeficiency virus (HIV) screening requirements: * Subjects must undergo a human immunodeficiency virus (HIV) test during screening, and the HIV test result must be negative; alternatively, a positive HIV result is acceptable, provided the HIV-positive subject meets all of the below four criteria: * CD4+ T-cell (CD4+) counts must be \>= 350 cells/uL, * HIV viral load must be =\< 400 copies/mL, * Must not have any history of an acquired immunodeficiency syndrome (AIDS)-defining opportunistic infection within 12 months prior to registration on study, * Must be on an established antiretroviral therapy (ART) for at least 4 weeks prior to initiation of study drug dosing. (Effective ART is defined as a drug, dosage, and schedule associated with reduction and control of the viral load.) * HIV-positive subjects who do not meet all four of these inclusion criteria are not eligible * Subjects must undergo a hepatitis C virus antibody (HCV antibody) test during screening, and the HCV antibody test result must be negative; alternatively, a positive HCV antibody test result is acceptable, provided the subject has a subsequent negative hepatitis C virus ribonucleic acid (RNA) (HCV RNA) test at screening. (The HCV RNA test only needs to be performed for subjects who have a positive HCV antibody test. The HCV RNA test is conducted to determine if the patient has an HCV infection.) * Subjects with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen for this trial are eligible * Subjects must have adequately recovered from toxicities related to prior anti-cancer therapy. (Inadequately recovered toxicity is defined as an ongoing \>= grade 2 adverse event \[NCI-CTCAE version 5.0\] that is attributed to prior anti-cancer therapy, with the exception of treatment-related grade 2 alopecia and treatment-related grade 2 peripheral neuropathy, which are both permitted) * Subjects who enroll in this trial must agree to follow the below contraception requirements. Female subjects should immediately inform the investigator if they become pregnant or suspect pregnancy while participating in the trial. Male subjects should immediately inform the investigator if their partner becomes pregnant or suspects pregnancy while they are participating in the trial * Contraception Requirements for Females: Female subjects of reproductive potential must agree to remain abstinent (refrain from heterosexual intercourse) or use a highly effective method of contraception while receiving trial therapy and for 8 months following completion of trial therapy. Women must also not donate eggs during this same period * Contraception Requirements for Males: With a female partner of reproductive potential, males must agree to remain abstinent (refrain from heterosexual intercourse) or use a highly effective method of contraception while receiving trial therapy and for 5 months following completion of trial therapy. Men must also not donate sperm during this same period * With a pregnant female partner, males must remain abstinent (refrain from heterosexual intercourse) or use a condom while receiving trial therapy and for 5 months following completion of trial therapy to avoid exposing the embryo * A female subject is considered to be of reproductive potential (regardless of sexual orientation, having undergone a tubal ligation, or remaining celibate by choice) unless it is documented that she meets either of the following two criteria: * Has reached a postmenopausal state (\>= 12 continuous months of amenorrhea with no identified cause other than menopause) * Has undergone surgical sterilization (i.e., hysterectomy and/or bilateral oophorectomy for removal of uterus and/or ovaries) * Sexual abstinence is permitted only if it is the preferred and usual lifestyle of the subject. Periodic abstinence (e.g., calendar, ovulation, symptothermal, postovulation methods) and withdrawal are not adequate methods of contraception * Highly effective methods of birth control include hormonal birth control (oral, intravaginal, transdermal, implantable, or intrauterine devices \[IUDs\]), IUDs (non-hormonal), male vasectomy, or any double-barrier method (combination of male condom and spermicide with either cap, diaphragm, or sponge) * Females of reproductive potential must agree to undergo a serum pregnancy test (i.e., human chorionic gonadotropin test) prior to start of treatment on day 1 of each cycle, and the result must be negative in order to initiate and continue treatment * Subjects (or subject's legally authorized representative if subject has impaired decision-making capacity) must have the ability to understand and the willingness to sign a written informed consent document for study-specific procedures and treatment, prior to registration on trial Exclusion Criteria: * Subjects must not have a known history of hypersensitivity reactions attributed to compounds of similar chemical or biologic composition to XmAb20717 or any of its excipients * Subjects must not have previously received XmAb20717 * Subjects must not have received prior CTLA4 antibody or PD-1/PD-L1 therapy * Washout from prohibited anti-cancer therapy: Subjects must not currently be receiving anticancer therapy (e.g., chemotherapy, radiation therapy, targeted therapy, investigational anticancer agents, immunotherapy, hormonal therapy, biologics, anti-cancer radionucleotides, other systemic anticancer agents, etc.) * For subjects with HCC, all anticancer therapies (with the exception of surgery) must be discontinued at least 2 weeks prior to the initiation of trial therapy * For subjects with ATC, all anticancer therapies (with the exception of surgery) must be discontinued at least 1 week prior to the initiation of trial therapy * Notes: There is no required washout from prior surgical procedures. Due to the aggressive nature of ATC, wherein the disease progresses extremely rapidly, a 2-week washout from prior anticancer therapy is not indicated for this patient population. A 1-week washout is selected for ATC (due to it being aggressive), while a 2-week washout is selected for HCC (due to it being less aggressive) * Washout from prohibited concomitant medication: Subjects must not have any condition requiring systemic treatment with corticosteroids, prednisone equivalents, or other immunosuppressive medication. Corticosteroids, prednisone equivalents, and other immunosuppressive medications must be discontinued within 14 days prior to first dose of trial therapy (except that inhaled or topical corticosteroids or brief courses of corticosteroids given for prophylaxis of contrast dye allergic response are permitted) * Washout from prohibited concomitant vaccine: Subjects must not have received a live-virus vaccine within 30 days prior to the first dose of trial therapy. Subjects must not have received any other vaccine within 24 hours prior to the first dose of trial therapy. Note that while the majority of seasonal influenza vaccines (e.g., intramuscular route) do not contain live virus, the intranasal Flu-Mist is considered a live attenuated vaccine. * Subjects must not have a known active central nervous system (CNS) metastases and/or carcinomatous meningitis. Participants with previously adequately treated brain metastases may participate, provided they are clinically stable and without requirement of steroid treatment for at least 14 days prior to first dose of trial therapy * Subjects must not have a known or suspected active autoimmune disease (except that subjects are permitted to enroll if they have vitiligo; type 1 diabetes mellitus or residual hypothyroidism due to an autoimmune condition that is treatable with hormone replacement therapy only; psoriasis, atopic dermatitis, or another autoimmune skin condition that is managed without systemic therapy; or arthritis that is managed without systemic therapy beyond oral acetaminophen and non-steroidal anti-inflammatory drugs; or celiac disease that, in the opinion of the investigator, is clinically-controlled) * Subjects must not have received an organ allograft * Female subjects must not be pregnant or intending to conceive from the time of informed consent through 8 months after the last dose of trial treatment. (These subjects are excluded because there is an unknown but potential risk for adverse events to the developing fetus) * It is unknown whether XmAb20717 is excreted in human milk. Since many drugs are excreted in human milk, and due to the potential for serious adverse reactions in the nursing infant, women who are breastfeeding are not eligible for enrollment * Subjects must not have evidence of any serious bacterial (including tuberculosis), viral (including severe acute respiratory syndrome coronavirus 2 \[SARS CoV-2\] and influenza), parasitic, or systemic fungal infections within the 30 days prior to the first dose of trial therapy * NOTE: Tuberculosis, SARS CoV-2 (coronavirus disease 2019 \[COVID-19\]), and influenza testing are not required, unless clinically indicated or mandated by a local health authority * Subjects must not have a history or evidence of any other clinica