Testing the Anti-cancer Drug Erdafitinib for Brain Cancers That Have Returned or Progressed Following Treatment

Inclusion Criteria: * Patient must be \>= 18 years of age * Patient must have histologically confirmed IDH-WT gliomas (grade 2-4) as per World Health Organization (WHO) 2016 or 2021 classification * Tumor tissue should be positive for FGFR-TACC gene fusion as per any local next generation sequencing (NGS) (Clinical Laboratory Improvement Act \[CLIA\]-approved) assay described in background section * The disease should be recurrent or progressive glioma after initial anti-tumor treatment with at least 1 line of treatment including surgical resection, radiation therapy and/or chemotherapy * For patients with WHO grade 3 or 4 glioma and progressive disease \< 12 weeks after completion of chemoradiotherapy, progression can be defined by the following set of criteria: * New enhancement outside of the radiation field (beyond the high-dose region or 80% isodose line) * If there is unequivocal evidence of viable tumor on histopathologic sampling (e.g., solid tumor areas. i.e., \> 70% tumor cell nuclei in areas), high or progressive increase in Ki-67 proliferation index compared with prior biopsy, or evidence for histologic progression or increased anaplasia in tumor) * For patients with WHO grade 3 or 4 glioma and progressive disease \>= 12 weeks after completion of chemoradiotherapy, progression can be defined by the following set of criteria: * New contrast-enhancing lesion outside of radiation field on decreasing, stable, or increasing doses of corticosteroids * Increase by \>= 25% in the sum of the products of perpendicular diameters between the first post-radiotherapy scan, or a subsequent scan with smaller tumor size, and the scan at 12 weeks or later on stable or increasing doses of corticosteroids * For patients receiving antiangiogenic therapy, significant increase in T2/fluid attenuated inversion recovery (FLAIR) non-enhancing lesion may also be considered progressive disease. The increased T2/FLAIR must have occurred with the patient on stable or increasing doses of corticosteroids compared with baseline scan or best response after initiation of therapy and not be a result of comorbid events (e.g., effects of radiation therapy, demyelination, ischemic injury, infection, seizures, postoperative changes, or other treatment effects) * For patients with WHO grade 2 glioma progression is defined by any one of the following: * Development of new lesions or increase of enhancement (radiological evidence of malignant transformation) * A 25% increase of the T2 or FLAIR non-enhancing lesion on stable or increasing doses of corticosteroids compared with baseline scan or best response after initiation of therapy, not attributable to radiation effect or to comorbid events * There must be measurable disease (enhancing or non-enhancing as per Response Assessment in Neuro-Oncology \[RANO\] or RANO-low-grade glioma \[LGG\] criteria), as evaluated on pre-treatment MRI * Patient understands the procedures and investigational nature of the study drug and agrees to comply with study requirements by providing written informed consent * Patient must have Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 (or Karnofsky \>= 60%) * Absolute neutrophil count \>= 1000/uL * Hemoglobin \> 8 g/dL (Patients are allowed to be transfused to this level) * Platelets \>= 100 x 10\^9/L * Serum total bilirubin =\< 1.5 x upper limit of normal (ULN), unless considered due to Gilbert's disease or disease involvement following approval by the medical monitor * Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamic pyruvic transaminase \[SGPT\]) =\< 3 x institutional ULN * Creatinine clearance \> 30 mL/min (patients with mild or moderate renal impairment) based on the Cockroft-Gault glomerular filtration rate (GFR) estimation * Patient must have normal serum phosphate level as per local laboratory parameters. (Medical management allowed) * Patient must be recovered from any clinically relevant toxic effects of any prior surgery, radiotherapy, or chemotherapy treatment with temozolomide * Human immunodeficiency virus (HIV)-infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial * For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated * Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load * Patients should be New York Heart Association Functional Classification class 2B or better * The effects of erdafitinib on the developing human fetus are unknown. For this reason and because FGFR inhibitors are known to be teratogenic, women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry, for the duration of study participation, and one month after completion of erdafitinib administration. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. Men treated or enrolled on this protocol must also agree to use adequate contraception prior to the study, for the duration of study participation, and one month after completion of erdafitinib administration * Ability to understand and the willingness to sign a written informed consent document. Legally authorized representatives may sign and give informed consent on behalf of study participants Exclusion Criteria: * Patients who are receiving any other investigational agents * History of allergic reactions attributed to compounds of similar chemical or biologic composition to erdafitinib * Patients requiring any medications or substances that are moderate CYP2C9 inducers and strong CYP3A4 inducers are ineligible. Because the lists of these agents are constantly changing, it is important to regularly consult a frequently-updated medical reference. As part of the enrollment/informed consent procedures, the patient will be counseled on the risk of interactions with other agents, and what to do if new medications need to be prescribed or if the patient is considering a new over-the-counter medicine or herbal product * Patients with uncontrolled intercurrent illness * Pregnant women are excluded from this study because erdafitinib is an FGFR inhibitor agent with the potential for teratogenic or abortifacient effects. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with erdafitinib, breastfeeding should be discontinued if the mother is treated with erdafitinib * Current central serous retinopathy (CSR) or retinal pigment epithelial detachment of any grade * Corrected QT interval (QTc) prolongation as confirmed by electrocardiography (ECG) at screening (Fridericia; QTc \> 480 milliseconds) * Patients who have previously received FGFR inhibitors