Comparing Whole Brain Radiotherapy Using a Technique That Avoids the Hippocampus to Stereotactic Radiosurgery in Patients With Cancer That Has Spread to the Brain and Come Back in Other Areas of the Brain After Earlier Stereotactic Radiosurgery

Inclusion Criteria: * Patients must have developed their distant brain relapse(s) at least 8 weeks after last SRS and within 21 days prior to randomization * Distant brain relapse lesions to be treated must measure =\< 3.0 cm in maximal extent and total volume of distant brain relapses to be treated must measure \< 30 mL on the contrast-enhanced diagnostic MRI brain scan obtained within 21 days prior to randomization * "last SRS" refers to the most recent SRS procedure that the patient received prior to enrollment on this study * Distant brain relapse lesions must be diagnosed on MRI, which will include the following elements: * REQUIRED MRI ELEMENTS * Post gadolinium contrast-enhanced T1-weighted three-dimensional (3D) spoiled gradient (SPGR). Acceptable 3D SPGR sequences include magnetization-prepared 3D gradient recalled echo (GRE) rapid gradient echo (MP-RAGE), turbo field echo (TFE) MRI, BRAVO (brain volume imaging) or 3D fast FE (field echo). The T1-weighted 3D scan should use the smallest possible axial slice thickness, not to exceed 1.5 mm * Pre-contrast T1 weighted imaging (3D imaging sequence strongly encouraged) * A minimum of one axial T2 fluid attenuated inversion recovery (FLAIR) (preferred) or T2 sequence is required. This can be acquired as a 2D or 3D image. If 2D, the images should be obtained in the axial plane * ADDITIONAL RECOMMENDATIONS * Recommendation is that an axial T2 FLAIR (preferred) sequence be performed instead of a T2 sequence * Recommendation is that that pre-contrast 3D T1 be performed with the same parameters as the post-contrast 3D T1 * Recommendation is that imaging be performed on a 3 Tesla (3T) MRI * Recommendation is that the study participants be scanned on the same MRI instrument at each time point * Recommendation is that if additional sequences are obtained, these should meet the criteria outlined in Kaufmann et al., 2020 * If additional sequences are obtained, total imaging time should not exceed 60 minutes * Brain metastasis velocity (BMV) since last SRS must be \>= 4 brain metastases/year * The patient or a legally authorized representative must provide study-specific informed consent prior to study entry * Pathologically (histologically or cytologically) proven diagnosis of small cell cancer, non-small cell lung cancer, melanoma, breast cancer, renal cell carcinoma, or gastrointestinal cancer within 10 years prior to randomization. If the original histologic proof of malignancy is greater than 10 years, then pathological (i.e., more recent) confirmation is required (e.g., from a systemic metastasis or brain metastasis) * Other histologies are not permitted * History and physical examination within 28 days prior to randomization * Age \>= 18 * Karnofsky performance status of \>= 70 within 28 days prior to randomization * Calculated creatinine clearance (CrCl) \>= 30 ml/min (within 28 days prior to randomization) * Blood urea nitrogen (BUN) within 1.5 times the institutional upper limit of normal (ULN) (e.g., if the ULN is 20 mg/dL, then BUN up to 30 mg/dL is permitted) (within 28 days prior to randomization) * Negative urine or serum pregnancy test (in women of childbearing potential) within 14 days prior to randomization Exclusion Criteria: * BMV \>= 4 brain metastases/year at time of any SRS prior to enrollment. * Patients are permitted to have undergone multiple SRS treatments to different brain metastases so long as prior BMV has been less than 4 brain metastases/year * Prior WBRT or prophylactic cranial irradiation * Local relapse of metastasis previously treated with upfront SRS (i.e., relapse outside previously SRS-treated metastases is allowed) * Brain metastases from primary germ cell tumor or lymphoma * Definitive leptomeningeal metastasis * Planned cytotoxic chemotherapy on the same day as SRS or HA-WBRT; concurrent immunotherapy is permitted * Radiographic evidence of enlargement or other architectural distortion of the lateral ventricles, including placement of external ventricular drain or ventriculoperitoneal shunt * Known history of demyelinating disease such as multiple sclerosis * Inability to swallow pills * Contraindication to MR imaging such as non-MR conditional implanted metal devices or unknown metallic foreign bodies, or contraindication to gadolinium contrast administration during MR imaging, such as anaphylactic allergy that cannot be adequately addressed with pre-contrast medications or acute kidney injury * Contraindications to memantine, including: * Allergy, including prior allergic reaction to memantine * Intractable seizures on adequate anticonvulsive therapy-more than 1 seizure per month for the past 2 months * Current use of N-methyl-D-asparatate (NMDA) agonist * Current alcohol or drug abuse, which can exacerbate lethargy/dizziness with memantine * Severe, active co-morbidity defined as follows: * Unstable angina and/or congestive heart failure requiring hospitalization within the last 6 months * Transmural myocardial infarction within the last 6 months * Acute bacterial or fungal infection requiring intravenous antibiotics at the time of randomization * Chronic obstructive pulmonary disease exacerbation or other acute respiratory illness precluding study therapy at the time of randomization * Severe hepatic disease defined as a diagnosis of Child-Pugh class B or C hepatic disease * Renal tubular acidosis or metabolic acidosis * Human immunodeficiency virus (HIV) positive with CD4 count \< 200 cells/microliter. Note that patients who are HIV positive are eligible, provided they are under treatment with highly active antiretroviral therapy (HAART) and have a CD4 count \>= 200 cells/microliter within 30 days prior to randomization. Note also that HIV testing is not required for eligibility for this protocol * Pregnant or lactating women, or women of childbearing potential and men who are sexually active and not willing/able to use medically acceptable forms of contraception; this exclusion is necessary because the medication and radiation involved in this study has unknown effects on the unborn fetus