Nivolumab and Multi-fraction Stereotactic Radiosurgery With or Without Ipilimumab in Treating Patients With Recurrent Grade II-III Meningioma

Inclusion Criteria: * Patients must have histologically confirmed World Health Organization (WHO) grade II-III meningioma which has relapsed after prior radiation therapy with radiologically progressive or recurrent disease * Patients must have measurable disease, defined as at least 1 lesion that can be accurately measured in at least one dimension as \>= 1 cm on brain magnetic resonance imaging (MRI) but with the maximum dimension =\< 5 cm OR gross tumor volume \< 20 cm\^3. All the relapsed disease would need to be eligible to be treated with reirradiation * Patients must have at least one prior surgery with available archival formalin-fixed paraffin-embedded (FFPE) tumor blocks of the initial or recurrent meningioma. If there are multiple tumor blocks from multiple surgeries, the most recent tumor block (and ideally of the relapsed tumor after initial radiation therapy) should be submitted. If a tumor block is not available, an alternative of 20-30 unstained slides may be submitted instead. Annotation regarding whether the tumor block is before or after initial radiation therapy should be provided * Prior initial radiation therapy may include external beam radiation or radiosurgery, or combination of both. However, the total dose of prior radiation exposure to the site of recurrent tumor (for consideration of re-irradiation) cannot be more than 70 Gy. The duration since the previous radiation exposure to the site of reirradiation need to be at least 6 months * Age \>= 18 years * Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 (Karnofsky \>= 60%) * Leukocytes \>= 3,000/mcL * Absolute neutrophil count \>= 1,500/mcL * Platelets \>= 100,000/mcL * Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN) * Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x institutional ULN * Serum creatinine =\< 1.5 x institutional ULN OR glomerular filtration rate (GFR) \>= 30 mL/min/1.73 m\^2 for patients with creatinine levels above 1.5 x institutional normal * The effects of nivolumab and/or ipilimumab on the developing human fetus are unknown. For this reason and because radiation therapy is known to be teratogenic, women of childbearing potential (WOCBP) and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation. WOCBP should use an adequate method to avoid pregnancy for 5 months after the last dose of investigational drug. Women of childbearing potential must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of human chorionic gonadotropin \[HCG\]) within 24 hours prior to the start of nivolumab. Women must not be breastfeeding. Men who are sexually active with WOCBP must use any contraceptive method with a failure rate of less than 1% per year. Men receiving nivolumab and who are sexually active with WOCBP will be instructed to adhere to contraception for a period of 7 months after the last dose of investigational product. Women who are not of childbearing potential (i.e., who are postmenopausal or surgically sterile as well as azoospermic men) do not require contraception * Women of childbearing potential (WOCBP) is defined as any female who has experienced menarche and who has not undergone surgical sterilization (hysterectomy or bilateral oophorectomy) or who is not postmenopausal. Menopause is defined clinically as 12 months of amenorrhea in a woman over 45 in the absence of other biological or physiological causes. In addition, women under the age of 55 must have a documented serum follicle stimulating hormone (FSH) level less than 40 mIU/mL * WOCBP receiving nivolumab will be instructed to adhere to contraception for a period of 5 months after the last dose of investigational product. Men receiving nivolumab and who are sexually active with WOCBP will be instructed to adhere to contraception for a period of 7 months after the last dose of investigational product. These durations have been calculated using the upper limit of the half-life for nivolumab (25 days) and are based on the protocol requirement that WOCBP use contraception for 5 half-lives plus 30 days and men who are sexually active with WOCBP use contraception for 5 half-lives plus 100 days * Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately * Ability to understand and the willingness to sign a written informed consent document or, if decision-making capacity is impaired, consent of a legally authorized representative (e.g., spouse, adult child, live-in caretaker). Exclusion Criteria: * Patients who have had chemotherapy within 4 weeks (6 weeks for nitrosoureas or mitomycin C) prior to entering the study * Patients who have had radiation therapy (to the site of reirradiation) within 6 months prior to entering the study * Patients who have not recovered from adverse events due to prior anti-cancer therapy (i.e., have residual toxicities \> grade 1); however, alopecia, sensory neuropathy =\< grade 2, or other =\< grade 2 not constituting a safety risk based on the investigator's judgment are acceptable * Patients who are receiving any other investigational agents * Patients who have previous treatment with an anti-PD-1, anti-PD-L1, anti-PD-L2, or anti-CTLA-4 antibody, or any other antibody or drug specifically targeting T-cell co-stimulation or immune checkpoint pathways * History of allergic reactions attributed to compounds of similar chemical or biologic composition to nivolumab and/or ipilimumab * History of severe hypersensitivity reaction to any monoclonal antibody * Current use of immunosuppressive medication (EXCEPT for the following: Intranasal, inhaled, topical steroids, or local steroid injection \[e.g. intra-articular injection\]; systemic corticosteroids at doses =\< 4 mg/day of dexamethasone or equivalent; steroids as premedication for hypersensitivity reactions \[e.g. computed tomography (CT) scan premedication\]) * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements. However, patients with human immunodeficiency virus (HIV) on stable therapy with minimal viral loads and patients with hepatitis B and hepatitis C who have received treatment with minimal viral loads will be eligible * Pregnant women are excluded from this study because radiation therapy is teratogenic and that the effects of nivolumab and/or ipilimumab on the developing human fetus are unknown. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with nivolumab and/or ipilimumab, breastfeeding should be discontinued if the mother is treated with nivolumab and/or ipilimumab * Patients with active autoimmune disease or history of autoimmune disease that might recur, which may affect vital organ function or require immune suppressive treatment including systemic corticosteroids, should be excluded. These include but are not limited to patients with a history of immune related neurologic disease, multiple sclerosis, autoimmune (demyelinating) neuropathy, Guillain-Barre syndrome, myasthenia gravis; systemic autoimmune disease such as systemic lupus erythematosus (SLE), connective tissue diseases, scleroderma, inflammatory bowel disease (IBD), Crohn's, ulcerative colitis, hepatitis; and patients with a history of toxic epidermal necrolysis (TEN), Stevens-Johnson syndrome, or phospholipid syndrome should be excluded because of the risk of recurrence or exacerbation of disease. Patients with vitiligo, endocrine deficiencies including thyroiditis managed with replacement hormones including physiologic corticosteroids are eligible. Patients with rheumatoid arthritis and other arthropathies, Sjogren's syndrome and psoriasis controlled with topical medication and patients with positive serology, such as antinuclear antibodies (ANA), anti-thyroid antibodies should be evaluated for the presence of target organ involvement and potential need for systemic treatment but should otherwise be eligible * Patients are permitted to enroll if they have vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger (precipitating event) * Prior organ transplantation including allogeneic stem cell transplantation * Other severe acute or chronic medical conditions including immune colitis, inflammatory bowel disease, immune pneumonitis, pulmonary fibrosis, or psychiatric conditions including recent (within the past year) or active suicidal ideation or behavior, or laboratory abnormalities that may increase the risk associated with study participation or study treatment administration or may interfere with the interpretation of study results and, in the judgment of the investigator, would make the patient inappropriate for entry into this study * Live vaccination within 4 weeks of the first dose of nivolumab and while on trial is prohibited except for administration of inactivated vaccines