Pomalidomide in Treating Younger Patients With Recurrent, Progressive, or Refractory Central Nervous System Tumors

Inclusion Criteria: * Patients must have received standard therapy (or generally accepted upfront therapy if no standard exists) and have no known curative therapy * Patients with a histologically confirmed diagnosis of a primary CNS tumor that is recurrent, progressive or refractory to standard therapy; refractory disease will be defined as the presence of persistent abnormality on conventional magnetic resonance imaging (MRI) imaging that is further distinguished by histology (biopsy or sample of lesion) or advanced imaging, OR as determined by the treating physician and discussed with the primary investigator prior to enrollment; all tumors must have histological verification at either the time of diagnosis or recurrence except for patients with diffuse intrinsic brain stem tumors or optic pathway gliomas; patients with neurofibromatosis type-I (NF-1) associated CNS tumors are eligible if they meet all other eligibility criteria * Patients must have evaluable disease on MRI imaging * Patients must have body surface area (BSA) \> 0.55 m\^2 at the time of enrollment * In the event of de-escalation from dose level 1 to dose level 0, patients with BSAs \< 0.67 m\^2 are not eligible * Patients must have recovered from clinically significant, acute, treatment-related toxicities of prior therapies; for those acute baseline adverse events attributable to prior therapy, recovery is defined as a toxicity grade =\< 2, using Common Terminology Criteria for Adverse Events (CTCAE) version (v) 4.0, unless otherwise specified in the inclusion and exclusion criteria * Agents that potentially fit into more than one category or do not clearly fit into any category listed above should be discussed with the study principal investigator (PI) prior to enrollment * Patients must have received their last dose of known myelosuppressive anticancer therapy greater than 28 days prior to study enrollment or \> 42 days if nitrosourea * Patients must have received their last dose of any other investigational agent greater than 28 days prior to enrollment (with exception of fluorothymidine F-18 \[FLT\]) * Patients must have received their last dose of any other biologic agent greater than 7 days prior to enrollment * Patients must have received their last dose of any other biologic agent greater than 7 days prior to enrollment * For agents that have known adverse events occurring beyond 7 days after administration, this period must be extended beyond the time during which adverse events are known to occur and discussed with the PI * Monoclonal antibody treatment and agents with known prolonged half-lives: at least three half-lives must have elapsed prior to enrollment * Immunomodulatory therapy: greater than 28 days must have elapsed since last dose of an immune modulating agent, including vaccine therapy * Administration of the radioisotope, 18-FLT, which is being concurrently investigated on an imaging study within the Pediatric Brain Tumor Consortium (PBTC), is allowed \> 72 hours prior to initiation of pomalidomide on this study; any adverse events related to the FLT must have resolved completely * Patients must have had their last fraction of: * Craniospinal irradiation, total body irradiation (TBI), or \>= 50% radiation of pelvis \> 3 months prior to enrollment * Focal irradiation \> 6 weeks prior to enrollment * Local palliative radiation therapy (XRT) (small port) \>= 4 weeks * Patient must be: * \>= 6 months since allogeneic bone marrow transplant prior to enrollment * \>= 3 months since autologous bone marrow/stem cell prior to enrollment * \>= 3 months since stem cell transplant or rescue without TBI with no graft vs. host disease prior to enrollment * No graft versus host disease * Patients on anticonvulsant therapy may continue these at the discretion of their treating physician; however, it is recommended that anticonvulsant levels be checked periodically as clinically indicated if possible * Patients on alternative supplements should strongly be encouraged to discontinue them prior to enrollment; if they opt to continue, they may enroll on study as long as they have been receiving the supplement for at least 30 days, there is NO evidence of hepatic, renal or other organ dysfunction, administration is approved by the PI, and administration is documented in the study diary * Patients must be on a stable or decreasing dose of corticosteroids for 5 days prior to enrollment; patient may be taking therapeutic doses of steroids during the initial dose escalations and prior to defining an RP2D; this should be recorded in the database; once the RP2D has been established, enrollment may be limited based on steroid use;\*physiologic replacement doses will be defined on this protocol as no more than 0.75 mg/m\^2/day of dexamethasone or equivalent of steroids; doses higher than this will be considered therapeutic * All races and ethnic groups are eligible for this study * Patients should have no significant worsening in clinical status for a minimum of 2 days prior to enrollment * Patients must be able to swallow whole capsules * Patients should undergo a repeat MRI prior to enrollment if there is a significant worsening or new neurologic symptoms in the interval between the eligibility scan and start of protocol therapy * The repeat scan will act as a new baseline and the eligibility scan for these patients * Karnofsky performance scale (KPS for \> 16 years of age) or Lansky performance score (LPS for =\< 16 years of age) assessed within 14 days of enrollment must be \>= 50 * Absolute neutrophil count \>= 1,000/mm\^3 * Platelets \>= 100,000/mm\^3 (unsupported, defined as no platelet transfusion within 7 days and recovery from nadir) * Hemoglobin \>= 8 g/dL (may receive transfusions) * Total bilirubin =\< 1.5 times institutional upper limit of normal (ULN) * Alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 3 x institutional upper limit of normal * Albumin \>= 3 g/dL * Serum creatinine based on age/gender as noted; patients that do not meet the criteria below but who have a 24-hour creatinine clearance or glomerular filtration rate (GFR) (radioisotope or iothalamate) \>= 70 ml/min/1.73 m\^2 are eligible * Serum creatinine for age/gender * 3 to \< 6 years: 0.8 mg/dL * 6 to \< 10 years: 1 mg/dL * 10 to \< 13 years: 1.2 mg/dL * 13 to \< 16 years: 1.5 mg/dL (male) and 1.4 mg/dL (female) * \>= 16 years: 1.7 mg/dL (male) and 1.4 mg/dL (female) * Oxygen saturation as measured by pulse oximetry must be \>= 93% on room air * Patients must be off all colony-forming growth factor(s) for at least 1 week prior to enrollment (i.e. filgrastim, sargramostim); two weeks must have elapsed if patients received polyethylene glycol (PEG) formulations * Pregnant or breast-feeding patients are excluded; female patients of childbearing potential must have a negative serum or urine pregnancy test at the time of enrollment; in addition, female patients of childbearing potential must have negative pregnancy tests within 10 - 14 days prior to starting pomalidomide (can use enrollment pregnancy test if within the 10-14 day limit) AND again within 24 hours prior to initiation of pomalidomide; this protocol defines the following childbearing potential risk categories as: * Female child/young adult of childbearing potential as a female who has: * Achieved menarche and/or breast development, in Tanner stage 2 or greater * Has not undergone a hysterectomy or bilateral oophorectomy, or has not been naturally postmenopausal for at least 24 consecutive months (i.e., has had menses at any time in the preceding 24 consecutive months) * Note: amenorrhea following cancer therapy does not rule out childbearing potential * Patients of childbearing or child fathering potential must use medically acceptable form(s) of birth control as stated within the pomalidomide Pregnancy Risk Minimization Plan, which includes abstinence, while being treated on this study; true abstinence is acceptable when this is in line with the preferred and usual lifestyle of the patient; periodic abstinence (e.g., calendar, ovulation, symptothermal, post-ovulation methods) and withdrawal are not acceptable methods of contraception * Female patients of childbearing potential agree to and will have had effective contraception without interruption for 28 days before starting pomalidomide * The patient or parent/guardian is able to understand the consent and is willing to sign a written informed consent document according to institutional guidelines Exclusion Criteria: * Patients with any clinically significant unrelated systemic illness (e.g., serious infections or significant cardiac, pulmonary, hepatic or other organ dysfunction), that in the opinion of the investigator would compromise the patient's ability to tolerate protocol therapy, put them at additional risk for toxicity or would interfere with the study procedures or results * Patients with a history of any other malignancy will not be eligible * Patients with radiation-associated gliomas will not be eligible * Patients with a history of non-central line related thrombosis, more than one prior central-line related thrombosis, or known coagulopathy will not be eligible; patients with a first degree family member with a known coagulopathy will be excluded, and therefore, obtaining a family history is essential when possible; patients actively on anticoagulation therapy are not eligible * Patients with a prior history of serious allergic reactions associated with thalidomide or lenalidomide * Patients who are receiving any other anti-cancer or investigational drug therapy are excluded * Patients taking a known moderate to potent inhibitor of CYP1A2 are excluded; pomalidomide is primarily metabolized by CYP1A2 and CYP3A; pomalidomide is also a substrate for permeability (P)-glycoprotein (P-gp) * Patients who in the opinion of the investigator are unwilling or unable to return for required follow-up visits or obtain follow-up studies required to assess toxicity to therapy or to adhere to drug administration plan, other study procedures, and study restrictions * Patients who have received pomalidomide in the past are not eligible; patients who have prior treatment with other immunomodulatory drugs (IMiDs) (thalidomide, lenalidomide) ARE eligible if they meet all other eligibility criteria and did not have "significant toxicity" associated with lenalidomide or thalidomide use; a "significant" toxicity will be defined as one that required a dose reduction or discontinuation due to toxicity; please discuss any questions with the PI