Testing the Combination of Two Immunotherapy Drugs (Nivolumab and Ipilimumab) in Children, Adolescent, and Young Adult Patients With Relapsed/Refractory Cancers That Have an Increased Number of Genetic Changes, The 3CI Study

Inclusion Criteria: * PART 1: Patients must have histologically or cytologically confirmed malignancy at the time of initial diagnosis, relapse, or recurrence. Patients must have recurrent or refractory cancer for which standard curative or palliative measures do not exist or are no longer effective * Patients with multiple concurrent and/or sequential neoplasms are eligible * Patients with central nervous system (CNS) tumors are eligible, except those with diffuse intrinsic pontine glioma * Patients with lymphoma are eligible; patients with leukemia are excluded * Chemotherapy-naive patients are eligible in cases where first-line therapy does not include chemotherapy (e.g., surgery only for ependymoma management) * PART 1: Patients must have evidence of one or more of the following criteria in current or previous tumor: * Microsatellite instability (MSI-H) * Mutation causing functional loss of mismatch repair gene expression (MLH1, MSH2, MSH6, PMS2, EPCAM, MSH3) * Hypermutation in any tumor (including primary malignancy for patients with relapse or previous cancer diagnoses) * Functional mutation of POLE or POLD1 genes * A syndrome linked to hypermutant cancer predisposition such as congenital mismatch repair deficiency (CMMRD), Lynch syndrome, or xeroderma pigmentosum (XP) is also permitted * Other factors or sequencing evidence not listed above but which may be predictive of hypermutant cancer may be permitted after discussion with the protocol principal investigator * PART 1: A tumor tissue specimen must be provided for molecular profiling, including TMB analysis. The specimen may be archival or prospective, from a medically necessary surgery, biopsy, or excision. Tissue will not be obtained solely for this trial. A specimen from the time of most recent relapse/progression is preferred, but not mandatory * Tissue is preferred. However, if necessary, previously extracted DNA may be used with the approval of the protocol principal investigator if extracted in a clinically certified laboratory and prepared in an Foundation Medicine Inc. (FMI), TMB assay-compatible manner * PART 1: All patients and/or their parents or legally authorized representatives must have the ability to understand and the willingness to sign a written informed consent. Assent, where appropriate, will be obtained according to local policy. Patients with impaired decision-making capacity will not be excluded * PART 2: Patients must have histologically or cytologically confirmed malignancy at the time of initial diagnosis, relapse, or recurrence. Patients must have recurrent or refractory cancer for which standard curative or palliative measures do not exist or are no longer effective * Patients with multiple concurrent and/or sequential neoplasms are eligible * Patients with CNS tumors are eligible, except those with diffuse intrinsic pontine glioma or bulky tumors * Patients with lymphoma are eligible (provided other criteria, such as bone marrow function, are met); patients with leukemia are excluded * Chemotherapy-naive patients are eligible in cases where first-line therapy does not include chemotherapy (e.g., surgery only for ependymoma management) * PART 2: Patients must have measurable disease * Patients with neuroblastoma without measurable soft tissue but with iobenguane (MIBG) avid disease are eligible * Patients with bone marrow only disease are excluded * PART 2: Patients must have confirmation of cancer with a TMB of \>= 10 mutations (mut)/megabase (Mb) as determined by an next generation sequencing (NGS) targeted cancer gene panel performed by Foundation Medicine Inc. (FMI). Proof of TMB eligibility can be from Part 1 participation or a previously acquired FMI report * PART 2: Patients must have recovered from the acute toxic effects of all prior anti-cancer therapies (with the exception of alopecia and lymphopenia) * Previous treatment with nivolumab and/or other anti-PD-1/PD-L1 inhibitors is permitted * Previous treatment with ipilimumab and/or other anti-CTLA-4 inhibitors is permitted * Previous treatment with combined anti-PD-1/PD-L1 and anti-CTLA-4 inhibitors is not be permitted * PART 2: The following time periods apply for prior therapy. Patients must have: * Cytotoxic chemotherapy: At least 21 days prior to treatment initiation from the last dose of cytotoxic or myelosuppressive chemotherapy; at least 42 days if prior nitrosourea (such as lomustine, CCNU) * Hematopoietic growth factors: At least 7 days prior to treatment initiation from the last dose of short-acting growth factor; at least 14 days for long-acting * Anti-cancer agents not known to be myelosuppressive: At least 7 days prior to treatment initiation from the last dose * Interleukins, interferons, and cytokines (other than hematopoietic growth factors): At least 21 days prior to treatment initiation from the last dose * Antibodies: At least 21 days prior to treatment initiation from the last dose and toxicity related to prior antibody therapy must be recovered to grade =\< 1 * Radiotherapy: At least 14 days prior to treatment initiation from local radiotherapy; at least 150 days from total body irradiation (TBI), craniospinal radiotherapy, or radiation to \>= 50% of the pelvis; at least 42 days from other substantial bone marrow radiation * Radiopharmaceutical therapy (e.g., 131I-MIBG): At least 42 days prior to treatment initiation from systemically administered radiopharmaceutical therapy * Autologous stem cell infusion including boost infusion: At least 42 days prior treatment initiation * Cellular therapy: At least 42 days prior to treatment initiation from any type of cellular therapy * PART 2: Lansky play score \>= 50 if =\<16 years of age; Karnofsky performance scale \>= 50 if =\< 16 years of age. Patients unable to walk due to paralysis but who are using a wheelchair will be considered ambulatory for the purpose of assessing performance status * PART 2: Peripheral absolute neutrophil count (ANC) \>= 750/mm\^3 (0.75 x 10\^9/L) * PART 2: Platelet count \>= 75,000/mm\^3 (75 x 10\^9/L), transfusion independent, defined as not receiving platelet transfusions at least 7 days prior to treatment initiation * PART 2: Creatinine clearance or radioisotope glomerular filtration rate (GFR) \>= 60 mL/min/1.73 m\^2; OR serum creatinine based on age/gender as follows: * Age: 1 to \< 2 years; Maximum serum creatinine: 0.6 mg/dL; 53 umol/L (male); 0.6 mg/dL; 53 umol/L (female) * Age: 2 to \< 6 years; Maximum serum creatinine: 0.8 mg/dL; 71 umol/L (male); 0.8 mg/dL; 71 umol/L (female) * Age: 6 to \< 10 years; Maximum serum creatinine: 1 mg/dL; 88 umol/L (male); 1 mg/dL; 88 umol/L (female) * Age: 10 to \< 13 years; Maximum serum creatinine: 1.2 mg/dL; 106 umol/L (male); 1.2 mg/dL; 106 umol/L (female) * Age: 13 to \< 16 years; Maximum serum creatinine: 1.5 mg/dL; 133 umol/L (male); 1.4 mg/dL; 124 umol/L (female) * Age: \>= 16 years; Maximum serum creatinine: 1.7 mg/dL; 150 umol/L (male); 1.4 mg/dL; 124 umol/L (female) * PART 2: Bilirubin (sum of conjugated and unconjugated) =\< 1.5 x upper limit of normal (ULN) for age * PART 2: Alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 135 U/L (i.e., 3 x ULN). For the purposes of this study, the ULN for ALT (SGPT) is 45 U/L * PART 2: No evidence of dyspnea at rest, no exercise intolerance due to pulmonary insufficiency, and pulse oximetry \>= 92% while breathing room air * PART 2: No signs or symptoms of heart failure in a patient who has no history of congestive heart failure, no prior exposure to cardiotoxic drugs, and no radiotherapy to the heart; OR shortening fraction of \>= 27% or ejection fraction of \>= 50% by echocardiogram * PART 2: Serum lipase =\< ULN at screening * PART 2: Patients with treated CNS metastasis are eligible if there is no evidence of progression for at least 4 weeks after CNS-directed treatment as ascertained by clinical examination and brain imaging (magnetic resonance imaging \[MRI\] or computed tomography \[CT\] scan) during screening * Patients with new or progressive CNS metastasis (active metastasis) or leptomeningeal disease are eligible if the treating physician determines that immediate CNS-directed treatment is not required and is unlikely to be required for at least 6 weeks after treatment initiation, and a risk-benefit analysis (discussion) by the patient and investigator favors participation in the trial * PART 2: Human immunodeficiency virus (HIV): Infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months prior to treatment initiation are eligible * Note: Routine screening for HIV status prior to enrollment is not required * PART 2: Hepatitis B virus (HBV): Patients with evidence of chronic infection with undetectable viral load are eligible. Suppressive therapy, if indicated, is allowed * Note: Routine screening for HBV status prior to enrollment is not required * PART 2: Hepatitis C virus (HCV): Infected patients currently on treatment with undetectable viral load are eligible. Patients with history of infection must have been treated and cured * Note: Routine screening for HCV status prior to enrollment is not required * PART 2: Patients must provide a pre-treatment tumor tissue specimen (baseline sample) for correlative exploratory biology studies. The specimen may be archival or prospective, from a medically necessary surgery, biopsy, or excision. Tissue will not be obtained solely for this trial. A specimen from the time of most recent relapse/progression is preferred, but not mandatory. Submission of a representative sample from all available lesions (archival and prospective) is strongly encouraged. If available, residual tissue from Part 1 may be used to fulfill the baseline tissue sample requirement; however additional tissue may be required if residual tissue is insufficient * PART 2: All patients and/or their parents or legally authorized representatives must have the ability to understand and the willingness to sign a written informed consent. Assent, where appropriate, will be obtained according to local policy. Patients with impaired decision-making capacity will not be excluded Exclusion Criteria: * PART 1: Patients with history of autoimmune disease * PART 1: Patients with history of interstitial lung disease or pneumonitis are not eligible * PART 1: Patients who have received solid organ transplant or allogenic stem cell transplant are not eligible * PART 1: Patients who have been previously treated with a combination of anti-PD-1/PD-L1 and anti-CTLA-4 inhibitors are not eligible * PART 2: Patients requiring systemic corticosteroids or other forms of immunosuppressive therapy within 7 days prior to treatment initiation are not eligible * Following treatment initiation, systemic corticosteroids or other forms of immunosuppressive therapy are permitted if administered for the treatment of toxicity, tumor flare, or pseudo-progression and can be tapered. In most cases study treatment must be held until the dose is tapered to 10 mg/day prednisone or equivalent. The protocol principal investigator must be consulted prior to resuming treatment * Physiologic corticosteroids up to 5 mg/day prednisone or equivalent are permitted * Topical, ocular, intra-articular, intra-nasal, inhaled corticosteroids are permitted * Patients with CNS tumors receiving steroids for intracranial mass effect must be able to discontinue these at least 7 days prior to treatment initiation * PART 2: Patients who are receiving other anticancer agent(s) are not eligible * PART 2: Patients who are receiving or have received any other investigational agent(s) within 14 days prior to treatment initiation are not eligible * PART 2: Patients with CNS tumors with any of the following characteristics on imaging are not eligible: * Tumor with any evidence of uncal herniation or mass effect leading to severe midline shift * Tumor \> 6 cm in single maximal dimension * Tumor that in the opinion of the investigator shows significant mass effect * PART 2: Patients with uncontrolled intercurrent illness/condition that would limit compliance with the study requirements are not eligible. This includes, but is not limited to, ongoing active infection, symptomatic congestive heart failure (New York Heart Association class III or IV), unstable angina pectoris, cardiac arrhythmia, psychiatric illness/social situations * PART 2: The study agents have the potential for teratogenic or abortifacient effects. Females of reproductive potential must have a negative serum pregnancy test within 72 hours prior to treatment initiation. Additional pregnancy tests (serum or urine) should be obtained during study participation in accordance with local standards and guidelines * Females of reproductive potential may not participate unless they have agreed to use an effective method of contraception (hormonal or barrier method of birth control; abstinence) prior to study entry, for the duration of treatment, and as follows: * A period of 5 months after the last dose of nivolumab * A period of 3 months after the last dose of ipilimumab * Should a female patient become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform the investigator immediately * Due to the unknown but potential risk for adverse events (AEs) in nursing infants secondary to treatment of the mother with the study agents, breastfeeding must be discontinued if the mother is treated on study * Males will not be required to use contraceptive measures * Note: Females of reproductive potential are defined as those who are past the onset of menarche and are not surgically sterile (i.e., bilateral salpingectomy, bilateral oophorectomy, complete hysterectomy) * PART 2: Patients with history of autoimmune disease (such as autoimmune thyroid disease or inflammatory bowel disease) that has required systemic treatment within 2 years prior to treatment initiation are not eligible * Asymptomatic laboratory abnormalities (e.g., antinuclear antibody \[ANA\], rheumatoid factor, altered thyroid studies) are permitted in the absence of an autoimmune disorder diagnosis * Atopy-related conditions (e.g., asthma, allergic rhinitis, atopic dermatitis) are permitted * Replacement therapy (e.g., thyroxine, insulin, physiological corticosteroid replacement therapy) is not considered a form of systemic treatment * PART 2: Patients with history of interstitial lung disease or pneumonitis are not eligible * PART 2: Patients who have received solid organ transplant or allogenic stem cell transplant are not eligible * PART 2: Patients with previous grade 4 life-threatening reaction or other adverse reaction that in the opinion of the investigator would preclude