Nivolumab With or Without Ipilimumab in Treating Younger Patients With Recurrent or Refractory Solid Tumors or Sarcomas

Inclusion Criteria: * Parts A \& C: patients must be \>= 12 months and \< 18 years of age at the time of study enrollment * Parts B1-B6, B8, D1-D6, E3, E4: patients must be \>= 12 months and =\< 30 years of age at the time of study enrollment * Part B7: patients must be \>= 12 months and \< 18 years of age at the time of study enrollment * Patients must have had histologic verification of malignancy at original diagnosis or relapse * Parts A \& C: patients with recurrent or refractory solid tumors, without central nervous system (CNS) tumors or known CNS metastases, are eligible; note: CNS imaging for patients without a known history of CNS disease is only required if clinically indicated * Part B1: patients with relapsed or refractory neuroblastoma * Part B2: patients with relapsed or refractory osteosarcoma * Part B3: patients with relapsed or refractory rhabdomyosarcoma * Part B4: patients with relapsed or refractory Ewing sarcoma or peripheral primitive neuroectodermal tumor (PNET) * Part B5: patients with relapsed or refractory Hodgkin lymphoma * Part B6: patients with relapsed or refractory non-Hodgkin lymphoma * Part B7: patients with unresectable melanoma or metastatic melanoma or relapsed melanoma or refractory melanoma * Part B8: Patients with relapsed or refractory neuroblastoma (MIBG evaluable disease without Response Evaluation Criteria in Solid Tumors \[RECIST\] measurable lesion) * Once the dose-escalation portion of Part A is completed, cohorts that are open concurrently for eligible patients (including Parts B and C and potential pharmacokinetic \[PK\] expansion cohorts) may be selected at the treating physician's discretion pending slot availability; in the event a disease group cohort in Part B is completed after the initial stage of Simon's optimal two-stage design, for selected disease cohorts, a corresponding cohort in the same disease group for select disease types will be open in Part D: * Part D1: Patients with relapsed or refractory neuroblastoma * Part D2: Patients with relapsed or refractory osteosarcoma * Part D3: Patients with relapsed or refractory rhabdomyosarcoma * Part D4: Patients with relapsed or refractory Ewing sarcoma or peripheral PNET * Part D5: Patients with relapsed or refractory non-Hodgkin lymphoma * Part D6: Patients with relapsed or refractory neuroblastoma (MIBG evaluable disease without RECIST measurable lesion) * Part E3: Patients with relapsed or refractory rhabdomyosarcoma * Part E4: Patients with relapsed or refractory Ewing sarcoma or peripheral PNET * Parts A \& C: patients must have either measurable or evaluable disease * Parts B, D \& E: patients must have measurable disease for Parts B1-B6, D1-D5, E3 and E4; melanoma patients in Part B7 must have either measurable or evaluable disease; neuroblastoma patients in Parts B8 and D6 must be evaluable for MIBG response without evidence of RECIST measurable lesions * Patient's current disease state must be one for which there is no known curative therapy or therapy proven to prolong survival with an acceptable quality of life * Karnofsky \>= 50% for patients \> 16 years of age and Lansky \>= 60 for patients =\< 16 years of age; patients who are unable to walk because of paralysis, but who are up in a wheelchair, will be considered ambulatory for the purpose of assessing the performance score * Patients must have fully recovered from the acute toxic effects of all prior anti-cancer therapy and must meet the following minimum duration from prior anti-cancer directed therapy prior to enrollment; if after the required timeframe, the defined eligibility criteria are met, e.g. blood count criteria, the patient is considered to have recovered adequately * Cytotoxic chemotherapy or other anti-cancer agents known to be myelosuppressive * At least 21 days after the last dose of cytotoxic or myelosuppressive chemotherapy (42 days if prior nitrosourea) * Hematopoietic growth factors: At least 14 days after the last dose of a long-acting growth factor (e.g. pegfilgrastim) or 7 days for short-acting growth factor; for agents that have known adverse events occurring beyond 7 days after administration, this period must be extended beyond the time during which adverse events are known to occur; the duration of this interval must be discussed with the study chair * Anti-cancer agents not known to be myelosuppressive (e.g. not associated with reduced platelet or absolute neutrophil count \[ANC\] counts): At least 7 days after the last dose of agent * Interleukins, interferons and cytokines (other than hematopoietic growth factors): \>= 21 days after the completion of interleukins, interferon or cytokines (other than hematopoietic growth factors) * Antibodies: \>= 21 days must have elapsed from infusion of last dose of antibody, and toxicity related to prior antibody therapy must be recovered to grade =\< 1 * External beam radiation therapy (XRT)/external beam irradiation including protons: \>= 14 days after local XRT; \>= 150 days after total body irradiation (TBI), craniospinal XRT or if radiation to \>= 50% of the pelvis; \>= 42 days if other substantial bone marrow (BM) radiation. * Radiopharmaceutical therapy (e.g., radiolabeled antibody, 131I-MIBG): \>= 42 days must have elapsed since systemically administered radiopharmaceutical therapy * Cellular therapy: \>= 42 days must have elapsed since the completion of any type of cellular therapy (e.g. modified T cells, natural killer \[NK\] cells, dendritic cells, etc.) * Patients must not have received prior exposure to nivolumab; for patients enrolled in Parts C, D, and E patients must not have received prior nivolumab or ipilimumab * For patients with solid tumors without known bone marrow involvement: * Peripheral absolute neutrophil count (ANC) \>= 750/mm\^3 * Platelet count \>= 75,000/mm\^3 (transfusion independent, defined as not receiving platelet transfusions for at least 7 days prior to enrollment) * Patients with known bone marrow metastatic disease will be eligible for study provided they meet the blood counts above (may receive transfusions provided they are not known to be refractory to red cell or platelet transfusions); these patients will not be evaluable for hematologic toxicity; at least 5 of every cohort of 6 patients with a solid tumor must be evaluable for hematologic toxicity, for Parts A and C; if dose-limiting hematologic toxicity is observed on either Part A or C, all subsequent patients enrolled must be evaluable for hematologic toxicity on that Part * Creatinine clearance or radioisotope glomerular filtration rate (GFR) \>= 70 ml/min/1.73 m\^2 or a serum creatinine based on age/gender as follows: * Age 1 to \< 2 years: maximum serum creatinine (mg/dL) 0.6 for males and females * Age 2 to \< 6 years: 0.8 for males and females * Age 6 to \< 10 years: 1 for males and females * Age 10 to \< 13 years: 1.2 for males and females * Age 13 to \< 16 years: 1.5 for males and 1.4 for females * Age \>= 16 years: 1.7 for males and 1.4 for females * Bilirubin (sum of conjugated + unconjugated) =\< 1.5 x upper limit of normal (ULN) for age * Serum glutamate pyruvate transaminase (SGPT) (alanine aminotransferase \[ALT\]) =\< 135 U/L; for the purpose of this study, the ULN for SGPT is 45 U/L * No evidence of dyspnea at rest, no exercise intolerance due to pulmonary insufficiency, and a pulse oximetry \> 92% while breathing room air * Serum lipase =\< ULN at baseline; patients with glucose intolerance should be on a stable regimen and be monitored * All patients and/or their parents or legally authorized representatives must sign a written informed consent; assent, when appropriate, will be obtained according to institutional guidelines * Tissue blocks or slides must be sent for all patients; if tissue blocks or slides are unavailable, the study chair must be notified prior to enrollment Exclusion Criteria: * Pregnant or breast-feeding women will not be entered on this study due to risks of fetal and teratogenic adverse events as there is yet no available information regarding human fetal or teratogenic toxicities; pregnancy tests must be obtained in girls who are post-menarchal; women of childbearing potential (WOCBP) receiving nivolumab will be instructed to adhere to contraception for a period of 5 months after the last dose of nivolumab; men receiving nivolumab and who are sexually active with WOCBP will be instructed to adhere to contraception for a period of 7 months after the last dose of nivolumab * Patients requiring daily systemic corticosteroids are not eligible; patients must not have received systemic corticosteroids within 7 days prior to enrollment; if used to modify immune adverse events related to prior therapy, \>= 14 days must have elapsed since last dose of corticosteroid; Note: use of topical or inhaled corticosteroids will not render a patient ineligible * Patients who are currently receiving another investigational drug are not eligible * Patients who are currently receiving other anti-cancer agents are not eligible * Patients with CNS tumors or known CNS metastases will be excluded from this trial; patients with a history of CNS metastases that have been previously treated may enroll if sequential imaging shows not evidence for active disease; patients with extra axial disease (e.g. skull \[bone\] metastasis that do not invade the dura) may enroll if there is no evidence for CNS edema associated with the lesion * Patients with a history of any grade autoimmune disorder are not eligible; asymptomatic laboratory abnormalities (e.g. antinuclear antibody \[ANA\], rheumatoid factor, altered thyroid function studies) will not render a patient ineligible in the absence of a diagnosis of an autoimmune disorder * Patients with \>= grade 2 hypothyroidism due to history of autoimmunity are not eligible; note: hypothyroidism due to previous irradiation on thyroidectomy will not impact eligibility * Patients who have an uncontrolled infection are not eligible * Patients with a history of congestive heart failure (CHF) or are at risk because of underlying cardiovascular disease or exposure to cardiotoxic drugs must have adequate cardiac function as clinically indicated: * Corrected QT interval (QTC) =\< 480 msec * Shortening fraction of \>= 27% by echocardiogram or ejection fraction of \>= 50% by gated radionuclide study * Patients with known human immunodeficiency virus (HIV) or hepatitis B or C are excluded * Patients who have received prior solid organ transplantation are not eligible * Patient who have received allotransplantation are not eligible * Patients who in the opinion of the investigator may not be able to comply with the safety monitoring requirements of the study are not eligible * Patients who have received prior anti-PD1 directed therapy (monoclonal antibody \[mAb\] or small molecule) are not eligible * Parts C, D, and E: patients who have received prior ipilimumab are not eligible