A Study of the Drugs Prexasertib, Irinotecan, and Temozolomide in People With Desmoplastic Small Round Cell Tumor and Rhabdomyosarcoma

Inclusion Criteria: * Consent/Assent: all patients and/or their parents or legally authorized representatives must sign written informed consent; assent, when appropriate, will be obtained according to institutional guidelines * Age: patients must be ≥12 months of age at the time of study enrollment * Diagnosis: patients must have histologically documented locally advanced or metastatic desmoplastic small round cell tumor or rhabdomyosarcoma (confirmed at MSK) * Therapeutic options: patient's current disease state must be one which has failed standard therapy and for which there is no known curative therapy * Disease Status: patients must have measurable disease based on RECIST 1.1 * Performance level: Karnofsky ≥ 50% for patients \> 16 years of age and Lansky ≥ 50 for patients ≤ 16 years of age * Prior Therapy: patients may have had any number of prior therapies, but must have recovered from the acute toxic effects of all prior anti-cancer therapy (other than alopecia) as described below and must meet the following minimum duration from prior anti-cancer directed therapy prior to enrollment °patients who have previously received irinotecan and/or temozolomide will be allowed * 21 days must have elapsed after the last dose of cytotoxic or myelosuppressive chemotherapy * 7 days must have elapsed after the last dose of anti-cancer agents not known to be myelosuppressive * 14 days must have elapsed after radiation therapy, and toxicity related to prior radiation therapy must be recovered to grade ≤ 1 * 21 days must have elapsed after the last dose of antibody therapy, and toxicity related to prior antibody therapy must be recovered to grade ≤ 1 * Organ Function Requirements: Adequate bone marrow function defined as: * absolute neutrophil count (ANC) ≥ 1500/mm\^3 * platelet count ≥ 100,000/ mm\^3 * hemoglobin ≥ 8 g/dl * Adequate renal function defined as: * Creatinine clearance or radioisotope glomerular filtration rate ≥ 70 mL/min/1.73m2 OR * Serum creatinine based on age/gender derived from the Schwartz formula for estimating GFR53 * Adequate liver function defined as: * Bilirubin (sum of conjugated + unconjugated) ≤ 1.5 x upper limit of normal for age * AST or ALT ≤ 5 x upper limit of normal for patients with liver metastases * Serum albumin ≥ 2.5 g/dl * Adequate cardiac function defined as: * echocardiogram with left ventricular ejection fraction (LVEF) \>45% * QTc \< 470 ms on screening 12 lead electrocardiogram * Pregnancy/Contraception * post-menarchal females must have a negative urine or serum pregnancy test at screening and ≤ 24 hours prior to study treatment * males or females of reproductive potential must be willing to use a barrier method of contraception throughout the course of the study and for 6 months after participation Exclusion Criteria: * Patients for whom the investigator deems that irinotecan and temozolomide are not appropriate are not eligible. * Patients who have an uncontrolled infection are not eligible. * Patients who are pregnant or breast feeding are not eligible. * Patients who have a history of Torsades de Pointes, carry a diagnosis of congestive heart failure, or have a family history of prolonged QT syndrome are not eligible. * Patients who, in the opinion of the investigator, may not be able to comply with the safety monitoring requirements of the study are not eligible. * Patients with known hypersensitivity to irinotecan or its excipients are not eligible.