Phase 2 Study of Alisertib Therapy for Rhabdoid Tumors

INCLUSION CRITERIA for Patients on All Strata EXCEPT Stratum P * Patients must be \< 22 years of age at time of diagnosis (e.g., eligible until 22nd birthday). * Histologic diagnosis of AT/RT or MRT as documented by the institutional pathologist with loss of INI1 or BRG1 expression in tumor cells confirmed by immunohistochemistry, or by molecular confirmation of tumor-specific biallelic SMARCB1/SMARCA4 loss/mutation if INI1/BRG1 immunohistochemistry is not available. For Stratum A participants, histologic confirmation of the diagnosis of AT/RT or MRT may be from the original diagnosis or at the time of recurrence/progression. * Patients must have adequate organ function (bone marrow, renal, liver), as defined in the protocol. * Female patients who are at least 10-years-old or are post-menarchal must have a negative serum or urine pregnancy test prior to enrollment. * Patients of childbearing or child fathering potential must be willing to use a medically acceptable form of birth control, which includes abstinence during study treatment and 12 months after the last dose of alisertib. Inclusion Criteria for Stratum A Participants: * Patients with recurrent or progressive AT/RT/MRT (either CNS and/or extra-CNS) with radiographically measurable disease as defined by at least 1 lesion that can be measured in 2 dimensions or with tumor cells present in the CSF taken within 2 weeks prior to enrollment. * Performance status defined by Karnofsky or Lansky \> 60 (except for patients with posterior fossa syndrome). Use Karnofsky for patients \> 16 years and Lansky for patients \< 16 years. Note: Patients who are unable to walk because of paralysis, but who are up in a wheelchair, will be considered to be ambulatory for the purpose of assessing the performance score. * Patient has fully recovered from the acute toxic effects of chemotherapy, immunotherapy, or radiation therapy prior to entering this study: * Myelosuppressive chemotherapy: Patient has not received myelosuppressive chemotherapy within 3 weeks of enrollment onto this study (4 and 6 weeks if prior temozolomide and nitrosourea, respectively). * Hematopoietic growth factors: At least 7 days must have elapsed since the completion of therapy with a growth factor. At least 14 days must have elapsed after receiving pegfilgrastim. * Biologic (anti-neoplastic agent): At least 7 days must have elapsed since completion of therapy with a biologic agent. For agents that have known adverse events occurring beyond 7 days after administration, this period prior to enrollment must be extended beyond the time during which adverse events are known to occur. * Monoclonal antibodies: At least 3 half-lives must have elapsed since prior therapy that included a monoclonal antibody (see Appendix I). * Radiation therapy: at least 3 months must have elapsed since any irradiation unless measurable disease progression occurs at a site separate from the irradiated area and the patient has recovered from toxicities associated with radiation therapy. * Patients with progressive synchronous/metachronous AT/RT and MRT will be eligible for stratum A3. * Patients may not have previously received alisertib. * Live expectancy \>8 weeks. * Stable neurologic deficits on a stable dose of corticosteroids (if applicable) for at least 1 week before study enrollment. Inclusion Criteria for Strata B or C Participants: * Patients with newly diagnosed AT/RT. * Performance status defined by Karnofsky or Lansky \> 30 (except for patients with posterior fossa syndrome). Other requirements of performance evaluation are the same as for Stratum A participants. * No previous anticancer therapy (radiation therapy or chemotherapy) other than the use of corticosteroids. * Patients must begin treatment as outlined in the protocol within 42 days of definitive surgery (day of surgery is day 0; definitive surgery includes last surgery to resect residual tumor). Inclusion Criteria for Stratum D Participants: * Patients with newly diagnosed AT/RT and synchronous extra-CNS MRT. * Performance status defined by Karnofsky or Lansky \> 30 (except for patients with posterior fossa syndrome). Other requirements of performance evaluation are the same as for Stratum A participants. * No previous anticancer therapy (radiation therapy or chemotherapy) other than the use of corticosteroids. * Patients must begin treatment within 42 days of definitive surgery (day of surgery is day 0; definitive surgery includes repeat surgeries to resect residual tumor). Inclusion Criteria for Stratum P Participants: * Biological parent of patient enrolling on the protocol (SJATRT) will be assigned to Stratum P. EXCLUSION CRITERIA for All Strata Except Stratum P: * Clinically significant medical disorders that could compromise the ability to tolerate protocol therapy or that would interfere with the study procedures or results history. * Presence of an active, uncontrolled infection. * Known history of uncontrolled sleep apnea syndrome or other conditions that could result in excessive daytime sleepiness, such as severe chronic obstructive pulmonary disease or a requirement for supplemental oxygen. * Requirement for constant administration of proton-pump inhibitor, H2 antagonist, or pancreatic enzymes. Intermittent uses of antacids or H2 antagonists are allowed while patients are on dexamethasone as described in the protocol. * Inability to comply with the safety monitoring requirements of the study, as judged by the investigator. * Female participants of childbearing potential cannot be pregnant or breast-feeding. * Patients who are receiving other investigational drugs 14 or fewer days before enrollment. * Treatment with clinically significant enzyme inducers, such as the enzyme-inducing antiepileptic drugs phenytoin, carbamazepine or phenobarbital, or with rifampin, rifabutin, rifapentine, or St. John's wort within 7 days prior to initiation of alisertib. * Known gastrointestinal disease or procedures that could interfere with the oral absorption or tolerance of alisertib. Examples include, but are not limited to partial gastrectomy, history of small intestine surgery, and celiac disease. * Myocardial infarction within 6 months prior to enrollment or New York Heart Association Class III or IV heart failure, uncontrolled angina, severe uncontrolled ventricular arrhythmias, or electrocardiographic evidence of acute ischemia or active conduction system abnormalities. If for some reason an electrocardiogram is obtained before study enrollment, any abnormalities detected should be documented as clinically irrelevant. * Other severe acute or chronic medical or psychiatric condition, including uncontrolled diabetes, malabsorption, resection of the pancreas or upper small-bowel, or requirement for pancreatic enzymes, any condition that would modify the absorption of oral medications in the small bowel or any laboratory abnormality that may increase the risk associated with study participation or investigational product administration or that may interfere with the interpretation of study results and, in the judgment of the investigator, would make the patient inappropriate for enrollment in this study.