Study of the Safety, Tolerability and Efficacy of KPT-8602 in Participants With Relapsed/Refractory Cancer Indications

INCLUSION CRITERIA 1. Written informed consent signed prior to any screening procedures and in accordance with federal, local, and institutional guidelines. 2. Age ≥ 18 years. 3. Adequate hepatic function: 1. total bilirubin ≤ 2 times the upper limit of normal (ULN) (except participants with Gilbert's syndrome \[hereditary indirect hyperbilirubinemia\] who must have a total bilirubin of ≤ 4 times ULN), 2. aspartate aminotransferase (AST) and alanine aminotransferase (ALT) ≤ 2.5 times ULN (except participants with known liver involvement of their tumor who must have their AST and ALT ≤ 5.0 times ULN). 4. Adequate renal function: estimated creatinine clearance of ≥ 30 mL/min, calculated using the formula of Cockcroft and Gault (140-Age) × Mass (kg)/(72 × creatinine mg/dL); multiply by 0.85 if female. 5. Contraception: 1. Participants with RRMM, CRC, RR high-risk MDS (Part F Phase 2), and AML (Part H): Female participants of child-bearing potential must agree to use dual methods of contraception (including 1 highly effective and 1 effective method of contraception) and have a negative serum test at Screening, and male participants must use an effective barrier method of contraception if sexually active. For both male and female participants, effective methods of contraception must be used throughout the study and for 3 months following the last dose 2. Participants with RR mCRPC: Participants must use an effective barrier method of contraception if sexually active. Effective methods of contraception must be used throughout the study and for 3 months following the last dose 3. Participants with newly diagnosed intermediate/high-risk MDS (Part G): Female participants of child-bearing potential must agree to use dual methods of contraception (including 1 highly effective and 1 effective method of contraception) and have a negative serum pregnancy test at Screening, and male participants must use an effective barrier method of contraception if sexually active. For both male and female participants, effective methods of contraception must be used throughout the study and for 6 months following the last dose INDICATION-SPECIFIC INCLUSION CRITERIA Relapsed/Refractory Multiple Myeloma (Parts A1, A2, and B - Completed): 6. Symptomatic, histologically confirmed MM and evidence of disease progression, based on IMWG guidelines. 7. Participants must have measurable disease as defined by at least 1 of the following: 1. Serum M-protein ≥ 0.5 g/dL by serum protein electrophoresis (SPEP) or for immunoglobulin (Ig) A myeloma, by quantitative IgA. If SPEP is felt to be unreliable for routine M-protein measurement (e.g., for participants with IgA MM), then quantitative Ig levels by nephelometry; or 2. Urinary M-protein excretion at least 200 mg/24 hours; or 3. Serum free light chain (FLC) whereby the involved light chain measures ≥ 10 mg/dL and with an abnormal ratio. 8. Previously treated with ≥ 3 prior regimens (lines of therapy) that included at least 1 of each of the following: an immunomodulatory drug, a proteasome inhibitor, and a steroid. 9. MM refractory to the participants most recent anti-MM regimen. 10. Participants receiving hematopoietic growth factor support including erythropoietin, darbepoetin, granulocyte-colony stimulating factor, granulocyte macrophage-colony stimulating factor, and platelet stimulators can continue to do so, but must be transfusion independent for at least 1 week prior to Cycle 1 Day 1 (C1D1) in the study. 11. Adequate hematopoietic function: total white blood cell (WBC) count ≥ 1500/mm\^3, absolute neutrophil count (ANC) ≥ 800/mm\^3, hemoglobin (Hb) ≥ 8.0 g/dL, and platelet count ≥ 75,000/mm\^3. 12. Eastern Cooperative Oncology Group (ECOG) performance status of ≤1. 13. Life expectancy of ≥ 4 months. Relapsed/Refractory Colorectal Cancer (Part C - Completed): 14. Histological or cytological documentation of adenocarcinoma of the colon or rectum. 15. Measurable disease by RECIST v1.1. 16. Metastatic disease not suitable for upfront curative-intent surgery. 17. Participants with site-defined KRAS status (wild-type or mutant) from a fresh or archival tumor biopsy prior to enrollment. All participants must be willing to have fresh biopsies to obtain tumor tissue for biomarker analysis. 18. Documented evidence of progressive disease according to RECIST v1.1. 19. Prior treatment (with completion of a course of therapy, or to disease progression or intolerability) with each of the following: 1. Fluoropyrimidine-, oxaliplatin-, irinotecan-based chemotherapies (e.g., FOLFOX and/or FOLFIRI) 2. if KRAS wild-type, an anti-EGFR therapy, 3. Prior third line treatment with regorafenib or TAS-102, will be assessed on an individual basis, 4. Note: The requirement for prior third line regorafenib will be assessed on an individual basis by the investigator in consultation with the Karyopharm Medical Monitor 5. Radiation and surgery are not considered as prior anticancer regimens 20. Participants should not be transfusion dependent. 21. Adequate hematopoietic function: ANC ≥ 1000/mm\^3, hemoglobin (Hb) ≥ 9.0 g/dL, and platelet count ≥ 100,000/mm\^3. 22. ECOG performance status of ≤ 1. 23. Life expectancy of ≥ 4 months. Relapsed/Refractory Metastatic Castration-resistant Prostate Cancer (Parts D and E - Completed): 24. Histologically confirmed adenocarcinoma of the prostate with archival tumor tissue available for molecular analyses. If the participants does not have a prior histological diagnosis, then a fresh biopsy at Screening may be used for this purpose. a. Optional: All participants will be asked to have fresh biopsies to obtain tumor tissue for biomarker analysis. 25. Surgically or medically castrated, with testosterone levels of \< 50 ng/dL (\< 2.0 nM). If the participant is being treated with luteinizing hormone-releasing hormone (LHRH) agonists (participants who have not undergone orchiectomy), and participants must have shown to progress on this. 26. Documented mCRPC progression as assessed by the Investigator with 1 of the following: 1. Prostate-specific antigen (PSA) progression defined by a minimum of 3 rising PSA levels (at approximately Day -30 and approximately Day -45) with an interval of \> 1 week between each determination. The PSA values at the Screening visit should be \> 2 μg/L (\> 2 ng/mL); participants on systemic glucocorticoids for control of symptoms must have documented PSA progression by Prostate Cancer Working Group 3 (PCWG3) while on systemic glucocorticoids prior to commencing C1D1 of treatment. 2. Radiographic progression of soft tissue disease by modified RECIST criteria 1.1 or of bone metastasis with 2 or more documented new bone lesions on a bone scan with or without PSA progression. 27. Initial response (per modified PCWG3 Guidelines) to second generation anti-hormonal therapy (examples: abiraterone, enzalutamide, TAK 700), but later relapsed. Disease relapse would be defined as progressive disease at the time of entry per inclusion criterion 24. 28. Zero to 2 previous taxane-based chemotherapy regimens. If docetaxel chemotherapy is used more than once, this will be considered as 1 regimen. Participants may have had prior exposure to cabazitaxel treatment. Participants may be taxane naïve. 29. At least 2 weeks from completion of any radiotherapy including a single fraction of radiotherapy for the purposes of palliation (confined to 1 field) is permitted. 30. Participants should not be transfusion dependent. 31. Albumin \> 2.5 g/dL. 32. Adequate hematopoietic function: ANC ≥ 1000/mm\^3, hemoglobin (Hb) ≥ 9.0 g/dL, and platelet count ≥ 100,000/mm\^3. 33. Part E only: Participants currently receiving treatment with abiraterone and appropriate to continue in the opinion of the Investigator. Participants must also have been on and continue on a stable dose of corticosteroids (prednisone or dexamethasone) for 30 days prior to C1D1. 34. ECOG performance status of ≤ 1. 35. Life expectancy of ≥ 4 months. RR High-risk Myelodysplastic Syndrome (Part F Phase 2): 36. Documented diagnosis of MDS with 5% to 19% myeloblasts in the bone marrow (2016 WHO classification). 37. The marrow histopathology must be documented by recent bone marrow biopsy (within 30 days prior to C1D1). 38. IPSS-R: intermediate, high- or very-high-risk MDS. 39. RR MDS defined as having one of the following: 1. ≥ 2 cycles of hypomethylating agents (azacitidine and/or decitabine, ASTX727, or experimental agents) with clear PD (pancytopenia with ≥ 50% increase in bone marrow blasts) or participant progressed to a higher risk category of MDS OR 2. ≥ 4 cycles of HMA therapy with SD/lack of improvement (no CR/mCR/PR/HI) per International Working Group (IWG) 2006 criteria, or intolerance to treatment (≥ 6 cycles of azacitidine if required per local standard of care guidelines to establish lack of improvement/response to azacitidine) OR 3. Relapse or disease progression after an initial response to HMA (CR/mCR/PR/HI) per IWG 2006 criteria. 40. ECOG performance status of \< 2. 41. Prior to enrolling a participant with imminent risk of AML transformation (per opinion of the Investigator) or for participants with RAEB-2 MDS, the Medical Monitor must be contacted. Newly Diagnosed Intermediate/High-risk Myelodysplastic Syndrome (Part G): 42. Documented diagnosis of MDS with 5% to 19% myeloblasts in the bone marrow (2016 WHO classification) a. The marrow histopathology must be documented in bone marrow biopsy (within 30 days prior to C1D1) 43. IPSS-R intermediate, high- or very-high-risk MDS. 44. No prior therapy for HR-MDS (up to one prior cycle of an HMA is allowed). Prior supportive care in the form of transfusions, growth factors, etc. is permitted. 45. ECOG performance status of \< 2. AML Maintenance (Post-alloSCT) Therapy (Part H): 46. Participants with de novo AML or AML secondary to prior myelodysplastic disease. 47. Received one allogeneic SCT (alloSCT). 48. Participants must be able to start study treatment between 40 and 100 days following alloSCT 49. Participants must be CR/CRi at the time of study enrollment, and must meet at least one of the following criteria: 1. MRD positive at time of enrollment 2. Evidence of disease pre-alloSCT (received alloSCT while not in CR/CRi, or while MRD positive) 3. In CR2 or greater pre-alloSCT (regardless of MRD status) 4. Adverse cytogenetics at time of diagnosis (regardless of MRD status) per ELN 2017 criteria 50. Adequate engraftment within 14 days prior to starting study therapy: absolute neutrophil count (ANC) ≥ 1.0 x 10\^9/L without daily use of myeloid growth factor, and platelet count 75 x 10\^9/L without platelet transfusion within 1 week. 51. ECOG performance status of ≤ 2 EXCLUSION CRITERIA Participants in All Parts of the Study: 1. Female participants who are pregnant or lactating. 2. Major surgery within 4 weeks before C1D1. 3. Impaired cardiac function or clinically significant cardiac diseases, including any of the following: 1. Unstable angina or acute myocardial infarction ≤ 3 months prior to C1D1 2. Clinically significant heart disease (e.g., symptomatic congestive heart failure \[e.g., \> NYHA Class 2\]; uncontrolled arrhythmia, or hypertension; history of labile hypertension or poor compliance with an antihypertensive regimen) 4. Uncontrolled active severe systemic infection requiring parenteral antibiotics, antivirals, or antifungals within 1 week prior to C1D1. 5. Participants with known symptomatic brain metastasis are not suitable for enrollment. Participants with asymptomatic, stable, treated brain metastases are eligible for study entry 6. Participants with a known history of human immunodeficiency virus (HIV); HIV testing is not required as part of this study 7. Known, active hepatitis A, B, or C infection; or known to be positive for HCV RNA or HBsAg (HBV surface antigen) 8. Prior malignancies: 1. Participants with adequately resected basal or squamous cell carcinoma of the skin, or adequately resected carcinoma in situ (i.e., cervix) may enroll irrespective of the time of diagnosis 2. Participants with relapsed/refractory MM, CRC, and mCRPC only: Prior malignancies which may interfere with the interpretation of the study. Cancer treated with curative intent \< 5 years previously will not be allowed unless approved by the Sponsor. Cancer treated with curative intent \> 5 years previously and without evidence of recurrence will be allowed. 3. For participants in Arms F Phase 2, G, and H: Prior malignancy is not an exclusion 9. Participants with gastrointestinal tract disease (or uncontrolled vomiting or diarrhea) that could interfere with the absorption of eltanexor (or ASTX727 in Part G). 10. Serious psychiatric or medical conditions that, in the opinion of the Investigator, could interfere with treatment, compliance, or the ability to give consent. 11. Participants unwilling to comply with the protocol including required biopsies and sample collections required to measure disease. INDICATION-SPECIFIC EXCLUSION CRITERIA Relapsed/Refractory Multiple Myeloma (RRMM) (Parts A1, A2 and B - Completed): 12. Time since the last prior therapy for treatment of RRMM: 1. Radiation, chemotherapy, immunotherapy or any other anticancer therapy, including investigational anticancer therapy ≤ 2 weeks prior to C1D1 2. Palliative steroids for disease related symptoms are allowed up to 3 days prior to C1D1. 3. Participants must have recovered or stabilized (≤ Grade 1 or to their baseline) from toxicities related to their previous treatment except for alopecia 13. Participants with active graft versus host disease after allogeneic stem cell transplantation. At least 3 months must have elapsed since completion of allogeneic stem cell transplantation. 14. Grade \> 2 peripheral neuropathy or Grade 2 peripheral neuropathy with pain within 2 weeks prior to C1D1. Relapsed/Refractory Colorectal Cancer (Part C - Completed): 15. Radiotherapy, chemotherapy, or any other anticancer therapy, including investigational anticancer therapy within 2 weeks prior to Screening. Participants must have recovered from clinically significant toxicities. The site of irradiation should have evidence of progressive disease (new lesions or increase in lesion size) if this is the only site of disease. 16. Participants who have been treated with their most recent chemotherapy or investigational drugs ≤21 days or 5 half-lives (whichever is longer) prior to the first dose of study treatment, and/or have any acute toxicities due to prior chemotherapy and/ or radiotherapy that have not resolved to a NCI CTCAE v4.03 Grade 0 or Grade 1 with the exception of chemotherapy induced alopecia and Grade 2 peripheral neuropathy. Relapsed/Refractory Metastatic Castration-Resistant Prostate Cancer (Parts D and E - Completed): 17. Participants who have been treated with their most recent chemotherapy or investigational drugs ≤ 21 days or 5 half-lives (whichever is longer) prior to the first dose of study treatment, and/or have any acute toxicities due to prior chemotherapy and/ or radiotherapy that have not resolved to a NCI CTCAE v4.03 Grade 0 or Grade 1 with the exception of chemotherapy-induced alopecia and Grade 2 peripheral neuropathy. 18. Initiating bisphosphonate therapy or adjusting bisphosphonate dose/regimen within 30 days prior to C1D1. Participants on a stable bisphosphonate or denosumab regimen are eligible and may continue. RR High-risk Myelodysplastic Syndrome (Part F Phase 2): 19. Very low or low-risk MDS per IPSS-R. 20. Evidence of transformation to AML by the World Health Organization (WHO) (≥ 20% blasts in bone marrow or peripheral blood). 21. Participants who are significantly below their ideal body weight as judged by the Investigator. 22. Any severe GVHD, or organ system dysfunction, which in the Investigator's opinion, could compromise the participant's safety. Newly Diagnosed Intermediate/High-risk Myelodysplastic Syndrome (Part G): 23. IPSS-R very low or low-risk MDS. 24. Evidence of transformation to AML by the WHO (≥ 20% blasts in bone marrow or peripheral blood). AML Maintenance (Post-alloSCT) Therapy (Part H): 25. Use of any anti-cancer maintenance therapy after alloSCT and prior to starting the study treatment 26. Active GVHD Grade 2 or higher. 27. Concurrent use of corticosteroids equivalent of prednisone at a dose \> 0.5 mg/kg.