Venetoclax and Selinexor in Treating Patients with Relapsed or Refractory High Risk Hematologic Malignancies

Inclusion Criteria: * Written informed consent in accordance with federal, local, and institutional guidelines. The patient must provide informed consent prior to the first screening procedure. * Age ≥ 18 years * Life expectancy \> 12 weeks. * Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1. * Histologically confirmed diagnosis of one of the following, in accordance with WHO diagnostic criteria: * Escalation: * Diffuse Large B-cell Lymphoma (DLBCL, including primary mediastinal large B cell lymphoma, T cell rich B cell lymphoma, and high-grade B cell lymphoma NOS). Patients with Burkitt's, lymphoblastic lymphoma, follicular lymphoma, and mantle cell lymphoma are not included. OR * Acute Myeloid Leukemia (AML) * Expansion: * Diffuse Large B-cell Lymphoma and acute myeloid leukemia as above. * VEN Refractory expansion cohort (Diffuse Large B-cell Lymphoma and acute myeloid leukemia only): Patients must have previously received and failed venetoclax therapy (either monotherapy or combination) during their treatment course (i.e., patients may receive non-VEN therapy immediately prior to enrollment on this study). Treatment failure is defined as evidence of disease progression after ≥ 1 cycle (four weeks) of full-intensity venetoclax-based therapy (i.e., 28 days exclusive of ramp-up. Patients that require dose reductions due to intolerance may be considered for this cohort after discussion with the sponsor.) * Relapsed or refractory following ≥ 1 line(s) of prior therapy * Patients that relapse ≥ 3 months after allogeneic hematopoietic cell transplantation (HCT) are eligible. * Female patients of childbearing potential must agree to use two methods of contraception (including one highly effective and one effective method of contraception) and have a negative serum pregnancy test at screening. Male patients must use an effective barrier method of contraception if sexually active with a female of childbearing potential. For both male and female patients, effective methods of contraception must be used throughout the study and for 3 months following the last dose of study treatment. * For leukemia and DLBCL with known or suspected marrow involvement, patients must have at least 10-15 mL of bone marrow aspirate material obtained within 14 days of beginning treatment on this study. Patients with DLBCL must have 3-5 unstained slides, or tissue block, available for evaluation within 14 days of study enrollment in the expansion cohorts. DLBCL patients enrolled during the escalation phase must have blocks available for submission within 28 days of beginning treatment. * CBC testing must confirm adequate marrow reserve, as demonstrated by: * DLBCL: Hgb ≥ 10g/dL, platelets ≥ 75,000 cells/mm3 , ANC ≥ 1,000 cells/mm3 * Adequate organ function, as demonstrated by: * Total bilirubin \< 1.5 x upper limit of normal (ULN) (except patients with Gilbert's syndrome (hereditary indirect hyperbilirubinemia), who must have a total bilirubin of \< 3 x ULN, and * Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) normal to \< 2 x ULN * Calculated creatinine clearance \> 50 mL/min (per the Cockcroft-Gault formula). * Patients with laboratory evidence of liver or kidney dysfunction secondary to underlying disease, that is expected to reverse with treatment, may be enrolled after discussion with the sponsor/investigator. Exclusion Criteria: * Patients who are pregnant or lactating * Patients who received any systemic anticancer therapy including investigational agents or radiation ≤3 weeks (or ≤5 half-lives of the drug \[whichever is shorter\]) prior to C1D1. Hydroxyurea is permitted for up to 14 days in AML patients * Inadequate recovery from toxicity attributed to prior anti-cancer therapy. With the exception of alopecia or fatigue, patients must have recovered to baseline or ≤ Grade 1 (NCI-CTCAE v4.03) residual toxicity prior to first dose of protocol-indicated treatment. * Participation in another clinical trial with any investigational drug within 14 days prior to study enrollment. * Patients included in the VEN refractory cohort that have discontinued venetoclax therapy (either monotherapy or combination) due to toxicity or hypersensitivity, including prior history of grade 3/4 TLS during prior VEN exposure * In dose expansion cohorts, except venetoclax refractory cohort, no prior treatment with SINE compounds, another XPO1 inhibitor, or BCL-2 inhibitors. * Active GVHD requiring calcineurin inhibitors or steroid dosing ≥ 10mg/day prednisone (or equivalent) or \< 3 months from allogenic hematopoietic cell transplant (HCT). * Uncontrolled (i.e., clinically unstable) infection requiring parenteral antibiotics, antivirals, or antifungals within one week prior to first dose; however, prophylactic use of these agents is acceptable even if parenteral. * Major surgery within 2 weeks of first dose of study drug. * Any life-threatening illness, medical condition or organ system dysfunction which, in the investigator's opinion, could compromise the patient's safety. * Unstable cardiovascular function: * Symptomatic ischemia, or * Uncontrolled clinically significant conduction abnormalities (i.e., ventricular tachycardia on anti-arrhythmia therapy are excluded; 1st degree AV block or asymptomatic LAFB/RBBB will not be excluded), or * Congestive heart failure (CHF) of NYHA Class ≥3, or * Myocardial infarction (MI) within 3 months. * Known active Hepatitis B or Hepatitis C infection (Hepatitis testing is not required as part of this study). * Known human immunodeficiency virus (HIV) infection (HIV testing is not required as part of this study). * Subject has received the following ≤ 7 days prior to Cycle 1, Day 1: Strong and moderate CYP3A inducers such as rifampin, carbamazepine, phenytoin, and St. John's wort. * Subject has received the following ≤ 5 days prior to Cycle 1, Day 1: Strong and moderate CYP3A inhibitors such as fluconazole, ketoconazole, and clarithromycin. * Subject has consumed grapefruit, grapefruit products, Seville oranges (including marmalade containing Seville oranges) or Starfruit ≤ 3 days prior to Cycle 1, Day 1. * Inability to swallow oral medication; or the presence of a poorly controlled gastrointestinal disease, disorder, syndrome, or dysfunction that could significantly affect the absorption of oral study drug - e.g. Crohn's disease, ulcerative colitis, chronic diarrhea (defined as \> 4 loose stools per day), malabsorption syndrome, or bowel obstruction. * Inability or unwillingness to take required and recommended medications intended to prevent and treat potential adverse events of tumor lysis syndrome (TLS), nausea and vomiting, loss of appetite, and fatigue. * Patients unwilling to comply with the protocol