Ibrutinib, Fludarabine, and Pembrolizumab in High-Risk or Relapsed/Refractory Chronic Lymphocytic Leukemia/Small Lymphocytic Lymphoma

* INCLUSION CRITERIA: * Men and women with histologically confirmed CLL or SLL * Active disease as defined by at least one of the following IWCLL consensus criteria: * Weight loss greater than or equal to 10% within the previous 6 months. * Extreme fatigue. * Fevers of greater than 100.5 degrees F for greater than or equal to 2 weeks without evidence of infection. * Night sweats for more than one month without evidence of infection. * Evidence of progressive marrow failure as manifested by the development of, or worsening of, anemia and/or thrombocytopenia. * Massive or progressive splenomegaly. * Massive nodes or clusters or progressive lymphadenopathy. * Progressive lymphocytosis with an increase of \>50% over a 2-month period, or an anticipated doubling time of less than 6 months. * High-risk disease defined by meeting at least one of the following three criteria: * Relapsed and/or refractory CLL/SLL. * Presence of high-risk mutations detected by FISH or targeted sequencing, regardless of prior treatments status. * FISH: deletion 17p (or TP53), complex cytogenetics (3 or more abnormalities) * Targeted sequencing: TP53 mutations, SF3B1 mutations, or NOTCH1 mutation. Pathologic mutations occurring at the coding regions are accepted as relevant mutations. * MYC aberration. Aberration includes rearrangement, amplification, and tissue expression by immunohistochemistry * CLL or SLL with disease transformation with Hodgkin-like cells regardless of prior treatment status. * Eastern Cooperative Oncology Group (ECOG) performance status of less than or equal to 1. * Adequate organ function as defined by the study protocol. * Agreement to use acceptable methods of contraception during the study and for 90 days after the last dose of study drug if sexually active and able to bear or beget children. * Willing and able to participate in all required evaluations and procedures in this study protocol including swallowing capsules without difficulty. * Ability to understand the purpose and risks of the study and provide signed and dated informed consent and authorization to use protected health information. * Individuals greater than or equal to 18 years old EXCLUSION CRITERIA: * Transformation of CLL into lymphomas other than those with Hodgkin-like cells. * Currently receiving or previously participated to receive an investigational agent within 4 weeks prior to study treatment. * Currently receiving or previously received monoclonal antibodies, immunomodulatory therapy, chemotherapy, radiation, or radioimmunotherapy within 4 weeks prior to study treatment, or has not recovered from non-hematologic adverse events due to a previously administered agent. * Major surgery within 4 weeks of first dose of study drug * Currently receiving systemic steroid therapy (i.e. prednisone) or any other form of immunosuppressive therapy within 7 days prior to the first dose of trial treatment. * Prior therapy with BTK inhibitor, anti-PD-1, anti-PD-L1, or anti-PD-L2 agent. * Active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (e.g.., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment. * Known additional malignancy that is progressing or requires active treatment. * Known history of, or any evidence of active, non-infectious pneumonitis that required steroids. * Known bleeding disorders (i.e., von Willebrand s disease or hemophilia). * Known HIV infection * Active hepatitis B or hepatitis C infection. * Recent known active infection requiring systemic therapy that was completed less than or equal to 14 days before the first dose of study drug. * Known history of active tuberculosis. * Any uncontrolled active systemic infection. * Known hypersensitivity to ibrutinib, fludarabine, or pembrolizumab. * Requires concomitant anticoagulation with Coumadin (warfarin) or other vitamin K antagonists. * Malabsorption syndrome, disease significantly affecting gastrointestinal function, or resection of the stomach or small bowel or ulcerative colitis, symptomatic inflammatory bowel disease, or partial or complete bowel obstruction * History of stroke or intracranial hemorrhage within 6 months before the first dose of study drug * Subjects who received a strong cytochrome P450 (CYP) 3A inhibitor within 7 days prior to the first dose of ibrutinib or subjects who require continuous treatment with a strong CYP3A inhibitor * Currently active, clinically significant cardiovascular disease including uncontrolled or symptomatic arrhythmias, Class 3 or 4 congestive heart failure as defined by New York Heart Association Functional Classification, or a history of myocardial infarction, unstable angina or acute coronary syndrome within 6 months of screening. * Life-threatening illness, medical condition or organ system dysfunction which, in the investigator's opinion, could compromise the subject's safety, interfere with the absorption or metabolism of ibrutinib and fludarabine, or put the study outcomes at undue risk * Female patients who are currently in pregnancy, or unwilling to use acceptable methods of contraception or refrain from pregnancy if of childbearing potential or currently breastfeeding. Male patients who are unwilling to follow the contraception requirements described in this protocol. * Psychiatric illness/social situations that would limit the patient s ability to tolerate and/or comply with study requirements. * Unable to understand the investigational nature of the study or give informed consent. * Currently active, clinically significant hepatic impairment Child-Pugh class B or C according to the Child Pugh classification. * Vaccinated with live, attenuated vaccines within 4 weeks of first dose of study drug.