A Study of Olaparib and Durvalumab in Prostate Cancer

Inclusion Criteria: * Willing and able to provide written informed consent and HIPAA authorization for the release of personal health information. A signed informed consent must be obtained before screening procedures are performed. NOTE: HIPAA authorization may be either included in the informed consent or obtained separately. * Males 18 years of age and above * Body weight \> 30kg * History of radical prostatectomy * Histologically confirmed prostate cancer with progressive disease defined as: * Rising PSA (50% or more increase to a level of 0.50 ng/mL or more, based on at least 3 PSA determinations obtained at least 1 week apart). The 50% rise in PSA is across the 3 determinations, and these determinations do not need to be sequential. * PSA doubling time of \</= 9 months as calculated according to the Memorial Sloan Kettering Cancer Center nomogram * No evidence of metastatic disease on conventional imaging (CT/MRI and bone scan). However, subjects with pelvic and/or retroperitoneal nodes \< 2cm in the short axis will be permitted on study, as they are considered not to have definitive metastases. (Note: Metastatic disease on investigational imaging, Prostate Specific Membrane Antigen-targeted (PSMA) PET, PET-choline, or other novel PET tracers who do not have evidence of metastatic disease using conventional imaging (CT/MRI, bone scan) are allowed.) * Molecular evidence of DDR deleterious mutations (somatic or germline), including BRCA1, BRCA2, ATM, CHEK2, FANCA, RAD51C, RAD51D, PALB2, BRIP1, BARD1, or CDK12. Mutations may be truncating, splice site mutations, missense or homozygous deletions. Mutation status is determined by a local laboratory with the result documented in the subject's medical record, previously obtained genomic testing from a CLIA-certified lab, or via archival or fresh tissue. * ECOG status of ≤1 * Normal organ function with acceptable initial laboratory values within 14 days of treatment start: * WBC ≥ 2000/ul * ANC ≥ 1500/uL * Hemoglobin ≥ 10g/dL * Platelet count ≥100,000/ul * Creatinine Clearance ≥ 51 mL/min estimated using the Cockcroft-Gault equation * Bilirubin ≤ 1.5 ULN (unless documented Gilbert's disease) * SGOT (AST) ≤ 2.5 x ULN (unless liver metastases are present, in which case AST must be 5 x ULN) * SGPT (ALT) ≤ 2.5 x ULN (unless liver metastases are present, in which case ALT must be 5 x ULN) * Non-castrate level of testosterone defined as a value ≥ 150 ng/dL * Life expectancy of ≥ 52 weeks. * Agree to use two medically acceptable, highly effective forms of birth control (e.g., spermicide in conjunction with a barrier such as a condom) or sexual abstinence for the duration of the study, including 180 days after the last dose of study drug. Sperm donation is prohibited during the study and for 3 months after the last dose of study drug. Female partners of child bearing potential should use hormonal or barrier contraception unless postmenopausal or abstinent. Exclusion Criteria: * No other malignancy from which the subject has been disease-free for less than 3 years, with the exception of adequately treated and cured non-invasive malignancies such as basal or squamous cell skin cancer or superficial bladder cancer. * Less than one month prior to treatment start from last prior regimen or radiation exposure. Prior radiotherapy to the prostate (adjuvant or salvage radiotherapy) is allowed. * No prior investigational use with an anti-PD(1) including durvalumab or anti-CTLA4 antibody. * No prior treatment with a PARP inhibitor, including olaparib. * No concomitant or prior therapy with any of the following: IL-2, interferon, or other non-study immunotherapy regimens; immunosuppressive agents; or chronic use of systemic corticosteroids within 6 weeks of treatment start. Exceptions include: intranasal, inhaled, topical steroids, or local steroid injections (e.g., intra articular injection); Systemic corticosteroids at physiologic doses not to exceed 10 mg/day ofprednisone or its equivalent; Steroids as premedication for hypersensitivity reactions (e.g., CT scan premedication) * No receipt of live attenuated vaccine within 30 days prior to treatment start Note: enrolled subjects should not receive live vaccine while receiving IP and up to 30 days after the last dose of study therapy * Concomitant use of known strong CYP3A inhibitors (e.g., itraconazole, telithromycin, clarithromycin, protease inhibitors boosted with ritonavir or cobicistat, indinavir, saquinavir, nelfinavir, boceprevir, telaprevir) or moderate CYP3A inhibitors (e.g. ciprofloxacin, erythromycin, diltiazem, fluconazole, verapamil). The required washout period prior to treatment start is 2 weeks. * Concomitant use of known strong (e.g., phenobarbital, enzalutamide, phenytoin, rifampicin, rifabutin, rifapentine, carbamazepine, nevirapine and St John's Wort) or moderate CYP3A inducers (e.g., bosentan, efavirenz, modafinil). The required washout period prior to starting olaparib is 5 weeks for enzalutamide or phenobarbital and 3 weeks for other agents. * More than 2 cycles of intermittent hormones for the treatment of biochemical recurrence, with a cycle defined as a period of consistent ADT (generally 3-12 months) followed by intentional cessation of ADT without re-initiation of ADT until the PSA rises. Prior ADT in the treatment of localized prostate cancer or with salvage radiation therapy is allowed. Prior use of abiraterone acetate with prednisone, enzalutamide, apalutamide, or other androgen receptor/androgen biosynthesis inhibitors are allowed if used in the localized or biochemically recurrent disease state provided that there was no evidence of disease progression while on these therapies. * No medical conditions such as uncontrolled hypertension, uncontrolled diabetes mellitus, cardiac disease that would, in the opinion of the investigator, make this protocol unreasonably hazardous. * Subjects considered a poor medical risk due to a serious, uncontrolled medical disorder or non-malignant systemic disease. Examples include, but are not limited to, uncontrolled ventricular arrhythmia, myocardial infarction within 3 months of treatment start, uncontrolled major seizure disorder, unstable spinal cord compression, superior vena cava syndrome, extensive interstitial bilateral lung disease on High Resolution Computed Tomography (HRCT) scan or any psychiatric disorder that prohibits obtaining informed consent. * No active infection including tuberculosis (TB) (clinical evaluation that includes clinical history, physical examination and radiographic findings, and TB testing in line with local practice), hepatitis B (known positive HBV surface antigen (HBsAg) result), hepatitis C, or active infection with human immunodeficiency virus (positive HIV 1/2 antibodies). Subjects with a past or resolved HBV infection (defined as presence of hepatitis B core antibody \[anti-HBc\] and absence of HBsAg) are eligible. Subjects positive for hepatitis C (HCV) antibody are eligible only if polymerase chain reaction is negative for HCV RNA * No autoimmune disease: subjects with a history of inflammatory bowel disease, including ulcerative colitis and Crohn's Disease, are excluded from this study, as are subjects with a history of symptomatic disease (e.g., rheumatoid arthritis, systemic progressive sclerosis \[scleroderma\], systemic lupus erythematosus, autoimmune vasculitis \*e.g., Sarcoidosis syndrome or Wegener's granulomatosis with polyangiitis+); motor neuropathy considered of autoimmune origin (e.g., Guillain-Barre syndrome and myasthenia gravis); Graves' disease. Exceptions include history of eczema, vitiligo, alopecia, hypothyroidism (e.g., following Hashimoto syndrome) , and any chronic skin condition that does not require systemic therapy; subjects without active disease in the last 5 years prior to treatment start may be included but only after consultation with the treating physician. Exceptions may be made on a case by case basis upon discussion with the Sponsor Principal Investigator. * No history of active primary immunodeficiency * No major surgery within 4 weeks of treatment start. Subjects must have recovered from any significant effects of any major surgery but investigators may discuss with the Sponsor Principal Investigator in the case of any exceptions. * No blood transfusion within 28 days of treatment start. * Resting ECG with QTc \> 470 msec on 2 or more time points within a 24 hour period or family history of long QT syndrome * Enrollment in another clinical trial with a therapeutic agent. Subjects may co-enroll on investigational imaging studies (e.g., PSMA PET) or correlative trials. * No previous allogeneic bone marrow transplant or double umbilical cord blood transplant. * No history of leptomeningeal carcinomatosis * No unresolved toxicity (Common Terminology Criteria for Adverse Event (CTCAE) Grade \>/= 2) caused by previous anticancer therapy, excluding alopecia, vitiligo, and the laboratory values described in the inclusion criteria. Subjects with Grade \>/= 2 neuropathy will be evaluated on a case-by-case basis after consultation with the Sponsor Principal Investigator. Subjects with irreversible toxicity not reasonably expected to be exacerbated by treatment with study drugs may be included only after consultation with the Sponsor Principal Investigator * No subjects who are HIV-positive on combination antiretroviral therapy because of the potential for pharmacokinetic interactions with olaparib. In addition, these subjects are at increased risk of lethal infections when treated with marrow suppressive therapy. * No subjects with baseline moderate to severe hepatic impairment (Child-Pugh Class B and C). * No subjects with myelodysplastic syndrome/acute myeloid leukemia or with features suggestive of MDS/AML. * No known allergy to any of the compounds under investigation or excipients of the product. * Subjects unable to swallow orally administered medication and subjects with gastrointestinal disorders likely to interfere with absorption of the study medication. * No other condition which, in the opinion of the investigator, would preclude participation in this trial.