Durvalumab and Olaparib for the Treatment of Prostate Cancer in Men Predicted to Have a High Neoantigen Load

Inclusion Criteria: * Histologic confirmation of adenocarcinoma of the prostate * The patient must have received definitive local therapy for prostate cancer, consisting of either radiation therapy and/or prostatectomy (salvage or adjuvant radiation post-prostatectomy is not exclusionary) * PSA must be \>= 2 ng/ml if received only prior definitive radiation (no PSA threshold required if prior prostatectomy was performed) with a PSA doubling time (PSADT) =\< 10 months: * PSADT calculation must include all recorded PSA values \> 0.2 ng/ml over the past 6 months prior to randomization, with a minimum of 3 values spaced at least 2 weeks apart, with each included value preferably measured at the same laboratory. PSA values obtained prior to localized therapy will be excluded * The calculation of PSADT is based on the natural log of PSA * Prior salvage radiation or not a candidate for localized salvage radiation due to subject preference or clinical assessment based upon disease characteristics and/or subject co-morbidities * Prior hormonal therapy (i.e. androgen deprivation therapy) when given as neoadjuvant/concurrent/adjuvant therapy along with definitive radiation is allowed, provided this was stopped \>= 6 months prior to starting treatment per protocol AND testosterone is \>= 150 ng/dl * No evidence of metastatic disease on imaging by whole body bone scan and computed tomography (CT) or magnetic resonance imaging (MRI) of the chest/abdomen/pelvis within 6 weeks before study therapy start day. PSMA positron emission tomography (PET) or fluciclovine scan within 6 weeks of start day may substitute other imaging studies. * Patients with oligometastatic disease (i.e. =\< 3 sites) detectable on advanced imaging only (e.g. PSMA or fluciclovine PET) are eligible * Abdominal or pelvic lymph nodes measuring =\< 2 cm in short axis are allowed * Biomarker positive: * Biallelic CDK12 inactivating mutations as documented using a clinical grade sequencing assay performed in a Clinical Laboratory Improvement Act (CLIA)/College of American Pathologists (CAP) certified laboratory or * MMRd/MSI-high as documented using a clinical grade assay performed in a CLIA/CAP certified laboratory or * Loss of function mutations in homologous recombination genes (i.e. homologous recombination deficiency; HRD) as documented using a clinical grade sequencing assay performed in a CLIA/CAP certified laboratory. Homologous recombination genes include, but not limited to BRCA1, BRCA2, ATM, CHEK2, PALB2, RAD51D, NBN, GEN1, RAD51C, MRE11A, BRIP11A, FAM175A. * Capable of giving signed informed consent which includes compliance with the requirements and restrictions listed in the informed consent form (ICF) and in this protocol. Written informed consent and any locally required authorization (e.g., Health Insurance Portability and Accountability Act) obtained from the patient/legal representative prior to performing any protocol-related procedures, including screening evaluations * Age \> 18 years at time of study entry. * Eastern Cooperative Oncology Group (ECOG) performance status of =\< 1 * Hemoglobin \>= 10.0 g/dL with no blood transfusion in the past 28 days (within 28 days prior to administration of study treatment) * Absolute neutrophil count (ANC) be \>= 1.5 x 10\^9/L (within 28 days prior to administration of study treatment) * Platelet count \>= 100 x 10\^9/L (within 28 days prior to administration of study treatment) * Serum bilirubin =\< 1.5 x institutional upper limit of normal (ULN) (within 28 days prior to administration of study treatment). This will not apply to patients with confirmed Gilbert's syndrome (persistent or recurrent hyperbilirubinemia that is predominantly unconjugated in the absence of hemolysis or hepatic pathology), who will be allowed only in consultation with their physician * Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x institutional upper limit of normal unless liver metastases are present, in which case it must be =\< 5 x ULN (within 28 days prior to administration of study treatment) * Patient must have creatinine clearance (CL) \>= 51 mL/min by the Cockcroft-Gault formula (Cockcroft and Gault 1976) or by 24-hour urine collection for determination of creatinine clearance (within 28 days prior to administration of study treatment) * Patient is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations including follow up * Patients must have a life expectancy \>= 16 weeks * Body weight \> 30 kg * Male patients must use a condom during treatment and for 3 months after the last dose of olaparib when having sexual intercourse with a pregnant woman or with a woman of childbearing potential. Female partners of male patients should also use a highly effective form of contraception if they are of childbearing potential Exclusion Criteria: * Prior chemotherapy for prostate cancer, unless done in the neoadjuvant setting, and if the last dose was \> 6 months prior to enrollment * Any prior treatment with a PD1 or PD-L1 inhibitor, including durvalumab * Any prior treatment with a PARP inhibitor, including olaparib * History of another primary malignancy except for * Malignancy treated with curative intent and with no known active disease \>= 3 years before the first dose of durvalumab and of low potential risk for recurrence * Adequately treated non-melanoma skin cancer or lentigo maligna without evidence of disease * Adequately treated carcinoma in situ without evidence of disease * History of leptomeningeal carcinomatosis * Resting electrocardiogram (ECG) indicating uncontrolled, potentially reversible cardiac conditions, as judged by the treating physician (e.g., unstable ischemia, uncontrolled symptomatic arrhythmia, congestive heart failure, corrected QT interval by Fridericia \[QTcF\] prolongation \> 470 ms, electrolyte disturbances, etc.), or patients with congenital long QT syndrome * Any unresolved toxicity National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) grade \>= 2 from previous anticancer therapy with the exception of alopecia, vitiligo, and the laboratory values defined in the inclusion criteria * Patients with grade \>= 2 neuropathy will be evaluated on a case-by-case basis after consultation with the study physician * Patients with irreversible toxicity not reasonably expected to be exacerbated by treatment with durvalumab or olaparib may be included only after consultation with the study physician * Patients with myelodysplastic syndrome/acute myeloid leukemia or with features suggestive of MDS/AML * Patients considered a poor medical risk due to a serious, uncontrolled medical disorder, non-malignant systemic disease or active, uncontrolled infection. Examples include, but are not limited to, extensive bilateral lung disease on high resolution computed tomography (HRCT) scan, uncontrolled ventricular arrhythmia, recent (within 3 months) myocardial infarction, uncontrolled major seizure disorder, unstable spinal cord compression, superior vena cava syndrome, interstitial lung disease, serious chronic gastrointestinal conditions associated with diarrhea or any psychiatric disorder that prohibits obtaining informed consent * Patients unable to swallow orally administered medication and patients with gastrointestinal disorders likely to interfere with absorption of the study medication * Immunocompromised patients, e.g., patients with uncontrolled human immunodeficiency virus (HIV). HIV+ patients will be allowed on the study if on highly active antiretroviral therapy (HAART) and disease is controlled: CD4 \>= 350 cell/mcl, undetectable viral load, and no prophylactic (PPX) antibiotics * Note: HIV screening is not required to be eligible for this study * Active infection including tuberculosis (TB testing only performed if deemed necessary per standard clinical practice), hepatitis B (known positive hepatitis B virus \[HBV\] surface antigen \[HBsAg\] result), hepatitis C. Patients with a past or resolved HBV infection (defined as the presence of hepatitis B core antibody \[anti-HBc\] and absence of HBsAg) are eligible. Patients positive for hepatitis C virus (HCV) antibody are eligible only if polymerase chain reaction is negative for HCV ribonucleic acid (RNA) * Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease \[e.g., colitis or Crohn's disease\], diverticulitis \[with the exception of diverticulosis\], systemic lupus erythematosus, Sarcoidosis syndrome, or Wegener syndrome \[granulomatosis with polyangiitis, Graves' disease, rheumatoid arthritis, hypophysitis, uveitis, etc\]). The following are exceptions to this criterion: * Patients with vitiligo or alopecia * Patients with hypothyroidism (e.g., following Hashimoto syndrome) stable on hormone replacement * Any chronic skin condition that does not require systemic therapy * Patients without active disease in the last 5 years may be included but only after consultation with the sponsor investigator * Patients with celiac disease controlled by diet alone * Current or prior use of immunosuppressive medication within 14 days before the first dose of durvalumab. The following are exceptions to this criterion: * Intranasal, inhaled, topical steroids, or local steroid injections (e.g., intra articular injection) * Systemic corticosteroids at physiologic doses not to exceed 10 mg/day of prednisone or its equivalent * Steroids as premedication for hypersensitivity reactions (e.g., CT scan premedication) * Receipt of live attenuated vaccine within 30 days prior to the first dose of durvalumab. Note: Patients, if enrolled, should not receive live vaccine whilst receiving durvalumab and up to 30 days after the last dose of durvalumab * Patients receiving any chemotherapy, immunotherapy, biologic, radiotherapy or hormonal therapy for cancer treatment concurrently or within 3 weeks of study treatment. Concurrent use of hormonal therapy for non-cancer related conditions (e.g., hormone replacement therapy) is acceptable * Radiotherapy treatment to more than 30% of the bone marrow or with a wide field of radiation within 4 weeks of the first dose of study drug * Concomitant use of known strong CYP3A inhibitors (e.g. itraconazole, telithromycin, clarithromycin, protease inhibitors boosted with ritonavir or cobicistat, indinavir, saquinavir, nelfinavir, boceprevir, telaprevir) or moderate CYP3A inhibitors (e.g. ciprofloxacin, erythromycin, diltiazem, fluconazole, verapamil). The required washout period prior to starting study treatment is 2 weeks * Concomitant use of known strong (e.g. phenobarbital, enzalutamide, phenytoin, rifampicin, rifabutin, rifapentine, carbamazepine, nevirapine and St John's wort) or moderate CYP3A inducers (e.g. bosentan, efavirenz, modafinil). The required washout period prior to starting study treatment is 5 weeks for enzalutamide or phenobarbital and 3 weeks for other agents * Major surgery within 2 weeks of starting study treatment and patients must have recovered from any effects of any major surgery * Previous allogenic bone marrow transplant or double umbilical cord blood transplantation (dUCBT) * Participation in another clinical study with an investigational product administered in the last 3 months * Concurrent enrollment in another clinical study, unless it is an observational (non-interventional) or supportive care, while on the clinical study or during the follow-up period of this interventional study * Patients with a known hypersensitivity to olaparib or durvalumab or any of the excipients of the product * Involvement in the planning and/or conduct of the study