Testing the Safety of the Anti-Cancer Drugs Durvalumab and Olaparib During Radiation Therapy for Locally Advanced Unresectable Pancreatic Cancer

Inclusion Criteria: * Patients must have histologically confirmed pancreatic cancer (excluding islets) (not otherwise specified \[NOS\]) (Medical Dictionary for Regulatory Activities \[MEDDRA\] code: 10033612). * Patients must have unresectable locally advanced pancreatic cancer as determined by a multidisciplinary tumor board applying National Comprehensive Cancer Network (NCCN) version (v)2.2021 criteria or as surgically determined during failed resection attempt. * Patients must have had prior first-line chemotherapy for this cancer for at least 16 weeks without clinical, biochemical, or radiologic progression. There should be a washout of at least 2 weeks from first-line chemotherapy and start of therapy on clinical trial. * Age \>= 18 years. Because no dosing or adverse event data are currently available on the use of durvalumab and olaparib in combination with radiation in patients \< 18 years of age, children are excluded from this study. * Body weight \> 30 kg. * Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (Karnofsky \>= 70%). * Hemoglobin \>= 9.0 g/dL without blood transfusion in last 4 weeks (within 2 weeks of enrollment). * Absolute neutrophil count \>= 1,500/mcL (within 2 weeks of enrollment). * Platelets \>= 100,000/mcL (within 2 weeks of enrollment). * Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN) (within 2 weeks of enrollment). * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine transferase (ALT) (serum glutamic pyruvic transaminase) \[SGPT\]) =\< 2.5 x institutional ULN (within 2 weeks of enrollment). * Creatinine =\< 1.5 x institutional ULN (within 2 weeks of enrollment). * Measured creatinine clearance \> 60 mL/min/1.73 m\^2 (within 2 weeks of enrollment). * Evidence of post-menopausal status or negative urinary or serum pregnancy test for female pre-menopausal patients. Women will be considered post-menopausal if they have been amenorrheic for 12 months without an alternative medical cause. The following age-specific requirements apply: * Women \< 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of exogenous hormonal treatments and if they have luteinizing hormone and follicle-stimulating hormone levels in the post-menopausal range for the institution or underwent surgical sterilization (bilateral oophorectomy or hysterectomy). * Women \>= 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of all exogenous hormonal treatments, had radiation-induced menopause with last menses \> 1 year ago, had chemotherapy-induced menopause with last menses \> 1 year ago, or underwent surgical sterilization (bilateral oophorectomy, bilateral salpingectomy or hysterectomy). * Life expectancy \>= 16 weeks. * Patients who are human immunodeficiency virus (HIV) positive may participate IF they meet the following eligibility requirements: * They must be stable on their anti-retroviral regimen with evidence of at least two undetectable viral loads within the past 6 months on the same regimen; the most recent undetectable viral load must be within the past 12 weeks. * They must have a CD4 count of greater than 250 cells/mcL over the past 6 months on this same anti-retroviral regimen and must not have had a CD4 count \< 200 cells/mcL over the past 2 years, unless it was deemed related to the cancer and/or chemotherapy induced bone marrow suppression. * For patients who have received chemotherapy in the past 6 months, a CD4 count \< 250 cells/mcL during chemotherapy is permitted as long as viral loads were undetectable during this same chemotherapy. * They must have an undetectable viral load and a CD4 count \>= 250 cells/mcL within 7 days of enrollment. * They must not be currently receiving prophylactic therapy for an opportunistic infection and must not have had an opportunistic infection within the past 6 months. * HIV-infected patients should be monitored every 12 weeks for viral load and CD4 counts. * For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated. Patients with a past or resolved HBV infection (defined as the presence of hepatitis B core antibody \[anti-HBc\] and absence of hepatitis B surface antigen \[HbsAg\]) are eligible. * Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load. Patients positive for HCV antibody are eligible only if polymerase chain reaction (PCR) is negative for HCV ribonucleic acid (RNA). * Patients with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial. * Willing to provide archived tissue, if available, from a previous biopsy. If tissue from initial biopsy is not available, a repeat biopsy is NOT required and patient will be eligible for enrollment. * Patients must have radiographically measurable or evaluable disease (as per Response Evaluation Criteria in Solid Tumors \[RECIST\]v1.1). * Must be able to tolerate CT and/or MRI with contrast. * Patients with known history or current symptoms of cardiac disease, or history of treatment with cardiotoxic agents, should behave a clinical risk assessment of cardiac function using the New York Heart Association Functional Classification. To be eligible for this trial, patients should be class 2B or better. * The effects of durvalumab and olaparib on the developing human fetus are unknown. For this reason and because radiotherapy used in this trial is known to be teratogenic, women of child-bearing potential and men must agree to use adequate contraception prior to study entry, for the duration of study participation, and for 6 months after the last dose of study treatment. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. Male patients should not donate sperm throughout the period of taking olaparib and for 6 months following the last dose of olaparib. All females of childbearing potential (not surgically sterilized, and between menarche and 1 year post menopause) must have a negative screening pregnancy test. * Ability to understand and the willingness to sign a written informed consent document. Participants with impaired decision-making capacity who have a legally-authorized representative (LAR) and/or family member available will also be eligible. Exclusion Criteria: * Patients who have had prior upper abdominal radiotherapy prior to entering the study. * Patients who have not recovered from grade \>= 2 adverse events (AEs) due to prior anti-cancer therapy with the exception of alopecia, vitiligo, and the laboratory values defined in the inclusion criteria. * Patients with grade \>= 2 neuropathy will be evaluated on a case-by-case basis after consultation with the Study Physician. * Patients with irreversible toxicity not reasonably expected to be exacerbated by treatment with durvalumab may be included only after consultation with the Study Physician. * Major surgical procedure within 28 days prior to enrollment. Note: Local surgery of isolated lesions for palliative intent is acceptable. Laparoscopy and/or laparotomy without resection does not constitute a major surgical procedure. * Current or prior use of immunosuppressive medication within 14 days before the first dose of durvalumab. The following are exceptions to this criterion: * Intranasal, inhaled, topical steroids, or local steroid injections (e.g., intra articular injection) * Systemic corticosteroids at physiologic doses not to exceed \<\<10 mg/day\>\> of prednisone or its equivalent * Steroids as premedication for hypersensitivity reactions (e.g., CT scan premedication) * Receipt of live attenuated vaccine within 30 of planned start of study therapy. Note: Patients, if enrolled, should not receive live vaccine whilst receiving investigational product (IP) and up to 30 days after the last dose of IP. * Patients who are receiving any other investigational agents. * History of allergic reactions attributed to compounds of similar chemical or biologic composition to olaparib or durvalumab. * Patients with any other significant condition(s) that would make this protocol unreasonably hazardous. * Patients receiving any medications or substances that are inhibitors or inducers of CYP3A4/5 are ineligible. Because the lists of these agents are constantly changing, it is important to regularly consult a frequently-updated medical reference. As part of the enrollment/informed consent procedures, the patient will be counseled on the risk of interactions with other agents, and what to do if new medications need to be prescribed or if the patient is considering a new over-the-counter medicine or herbal product. The required washout period prior to starting olaparib is 5 weeks for enzalutamide or phenobarbital, and 3 weeks for other agents. * Must not have prior history of organ transplantation, allogeneic transplantation, or double umbilical cord transplantation. * Must not have germline BRCA1 or BRCA2 mutation. * Prisoners or subjects who are involuntarily incarcerated, or compulsorily detained for treatment of either a psychiatric or physical (e.g. infectious disease) illness would be excluded. * Patients must not have previously received anti-PD-1/PD-L1 antibodies or PARP inhibitor for treatment of this cancer. * Patients must not have myelodysplastic syndrome/acute myeloid leukemia or features suggestive of myelodysplastic syndrome (MDS)/acute myeloid leukemia (AML). * Pregnant women are excluded from this study because olaparib is a small molecule agent with the potential for teratogenic or abortifacient effects. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with olaparib, breastfeeding should be discontinued if the mother is treated with olaparib. These potential risks may also apply to other agents used in this study. * Participants must not have an active, known or suspected autoimmune disease which may affect vital organ function or has/may require systemic immunosuppressive therapy for management. Participants with inflammatory disorders (including inflammatory bowel disease \[e.g., colitis or Crohn's disease\], diverticulitis \[with the exception of diverticulosis\], systemic lupus erythematosus, Sarcoidosis syndrome, or Wegener syndrome \[granulomatosis with polyangiitis, Graves' disease, rheumatoid arthritis, hypophysitis, uveitis, etc.\]) are also excluded with the exception of the following: * Patients with vitiligo or alopecia. * Patients with hypothyroidism (e.g. following Hashimoto syndrome) stable on hormone replacement. * Any chronic skin condition that does not require systemic therapy. * Patients without active disease in the last 5 years may be included but only after consultation with the study physician. * Patients with celiac disease controlled by diet alone. * Participants with type I diabetes mellitus, hypothyroidism only requiring hormone replacement, skin disorders (such as vitiligo, psoriasis, or alopecia) not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger (such as celiac disease controlled by diet) are permitted to enroll. Patients without active disease for 5 years may also be enrolled after consultation with the study monitor or sponsor. * Patients must not have serious, uncontrolled medical disorder, non-malignant systemic disease. Examples include but are not limited to active heart disease including symptomatic heart failure (New York Heart Association \[NYHA\] class 3 or 4), unstable angina pectoris, uncontrolled cardiac arrhythmia or interstitial lung disease, uncontrolled major seizure disorder, unstable spinal cord compression, superior vena cava syndrome, extensive interstitial bilateral lung disease on high resolution computed tomography (HRCT) scan. * Must not have known active inflammatory gastrointestinal disease, chronic diarrhea (other than exocrine insufficiency controlled by enzyme replacement therapy), short gut syndrome, or other conditions that would limit the absorption of the study drug. Patient must be able to swallow and retain an oral medication. * Active infection including tuberculosis (clinical evaluation that includes clinical history, physical examination and radiographic findings, and tuberculosis \[TB\] testing in line with local practice), hepatitis B (known positive HBV surface antigen \[HbsAg\] result), or hepatitis C. * In addition, the patient must not have resting electrocardiogram (ECG) indicating uncontrolled, potentially reversible cardiac conditions, as judged by the investigator (e.g., unstable ischemia, uncontrolled symptomatic arrhythmia, congestive heart failure, corrected QT interval by Fredericia \[QTcF\] prolongation \> 500 ms, electrolyte disturbances, etc.), or patients with congenital long QT syndrome.