Trimodality Approach of Low Dose iOnizing Radiation With or Without Neoadjuvant Pembrolizumab and Prostatectomy for Men With Intermediate/High Risk Prostate Cancer (TALON)

Inclusion Criteria: 1. Ability to understand and the willingness to sign a written informed consent document. 2. Age ≥ 18 years 3. Histologic evidence of adenocarcinoma of the prostate who are deemed candidates for radical prostatectomy. Variants such as neuroendocrine components or ductal carcinoma are allowed. Pure small cell carcinoma is not allowed. 4. At least one unfavorable intermediate or high risk factors or classified as high risk clinically localized disease as defined by NCCN guidelines: a. High risk: At least one of the following: i. cT3a disease ii. Gleason sum of 8, 9 or 10 iii. PSA ≥ 20 ng/ml b. At least one of the following unfavorable intermediate risk factors:: i. cT2b or cT2c disease ii. Grade group 3 iii. PSA 10-20 ng/ml iv. Grade group 2 and 50% or greater prostate biopsies positive for cancer 5. Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1 (See Appendix A) 6. Adequate normal organ and marrow function as defined below by the following criteria within 10 days prior to first dose of study treatment.: 1. Hemoglobin ≥ 9.0 g/dL 2. Absolute neutrophil count (ANC ≥1.5 x 10\^9/L) 3. Platelet count ≥100 x 10\^9/L 4. Serum bilirubin ≤ 1.5 x Institutional Upper Limit of Normal (ULN) 5. AST/SGOT and ALT/SGPT ≤ 2.5 x ULN 6. Measured creatinine clearance (CL) \>50 mL/min 8\. Patient is willing and able to comply with the protocol for the duration of the study including undergoing treatment and scheduled visits and examinations including follow up. Exclusion Criteria: 1. History of or known bone, brain, visceral, or soft tissue metastasis, including lymph nodes based on standard of care imaging with CT or pelvic MRI showing no LNs greater than 1.5cm and bone scan showing no evidence of bone metastasis. 2. Prior pelvic radiation or prostate cryotherapy or high-intensity focused ultrasound (HIFU) 3. Any prior treatment with PD-1 or PD-L1 checkpoint inhibitors or with an agent directed to another stimulatory or co-inhibitory T-cell receptor (e.g., CTLA-4, OX 40, CD137, PD-L2). 4. Has received an investigational agent (or used an investigational device) within 4 weeks prior to study intervention administration. a. Note: Participants who have entered the follow-up phase of an investigational study may participate as long as it has been 4 weeks after the last dose of the previous investigational agent. 5. Prior therapy for prostate cancer a. Exceptions: Previous alpha-reductase inhibitor use allowed IF patient has not been taking for at least 30 days prior to study treatment initiation, OR if alpha reductase inhibitor was not used as a primary treatment of prostate cancer and the PSA on alpha-reductase inhibitor remains within eligibility when doubled. 6. Any concurrent chemotherapy, investigational product, biologic, or hormonal therapy for cancer treatment. Concurrent use of hormonal therapy for non-cancer-related conditions (e.g., hormone replacement therapy) is acceptable. 7. Has active autoimmune disease that has required systemic treatment in the past 2 years except replacement therapy (e.g. thyroxine, insulin, or physiologic corticosteroid). 8. Presence of a condition requiring chronic steroid use (equivalent to \>10 mg of prednisone daily) or other immunosuppressive drugs (i.e., for organ transplant). The following are exceptions to this criterion: 1. Intranasal, inhaled, topical steroids, or local steroid injections (e.g., intra articular injection) 2. Steroids as premedication for hypersensitivity reactions (e.g., CT scan premedication) 9. Uncontrolled intercurrent illness, including but not limited to, ongoing or active infection, symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, cardiac arrhythmia, interstitial lung disease, serious chronic gastrointestinal conditions associated with diarrhea, or psychiatric illness/social situations that would limit compliance with study requirement, substantially increase risk of incurring AEs or compromise the ability of the patient to give written informed consent 10. History of another primary malignancy except for: 1. Malignancy treated with curative intent and with no known active disease ≥5 years before the first dose of investigational product (IP) and of low potential risk for recurrence 2. Adequately treated non-melanoma skin cancer or lentigo maligna without evidence of disease 3. Adequately treated carcinoma in situ without evidence of disease 11. History of allogenic stem cell transplant 12. History of active primary immunodeficiency 13. Known history of human immunodeficiency virus 14. Has a known history of Hepatitis B (defined as Hepatitis B surface antigen \[HBsAg\] reactive) or known active Hepatitis C virus (defined as HCV RNA \[qualitative\] is detected) infection. 15. Has not adequately recovered from major surgery or has ongoing surgical complications. 16. Has received a live vaccine within 30 days prior to the first dose of study drug. Examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, varicella/zoster (chicken pox), yellow fever, rabies, Bacillus Calmette-Guérin (BCG), and typhoid vaccine. Seasonal influenza vaccines for injection are generally killed virus vaccines and are allowed; however, intranasal influenza vaccines (eg, FluMist®) are live attenuated vaccines and are not allowed. 17. Has a history of (non-infectious) pneumonitis that required steroids or has current pneumonitis. 18. Has severe hypersensitivity (≥Grade 3) to pembrolizumab and/or any of its excipients. 19. Any condition which, in the opinion of the investigator, would preclude participation in this trial