Pembrolizumab +/- SD-101 in Hormone-Naïve Oligometastatic Prostate Cancer With RT and iADT

Inclusion Criteria: 1. Be willing and able to provide written informed consent/assent for the trial. 2. Be \>=18 years of age on day of signing informed consent. 3. Have a performance status of 0 or 1 on the Eastern Cooperative Oncology Group (ECOG) Performance Scale. 4. Histologically documented adenocarcinoma of the prostate 5. Oligometastatic disease. In order to be eligible, the participant must have a total of \<4 metastatic bone and/or metastatic lymph node sites based on bone and/or soft tissue lesions as defined by any of the following: 1. Bone metastases will be defined by bone imaging. If the participant has technetium bone scan, and/or sodium fluoride (NaF) PET performed, either study may be used for documenting metastases; both scans do not need to show the number of metastases required for study entry. For participants undergoing PSMA PET, only PSMA avid lesions that are consistent with metastasis will be counted as a site of metastasis. 2. Distant metastatic lymph node disease. A lymph node ≥1 cm in shortest dimension will be noted as involved with disease. Distant metastatic lymph nodes will be determined as any lymph nodes outside the confines of the true pelvis. For participants undergoing PSMA PET, only PSMA avid lesions that are consistent with metastasis will be counted as a site of metastasis. 3. Any other soft tissue lesion deemed by the physician to be consistent with distant metastatic disease. For participants undergoing PSMA PET, only PSMA avid lesions that have a CT or MRI correlate consistent with metastasis will be counted as a site of metastasis. * Note: Radiographic imaging performed as standard of care prior to obtaining informed consent and within 60 days of initiating study treatment may be used to assess oligometastatic disease during screening, rather than repeating scans. For participants who have started on ADT, they must have had imaging prior to initiation of hormonal therapy 6. Treatment naïve, defined as less than 2 months of standard of care ADT (e.g. GnRH agonist or antagonist with or without antiandrogen, including abiraterone) for metastatic hormone-sensitive prostate cancer prior to enrollment (at the time of consent) 7. No prior chemotherapy for prostate cancer 8. Not a candidate for or refuse chemotherapy 9. No prior prostatectomy or prostatic radiation 10. PSA \>2 ng/mL at baseline or prior to initiation of hormonal therapy 11. Baseline testosterone \>150 ng/dL if participant has not initiated hormonal therapy, for those participants who have already initiated hormonal therapy, baseline testosterone is not required 12. Consent to undergo mandatory prostatic core biopsies at the time of fiducial marker placement and 1-3 weeks after Cycle 1 Day 1 of pembrolizumab. 13. The effects of pembrolizumab on the developing human fetus is unknown. Men treated or enrolled on this protocol must agree to use adequate contraception prior to the first dose of study therapy, for the duration of the study participation, and for 120 days after the last dose of study therapy. Their partners should also be encouraged to use proper method of contraception. Their partners should also be encouraged to use proper method of contraception. 14. Demonstrate adequate organ function defined as: Adequate Organ Function Laboratory Values (Performed within 10 days of treatment initiation): * Absolute neutrophil count (ANC) ≥1,500 /microliter (mcL) * Platelets \>=100,000 / mcL * Hemoglobin \>=9 g/dL or ≥5.6 mmol/L without transfusion or erythropoietin (EPO) dependency (within 7 days of assessment) * Serum creatinine OR Measured or calculated creatinine clearance (GFR can also be used in place of creatinine or CrCl) \<=1.5 X upper limit of normal (ULN) OR ≥60 mL/min for subject with creatinine levels \> 1.5 X institutional ULN. Creatinine clearance should be calculated per institutional standard * Serum total bilirubin \<= 1.5 X ULN OR Direct bilirubin \<= ULN for subjects with total bilirubin levels \> 1.5 ULN * Aspartate aminotransferase (AST) (SGOT) and alanine aminotransferase (ALT) (SGPT) ≤ 2.5 X ULN OR ≤ 5 X ULN for subjects with liver metastases * Albumin \>2.5 mg/dL * International Normalized Ratio (INR) or Prothrombin Time (PT) * Activated Partial Thromboplastin Time (aPTT) ≤1.5 X ULN unless subject is receiving anticoagulant therapy as long as PT or PTT is within therapeutic range of intended use of anticoagulants ≤1.5 X ULN unless subject is receiving anticoagulant therapy as long as PT or PTT is within therapeutic range of intended use of anticoagulants * Creatinine clearance should be calculated per institutional standard. Exclusion Criteria: 1. Participants who are not appropriate candidates for prostate or oligometastasis-directed SBRT 2. Participants with neuroendocrine or small cell features are not eligible. 3. Participants with evidence of liver metastasis are excluded. 4. Gonadotropin-releasing hormone (GnRH) agonists or GnRH antagonists (e.g., leuprorelin, degarelix) for \> 2 months prior to consenting 5. Antiandrogens (e.g., bicalutamide, flutamide, nilutamide, abiraterone, enzalutamide) for \> 2 months prior to consenting. Participants on 5-alpha reductase inhibitors are allowed on study. 6. Estrogen containing compounds for \> 2 months prior to consenting 7. PC-SPES or PC-x products. Other herbal therapies or supplements will be considered by the Principle Investigator on a case-by-case basis based on their potential for hormonal or anti-cancer therapies. 8. Prior immunotherapy or chemotherapy for prostate cancer 9. Prior radiation therapy to the prostate 10. Prior prostatectomy 11. Is currently participating and receiving study therapy or has participated in a study of an investigational agent and received study therapy or used an investigational device within 4 weeks of the first dose of treatment. 12. Has a diagnosis of immunodeficiency or is receiving systemic steroid therapy or any other form of systemic immunosuppressive therapy within 7 days prior to the first dose of trial treatment, with the exception of steroids for adrenal insufficiency in which case prednisone =\<10mg/day or its equivalent is allowed. 13. Has a known history of active Bacillus Tuberculosis (TB) 14. Hypersensitivity to pembrolizumab or any of its excipients. 15. Has had a prior anti-cancer monoclonal antibody (mAb) within 4 weeks prior to study Day 1 or who has not recovered (i.e., ≤ Grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier. 16. Has had prior chemotherapy, targeted small molecule therapy, or radiation therapy within 2 weeks prior to study Day 1 or who has not recovered (i.e., ≤ Grade 1 or at baseline) from adverse events due to a previously administered agent. * Note: Subjects with \<= Grade 2 neuropathy are an exception to this criterion and may qualify for the study. * Note: If subject received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting 17. Has a known additional malignancy that is progressing or requires active treatment. Exceptions include carcinoid, basal cell carcinoma of the skin or squamous cell carcinoma of the skin that has undergone potentially curative therapy or in situ cervical cancer. 18. Has known active central nervous system (CNS) metastases and/or carcinomatous meningitis. Subjects with previously treated brain metastases may participate provided they are stable (without evidence of progression by imaging for at least four weeks prior to the first dose of trial treatment and any neurologic symptoms have returned to baseline), have no evidence of new or enlarging brain metastases, and are not using steroid treatment for at least 14 days prior to the first dose of study treatment. This exception does not include carcinomatous meningitis which is excluded regardless of clinical stability. 19. Has active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment. 20. Has known history of, or any evidence of active, non-infectious pneumonitis. 21. Has an active infection requiring systemic therapy. 22. Has a history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the trial, interfere with the subject's participation for the full duration of the trial, or is not in the best interest of the subject to participate, in the opinion of the treating investigator. 23. Has known psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the trial. 24. Expecting to father children within the projected duration of the trial, starting with the pre-screening or screening visit through 120 days after the last dose of trial treatment. 25. Has received prior therapy with an anti-programmed cell death protein 1 (PD-1), anti-programmed death-ligand 1 (PD-L1) , or anti-Programmed cell death 1 ligand 2 (PD-L2) agent. 26. Has a known history of Human Immunodeficiency Virus (HIV) (HIV 1/2 antibodies). 27. Has known active Hepatitis B (e.g., HBsAg reactive) or Hepatitis C (HCV) (i.e. HCV RNA \[qualitative\] is detected). 28. Has received a live vaccine within 30 days of planned start of study therapy. a. Note: Seasonal influenza vaccines for injection are generally inactivated flu vaccines and are allowed; however intranasal influenza vaccines (e.g., Flu-Mist®) are live attenuated vaccines, and are not allowed within 30 days.